REPLICATE REPLICATE - HOW YOU CAN TAKE PART
YSI Webinar on Replication
August 2022 - December 2022
Replicating Empirical Studies in Economics
Have you ever wondered if the results presented in an article or a study you read were correct? Have you ever wondered how the choice of methodology or data can influence results?
IN A JOINT INITIATIVE, ReplicationWiki, Project Teaching Integrity in Empirical Research (TIER), the Philosophy of Economics Working Group (INET-YSI) and the Institute for New Economics Thinking-Education Platform (INET.ED) welcome you to the world of replication supported by experienced instructors.
THE GOAL of the Webinar is to allow each participant to access lectures by replication experts and live Q&A while having a hands-on experience by replicating one study. Also, instructors can reduce their workload by using the webinar’s materials for their teaching.
REPLICATION IS IMPORTANT BECAUSE it scientifically reinforces the validity of methodologies and results. In a wide sense it can also be used to check generalizability of results using data that is updated or from different places, and it can help to check previous research using improved or alternative methods.
WE ENCOURAGE students from all levels, PostDocs, lecturers, junior professors to participate. Any social sciences background is welcome: economics, political science, sociology etc, as we will have lectures by experts from different areas who work on replication. No prior knowledge of replication is needed and you don’t need to master any specific software.
HOW CAN YOU PARTICIPATE: 3 SIMPLE STEPS
1) BEFORE THE WEBINAR: SELECT A STUDY
- Write a four-sentence summary of which study (or studies) you want to replicate and why. Add how would you like to replicate it (with the same data and code as in the original or do you want to deviate?) and specify the availability of replication material. DEADLINE FOR SUBMISSION OF YOUR REPLICATION PLAN: August 15 and send it to email@example.com
- Keep in mind: you can do this on your own or you can challenge one or more friend(s)/ colleague(s) to do it with you
2) DURING THE WEBINAR: ACTIVELY ENGAGE
- One week before each presentation at the latest a video lecture recorded by an expert will be available. LIVE Q&A sessions will take place at the Philosophy of Economics Working Group in order to discuss its content. Those who cannot make the live sessions can still contribute questions on the respective ReplicationWiki pages, also beforehand
- Participants are invited to present their own replication results both in live sessions and on the ReplicationWiki where they will be discussed and open to mutual peer review
- Additionally, to help you successfully accomplish your own replication study, all experts are available to advise you on the best way to move forward. Make the most of it!
3) AFTER THE WEBINAR: A CERTIFICATE AND A PUBLICATION
- “The International Journal of Pluralism and Economics Education” kindly offered to PUBLISH a special issue on the webinar. If you complete your replication study we offer help to publish it either in the special issue, the journal of your choice or in a book to be edited after the Webinar
- The Webinar features around 10 expert sessions plus the student presentations (September-December 2022) and if you are replicating a study, share your results and participate in at least 50% of all Q&A sessions, you will be awarded a CERTIFICATE of completion
This means that in a best-case scenario you can end the Webinar with a certificate, a replication study (prepared with the support of an expert in our research area), and, a publication.
More information about upcoming sessions already available HERE!
29 Sep 2022
S3: Replication in Quantitative Macroeconomics
WEBINAR REPLICATE REPLICATE: HOW YOU CAN TAKE PART SESSION 3: Replication in Quantitative Macroeconomics Macroeconomics differs from other fields in Economics in both the sources of data used and in the style of computation involved. In particular, data most often comes from reputable existing datasets, such as various IMF datasets, or those run by large Government organizations. On the other hand, many Macroeconomics papers involve substantial computing, often of models that have been coded specifically for that paper, rather than using standard software. As a result, there is both a substantial overlap in issues around replication in Macroeconomics and other fields, and also some important differences. We will discuss the state of replication in Macroeconomics (embryonic) and many related questions: Why is replication important? What does a successful/failed replication mean? How to find out if a paper has been replicated? How to go about performing a replication? How to make your work more easily replicable? Has the field improved over the past decade? What is most important to improve over the coming years? We will finally discuss how to do replication yourself: choosing a paper, performing the replication, useful resources, common issues, what if you need to contact the original authors?, what to do with the replication once completed? A full working paper is available here. A recorded VIDEO PRESENTATION is available to view prior to the session: https://vimeo.com/752871416 SHORT BIORobert Kirkby is a Macroeconomist at Victoria University of Wellington. He works on macroeconomic policy analysis as well as teaching macroeconomics. A substantial component of his research is the development of theoretical and computational tools to better perform such analysis.Learn more
13 Oct 2022
S4: Advice and Information on Computational Reproductions
WEBINAR REPLICATE REPLICATE: HOW YOU CAN TAKE PART SESSION 4: Advice and Information on Computational Reproductions: Students May Find Them Messier Than Expected Drawing on experience teaching and conducting replications this talk focuses on the most basic form of replication: the computational reproduction. This means simply trying to reproduce the numerical results of a previous study using the same data and statistical routines from that study. As it turns out, there are many potential pitfalls that come from both the original study and the replicator, that could lead to different results. Different computing environments or data versioning and formatting are also culprits. This talk reviews these pitfalls using examples from published literature and several attempts across disciplines to computationally reproduce findings published in journals. It also reports results from a crowdsourced replication, where eighty-five independent teams attempted a computational replication of results reported in an original study that is central to economics and many other disciplines on the links between policy preferences and immigration. Although when teams had the original data and code they were able to achieve a high rate of similar results (95.7%), a random half of teams did not get the original code and struggled (89.3%). What was more surprising was that exact numerical reproductions to the second decimal place were far less common (76.9% and 48.1%). This has wider implications for science obviously, but it should serve as a lesson for students using replication that things are not a clean cut as one might hope. (full paper on the crowdsourced study). SHORT BIONate Breznau, Postdoctoral Fellow at the University of Bremen. Researcher at the Comparative Research Center “The Global Dynamics of Social Policy”. Principal Investigator of the research project, “The Reciprocal Relationship of Public Opinion and Social Policy”. Principal Investigator of “The Crowdsourced Replication Initiative”. Open Science advocate. Open Science Fellow at Wikimedia with the project, “Giving the Results back to the Crowd”. User of preprints. Crowdsourcing and Mertonian-norm advocate. Author of the open science and crowdsourcing blog “Crowdid”.Learn more
27 Oct 2022
S5: Questionable research practices outside of the lab
WEBINAR REPLICATE REPLICATE: HOW YOU CAN TAKE PART SESSION 5: Questionable research practices outside of the lab Social scientists are increasingly interested in questionable research practices and their role in producing false and misleading results. Many scientists are prescribing open practices (e.g., open data, prospective registration) as a way to restore the credibility of academic research. But, what about the broader public significance of continued reliance on questionable research practices? What do lay people think about them? And, what happens when questionable research practices enter courtrooms and affect legal decisions? What can economists learn from this? This talk attempts to answer these questions. SHORT BIOJason Chin. I am a Lecturer at the School of Law at the University of Sydney, the former (2020-21) President of the Association for Interdisciplinary Meta-research and Open Science (AIMOS), and the inaugural registered reports editor for Forensic Science International: Synergy. I have a PhD in Social Psychology from the University of British Columbia and a JD from the University of Toronto. Prior to returning to academia, I practiced litigation at a large international law firm and was called to the bar in both New York and Ontario. Whenever feasible, I try to offer pragmatic solutions and guidance to the challenges faced by practicing lawyers. My research has been featured in the New York Times and Sydney Morning Herald.Learn more
8 Sep 2022
S1: Why replication? How is it done? Where to find replication material?
Webinar REPLICATE REPLICATE: HOW YOU CAN TAKE PART SESSION 1 discusses the basics of replication addressing: Why is the topic of replication relevant to promoting a renewed approach to Economics and other social sciences What is Philosophy of Economics, and how does it relate to replication Which purposes can it serve to replicate in the narrow sense (i.e. rerunning original studies' data and code to check results) How are results produced How to update data or how to use new data or methods to check for robustness and/or generalizability How to identify studies suitable for replication How to share results How do replications help to question published findings, detect errors, contribute to scientific progress and incentivize discussion but also how do they face resistance from authors of original studies and editors Why is replication useful and relevant to teachers and researchers, and how does the Webinar facilitate the integration of replication in your syllabus or research How can students overcome being intimidated by replication, why is it that everyone can do it, and in which way does it prepare you to produce more reliable and robust results This session includes a video-lecture available here and live Q&A. Use the discussion page to comment or ask questions in written form. SIGN UP! REPLICATION PLAN SUBMISSIONS were due on August 15, 2022. We already have a sufficient number of interesting contributions. Further ones may be accepted on a rolling basis. The webinar series may be continued after December 2022 depending on participant interest. Of course feel free to watch the videos and join the discussions also if you have not yet submitted an own replication plan. Project details and how to make the most of the Webinar here or at ReplicationWiki. Do not hesitate to address your questions to firstname.lastname@example.org or to project organizers.Learn more
15 Sep 2022
S2: Two Sides of the Replication Coin
WEBINAR REPLICATE REPLICATE: HOW YOU CAN TAKE PART SESSION 2: Two Sides of the Replication Coin: Ex Post Reproduction and Ex Ante Documentation We will begin by drawing a distinction between two distinct, but complementary, activities: ex post reproduction and ex ante documentation.Ex post reproduction refers to reproduction by an independent investigator of empirical results reported by a researcher in a previous publication. This notion of ex post reproduction corresponds to one of the types of replication that this symposium is focused on. Ex ante documentation refers to materials--data, code, and various forms of supplementary information--that are assembled by the original researchers who conduct a study, and posted publicly with the paper or report in which they present their results. This documentation should contain everything necessary to enable an interested reader to reproduce the results of the study, easily and exactly. The "replication files" that many journals now require authors of quantitative articles to submit are examples of ex ante documentation. We will then give an overview of Project TIER, an initiative that promotes incorporating instruction in transparent and reproducible methods in the research training of undergraduate and graduate students in quantitative disciplines. The focus of Project TIER has been on ex ante documentation. We will highlight the complementarities between teaching ex ante documentation and ex post reproduction. The central message: Conducting an ex post reproduction of a previously published study is excellent preparation for students who will be constructing ex ante documentation for an empirical project of their own. Constructing ex ante documentation for an empirical project of their own is excellent preparation for students who will be conducting an ex post reproduction of a previously published study. SESSION 2: Video links + slides available HERE | Live Q&A Sep 15 SHORT BIOSRichard Ball is Professor of Economics, and Norm Medeiros is Associate Librarian, both at Haverford College. Ball and Medeiros have been Co-Directors of Project TIER -Teaching Integrity in Empirical Research since they founded the initiative in 2013.Learn more
- Philosophy of Economics
Jan H. Höffler
For questions, the Project Organizers.
S5: Questionable research practices outside of the lab
October 27 2022, 20:00
WEBINAR REPLICATE REPLICATE: HOW YOU CAN TAKE PART
SESSION 5: Questionable research practices outside of the lab
Social scientists are increasingly interested in questionable research practices and their role in producing false and misleading results. Many scientists are prescribing open practices (e.g., open data, prospective registration) as a way to restore the credibility of academic research. But, what about the broader public significance of continued reliance on questionable research practices? What do lay people think about them? And, what happens when questionable research practices enter courtrooms and affect legal decisions? What can economists learn from this? This talk attempts to answer these questions.
Jason Chin. I am a Lecturer at the School of Law at the University of Sydney, the former (2020-21) President of the Association for Interdisciplinary Meta-research and Open Science (AIMOS), and the inaugural registered reports editor for Forensic Science International: Synergy. I have a PhD in Social Psychology from the University of British Columbia and a JD from the University of Toronto. Prior to returning to academia, I practiced litigation at a large international law firm and was called to the bar in both New York and Ontario. Whenever feasible, I try to offer pragmatic solutions and guidance to the challenges faced by practicing lawyers. My research has been featured in the New York Times and Sydney Morning Herald.
AttendLogin to attend
To participate, you must log in or sign up and attend this Webinar. Once you are attending you will receive a calendar invite via email, and you will receive relevant email updates related to this event.
You must attend to join.
Time & Date
Start: October 27 2022, 20:00*
Duration: 90 minutes
*Time is displayed in your local time zone (Africa/Abidjan).
Lecturer at the School of Law at the University of Sydney (AU)
Join us for a Webinar!
Topic: S5: Questionable research practices outside of the lab
Time: October 27 2022, 20:00 (Timezone: Africa/Abidjan)
Join from PC, Mac, Linux, iOS or Android: https://ysi.ineteconomics.org/project/62dddb8fa6561d550c7a9dcc/event/6310a38cbd0a0b02ba32db1e