Transparency and Data Sharing in Epidemiology

HESI's Environmental Epidemiology Committee hosted a joint presentation by Dr. Tim Lash (Rollins School of Public Health, Emory University) and Dr. Brian Nosek (Center for Open Science) on 29 June 2022 1:00-2:30 pm EST.

Link to Recording: https://youtu.be/iobUe_GYqUE

Bio: Dr. Tim Lash is the O. Wayne Rollins Distinguished Professor of Epidemiology and Chair of the Department of Epidemiology at Emory University’s Rollins School of Public Health. His research focuses on predictive and prognostic markers of breast, prostate, and colorectal cancer recurrence. Dr. Lash’s methodological interest focuses on developing and implementing methods to quantify the influence of systematic errors on epidemiologic research. Funding from the National Library of Medicine supports his work to develop methods that quantify the influence of systematic errors on the reproducibility of epidemiologic study results. He is Editor-in-Chief of EPIDEMIOLOGY, a leading general interest epidemiology journal, and coauthor of two epidemiology textbooks: Applying Quantitative Bias Analysis to Epidemiologic Research, 2nd edition and Modern Epidemiology, 4th edition.

Title: Reproducibility in Epidemiologic Research: History, Hysteria, and Hypothesis Testing

Abstract: In the last years, some in the scientific community have raised alarms about a perceived lack of reproducibility of scientific results. In reaction, guidelines for journals have been promulgated and grant applicants have been asked to address the rigor and reproducibility of their proposed projects. Neither solution addresses a root cause, which is the culture of null hypothesis significance testing that dominates statistical analysis and inference. In an innovative research enterprise, selection of results for further evaluation based on null hypothesis significance testing is doomed to yield a low proportion of reproducible results and a high proportion of effects that are initially overestimated. Compulsory pre-registration of non-randomized research projects has often been suggested to improve reproducibility, but this alternative fails to address the root cause and ignores substantial evidence that compulsory pre-registration of randomized trials has not yielded high compliance with pre-ordained protocols. Most importantly, epidemiologic research has long-been criticized for generating perceived false-positive associations, many of which have ultimately proved to have public health importance. This history should inform the current perceptions of poor reproducibility.

[1] Lash TL. The Harm Done to Reproducibility by the Culture of Null Hypothesis Significance Testing. Am J Epidemiol. 2017 Sep 15;186(6):627-635. doi: 10.1093/aje/kwx261. PMID: 28938715. https://academic.oup.com/aje/article/186/6/627/4091005?login=true

---------------------------------------------------------------------------------------------

Bio: Dr. Brian Nosek co-developed the Implicit Association Test, a method that advanced research and public interest in implicit bias. Nosek co-founded three non-profit organizations: Project Implicit to advance research and education about implicit bias (http://projectimplicit.net/), the Society for the Improvement of Psychological Science to improve the research culture in his home discipline (http://improvingpsych.org/), and the Center for Open Science (COS; http://cos.io/) to improve rigor, transparency, integrity, and reproducibility across research disciplines. Nosek is Executive Director of COS and a professor at the University of Virginia. Nosek’s research and applied interests are to understand why people and systems produce behaviors that are contrary to intentions and values; to develop, implement, and evaluate solutions to align practices with values; and, to improve research credibility and cultures to accelerate progress.

Title: Culture change toward more open, rigorous, and reproducible research

Abstract: Improving openness, rigor, and reproducibility in research is less a technical challenge and more a social challenge. Current practice is sustained by dysfunctional incentives that prioritizes publication over accuracy and transparency. The consequence is unnecessary friction in research progress. Successful culture change requires coordinated policy, incentive, and normative changes across stakeholders to improve research credibility and accelerate progress. Some stakeholder groups and disciplines are making more progress than others. We can change the system, but if we do not act collectively we will fail. Let’s not fail.

[1] Talk: https://osf.io/yzskd
[2] Center for Open Science (COS): http://cos.io/
[3] Nosek testimony on proposed EPA rule to House committee: https://science.house.gov/imo/media/doc/Nosek%20Testimony.pdf
[4] Redefine statistical significance paper: https://www.nature.com/articles/s41562-017-0189-z
[5] Observational study of Registered Reports: https://www.nature.com/articles/s41562-021-01142-4
[6] Example of using preregistration to get potential antagonists to agree in advance on methods and outcomes of interest: https://www.nature.com/articles/d41586-020-02142-6