Workshop 6: Reproducibility, replication, and perils of pass-fail science

The 6th Workshop on Quantitative Methods in Education, Health and the Social Sciences (QMEHSS) will be on Friday April 4th from 11:00-12:30 and will be led by Dr. Ronald Thisted. The seminar will be held in the NORC conference room 344. NORC is located at 1155 E. 60th Street.

Reproducibility, replication, and perils of pass-fail science

Ronald Thisted

Departments of Health Studies and Statistics

In 2005, Ioannidis published a calculation leading to the now-famous conclusion that “most published research findings are false,” followed by claims (2008) that “most discovered true associations are inflated,” and that statistically significant results from low-powered studies are unlikely to be real (Button, et al., 2013).  Ioannidis and Trikalinos (2007) proposed a screening test for bias in individual publications that report on multiple experiments.  Francis (2014) has adapted this procedure, dubbed a “test of excess significance”, applying it to 44 articles published in Psychological Science.  Francis concluded that 36 of the 44 are, in the language of an earlier paper (Francis, 2012), “too good to be true.”   Against this backdrop, I will examine assumptions on which these analyses rely, their implications for a theory of replication, and consequences for scientific practice.