The Replication Crisis

In brief

In 2015, a team of 270 researchers organized as the Open Science Collaboration published the results of an effort to redo 100 studies from three leading psychology journals, using the original materials wherever possible. Ninety-seven percent of the original studies had reported a statistically significant effect. Only 36 percent of the replications did, and the effects that reappeared were, on average, about half the size originally reported. The finding gave a name and a number to something researchers had quietly worried about for years: that a meaningful share of the field's best-known results might not be real, or might be far weaker than claimed. The causes were not mostly fraud, but a set of ordinary practices, small samples, flexible analysis, and a bias toward publishing only the exciting result, that had been treated as normal.

The full treatment

The proximate trigger

The crisis had a specific spark. In 2011, the Journal of Personality and Social Psychology published a paper by Daryl Bem titled "Feeling the Future," reporting nine experiments that appeared to show undergraduates could sense future events, such as which side of a screen an image would randomly appear on, before it happened. Bem used entirely conventional methods and the paper passed peer review at a top journal. If standard tools could produce apparent evidence for precognition, something was wrong with the tools, not just with one researcher. A Bayesian reanalysis by Eric-Jan Wagenmakers and colleagues, also in 2011, found Bem's own statistics, run differently, gave far weaker support, and later replication attempts largely failed.

The causes

Four mechanisms, working together, account for most of the crisis.

Small samples produce unstable results. Jacob Cohen warned as early as 1962 that the typical psychology study was underpowered, with a poor chance of detecting a real effect even when one existed. An underpowered study that still reports a significant finding has usually overestimated the true effect, the "winner's curse": the studies most likely to get published are the ones luck inflated the most, and inflated results are the least likely to reappear on a second try.

Publication bias compounds this. Journals have long favored novel, positive results over null ones, a pattern Robert Rosenthal named the "file drawer problem" in 1979: an unknown number of null results sit unpublished for every finding that reaches print, so the visible literature is a biased sample of what was actually run.

Researcher degrees of freedom turn the incentive into practice. In 2011, Joseph Simmons, Leif Nelson, and Uri Simonsohn published "False-Positive Psychology," showing with real data collection that a researcher choosing, after seeing the data, which measures to report and when to stop collecting subjects, could push almost any analysis to statistical significance, including for hypotheses later shown false. None of these choices required dishonesty; each looked, alone, like a defensible judgment call. A related practice, hypothesizing after results are known, or HARKing, a term coined by Norbert Kerr in 1998, let researchers present a post hoc finding as though predicted in advance. A 2012 anonymous survey by Leslie John, George Loewenstein, and Drazen Prelec found large majorities of psychologists admitting to at least one such practice.

The Open Science Collaboration's 2015 study

The Reproducibility Project: Psychology, coordinated by Brian Nosek and the Center for Open Science, measured the scale of the problem directly. Teams around the world selected 100 studies published in 2008 in Psychological Science, the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: Learning, Memory, and Cognition, and tried to reproduce each one as closely as possible, often with the original authors' help. Beyond the headline numbers, effect sizes shrank across the board, and the studies most likely to fail were those with weaker initial evidence and smaller samples, exactly what the power and publication-bias arguments predicted.

Individual cases became emblematic. Dana Carney, Amy Cuddy, and Andy Yap's 2010 finding that briefly holding an expansive "power pose" shifted hormones and risk-taking failed to replicate in a large 2015 study by Eva Ranehill and colleagues, and Carney, one of the original authors, published a statement in 2016 saying she no longer believed the effect was real.

Related distinctions

The crisis is not mainly a story about fraud, though fraud exists at its margins: the Dutch social psychologist Diederik Stapel was found in 2011 to have fabricated data in dozens of papers, a scandal that broke around the same time. Almost every study caught up in the replication crisis involved no fabrication, only the ordinary flexibility Simmons, Nelson, and Simonsohn described. A second distinction matters as much: a failed replication does not prove an original effect is false. It can mean the original was a false positive, that the true effect is smaller or more context-dependent than claimed, or that the replication missed a hidden moderator.

The reforms

Nosek, Jeffrey Spies, and Matt Motyl laid out the reform program in a 2012 paper, "Scientific Utopia II," arguing that psychology needed incentives that rewarded getting the answer right over getting a publishable answer fast. Pre-registration is the central tool: a researcher commits, before collecting data, to the hypothesis, sample size, and analysis plan, closing off the flexibility the crisis exposed, and the Open Science Framework gave the field free infrastructure to host these commitments. Registered reports go further: a journal accepts or rejects a paper on peer review of the introduction and methods alone, before results exist, and commits to publish regardless of outcome. Chris Chambers introduced the format at the journal Cortex in 2013, and it has since spread to several hundred journals. The Many Labs projects, led by Richard Klein, now test classic findings across dozens of laboratories at once.

Lineage

The statistical machinery behind the crisis traces to Ronald Fisher's popularization of the p < .05 threshold in the 1920s, a convention never meant to certify truth but widely treated as though it did. Paul Meehl argued as early as 1967 that psychology's habit of testing weak theories against weak alternatives let almost any plausible hypothesis find some support, a critique that anticipated the crisis by decades. John Ioannidis generalized the same logic across biomedicine in his 2005 paper "Why Most Published Research Findings Are False," arguing mathematically that low power, small effects, and bias toward positive results make most published findings, in fields with those traits, more likely false than true.

The strongest case for it

The diagnosis has held up under scrutiny rare in the social sciences. The pattern the Open Science Collaboration found, weaker original evidence and smaller samples predicting failure to replicate, is exactly what the power and publication-bias arguments predict. Later efforts reinforced rather than reversed it: Many Labs 2, led by Klein and reported in 2018, tested 28 classic effects across more than 60 samples worldwide and found the same uneven pattern, some effects replicating everywhere, others nowhere. The crisis also produced measurable reform: registered reports now show a far higher rate of null results reaching publication than the traditional literature ever did, direct evidence the old system was suppressing exactly what theory said it would.

The strongest case against it

The Open Science Collaboration's own numbers were disputed by credentialed critics, not fringe ones. In a 2016 comment in Science, Daniel Gilbert, Gary King, Stephen Pettigrew, and Timothy Wilson argued that a number of the replications departed enough from the originals in population or procedure that a failure to replicate said more about a changed test than about the original finding's truth. They also argued the project's own confidence intervals were wide enough to be consistent with a world in which most original effects were genuine. Nosek and his coauthors published a rebuttal, and the exchange remains a live dispute.

A second line of criticism came from within social psychology. In a widely discussed 2014 essay, the Harvard psychologist Jason Mitchell argued that many failed replications reflect a replicator's lack of the tacit skill needed to run subtle paradigms correctly, not a flaw in the original work, and that treating a null result as automatically more credible than a positive one is its own bias. Critics countered that this argument, taken seriously, would make findings unfalsifiable, since any failed replication could always be blamed on the replicator.

Where it stands now

The core empirical picture from 2015 has not been overturned by later, larger efforts, and the open-science reforms it triggered, pre-registration, registered reports, and open data, have moved from fringe advocacy to mainstream editorial policy at journals including Psychological Science itself. What remains contested is not whether the crisis was real, but how far its lessons generalize: whether specific effects are simply false or context-dependent in ways science has not yet mapped, and whether the same structural problems are equally severe in neighboring fields that have faced less scrutiny.

Test yourself

Think of a psychology finding you have repeated to someone else as settled fact, power posing, priming, or another lab result absorbed from a book or a talk. Ask when you last checked whether it survived a large, pre-registered replication, and notice how much of what you believe about the mind rests on a single early study you have never questioned.