P-Hacking and HARKing

In brief

Null hypothesis significance testing works on one condition: the test you run and report is the test you decided on before you saw the data. Break that condition and the printed p-value stops meaning what everyone reads it to mean. P-hacking breaks it on the analysis side, trying combinations of measures, exclusions, subgroups, and stopping rules until one clears the 0.05 line. HARKing, a term coined by the social psychologist Norbert Kerr in 1998, breaks it on the theory side, writing up a hypothesis discovered by looking at the results as though it had been predicted in advance. Neither requires fabricating a single number. Both convert an exploratory hunt into a false claim of confirmed prediction, and both were, for decades, ordinary parts of how research got done.

The full treatment

The logic they both undermine

A p-value answers a narrow question: if the null hypothesis were true and this exact test were run once, how often would data this extreme or more turn up by chance alone. The conventional 0.05 threshold accepts that one time in twenty, a real null effect will look significant anyway. That arithmetic holds only if one test corresponds to one reported result. Run twenty tests on the same question and report just the one that hit, and you have not found a five percent fluke, you have all but guaranteed one. Multiple-comparison correction exists precisely to fix this when the multiplicity is visible on the page, but p-hacking and HARKing hide the multiplicity. A paper built this way shows a single clean test and a single clean hypothesis, with no trace of the branches that were tried and discarded, so the reader has no way to apply a correction that the author never disclosed needing.

How p-hacking works

The term covers a family of ordinary-looking choices, each defensible on its own, that become dangerous in combination: measuring several outcomes and reporting only the one that reached significance, checking results after every few participants and stopping once the line is crossed, adding or dropping covariates and outliers until the result appears, or breaking the sample into subgroups and reporting the one that worked. Simmons, Nelson, and Simonsohn's 2011 paper named this bundle "researcher degrees of freedom" and simulated their effect: combining four common, individually reasonable choices could push the true false-positive rate to 60.7 percent, more than twelve times the nominal 5 percent. They then demonstrated it directly. Undergraduates listened to either a Beatles song, "When I'm Sixty-Four," or a neutral control track. Using father's age as a covariate, one of several defensible choices available after the fact, the researchers found that listeners of the Beatles song were, on average, statistically younger than the control group (20.1 years versus 21.5, p equals .040), an impossible finding produced entirely by flexible analysis of a real dataset with no fabricated number anywhere in it.

How HARKing works

Kerr's 1998 paper gave the practice its name and taxonomy. A researcher runs an exploratory study, notices an unpredicted but significant pattern, and writes the introduction as though that pattern had been the hypothesis all along, deleting any mention of the hypotheses that did not pan out. Kerr distinguished several varieties, including retrieving an existing theory after the fact and presenting it as the study's original motivation, and suppressing failed hypotheses to hide how many were tested. The statistical damage mirrors p-hacking's: if a study quietly tests many candidate patterns and reports whichever survives as the predicted one, the reported p-value describes the chance of that one pattern arising by luck, not the chance that at least one of many would, which is far higher.

Related distinctions

None of this indicts looking at data before forming ideas. Exploratory analysis is one of science's genuine engines, and labeled honestly as exploratory rather than confirmatory, it remains legitimate and valuable. The offense is not exploring, it is dressing the outcome of exploration as a prediction that was never made. Correcting for multiple comparisons when the comparisons are disclosed is likewise standard, sound practice, not p-hacking. The line is disclosure: a test reported as one of many is honest science; the same test, laundered to look like the only one, is not.

Lineage

The 1933 Neyman-Pearson framework that gave significance testing its error-rate logic assumed a single pre-specified test, and Paul Meehl's 1967 critique of soft psychology had already warned that loosely specified theories let almost any data count as support. Robert Rosenthal's 1979 "file drawer problem" described a sibling failure, whole null studies vanishing from the literature rather than single analyses being reshaped inside one. Kerr named the theory-side version in 1998, and Simmons, Nelson, and Simonsohn gave the analysis-side version its most cited demonstration and its name in 2011. Head and colleagues extended the case in 2015, text-mining p-values from papers across many fields and finding an excess clustered just under 0.05, a fingerprint consistent with widespread hacking. These papers fed directly into the open-science reform movement of the 2010s: the Center for Open Science, founded by Brian Nosek and Jeffrey Spies in 2013, and Chris Chambers's registered-reports format, introduced at the journal Cortex the same year.

The strongest case for it

Treating every instance as deliberate fraud misreads how it usually happens. Gelman and Loken's 2014 "garden of forking paths" argument makes the sharpest version of this case: a researcher can arrive at an inflated false-positive rate without running a single extra test, purely because the one analysis chosen was, without anyone noticing, selected from among several equally defensible options that different data would have called for. On this reading, the vulnerability is built into the ordinary, unsupervised practice of data analysis, not into the character of the people doing it, a more accurate diagnosis than a story about cheaters, and one that points toward transparency and pre-specification rather than blame as the fix. On the HARKing side, Kerr himself did not argue that noticing unexpected patterns in data is illegitimate. Generating hypotheses from surprising results, so long as the paper says so, is ordinary and valuable abductive reasoning, the same process behind most scientific insight outside the tidy textbook sequence of hypothesis, then test.

The strongest case against it

The case against is not merely theoretical. Ioannidis's 2005 paper "Why Most Published Research Findings Are False" argued that in fields marked by small effects, many tested relationships, and exactly this kind of analytic flexibility, the majority of published positive claims fail to hold up. The clearest confirmation came a decade later: the Open Science Collaboration's 2015 replication of 100 psychology studies found a significant effect in roughly one third of the direct replications, against 97 percent in the original papers. Flexible analysis and undisclosed hypothesis-shopping are not the whole explanation for that gap, but they are a central strand of it. The deeper harm is specific to HARKing: a literature full of "predicted" findings looks more theoretically confirmed than the data underneath it justify, which misleads other scientists building on the work, and misleads journalists, clinicians, and policymakers who reasonably treat a confirmed prediction as stronger evidence than an accidental pattern dressed up to look like one. Time, funding, and occasionally patient welfare in medicine ride on results that never had the evidential weight they were reported with.

Where it stands now

The response has been structural rather than moral. Preregistration, filing a hypothesis and analysis plan on a timestamped public registry such as the Open Science Framework before the data are collected, closes off HARKing by fixing the prediction in advance. Registered reports, peer-reviewed and accepted on the strength of the question and method before the results exist, remove the incentive for both practices by guaranteeing publication regardless of outcome. Simmons, Nelson, and Simonsohn followed their 2011 paper with a short 2012 disclosure norm, since adopted by numerous journals, requiring authors to report how sample size was set and to list all measures, conditions, and exclusions. The same trio built p-curve analysis in 2014, a tool for detecting the statistical signature of hacking across a set of published studies after the fact. None of this has eliminated the underlying incentive: publish-or-perish pressure still rewards clean, surprising results, adoption of preregistration remains partial outside psychology, and enforcement varies by journal.

Test yourself

Think of a time you kept cutting a set of numbers, a budget, a fitness tracker, a set of quarterly results, a comparison between two options, until one cut told the story you wanted to tell, and you presented that cut as if it were the obvious one all along. Ask whether you would have shown the same confidence in the conclusion if you had written down your prediction before you started slicing.