Statistical Significance

In brief

Run an experiment, collect data, and you face a question: could this pattern be a fluke? Significance testing answers with a single number. You state a null hypothesis, usually "there is no effect," then ask how likely it would be to see data at least as extreme as yours if that null were true. That likelihood is the p-value. If it is small enough, by convention below 0.05, you declare the result "statistically significant" and reject the null. The machinery is everywhere: t-tests, ANOVA, and chi-square tests all end in a p-value. And "significant" is among the most misread words in science. It does not mean the finding is true, large, or important. It means only that data this surprising would be uncommon in a world where nothing was happening.

The full treatment

The problem it answers

Data are noisy. Two groups will almost never post identical averages even when drawn from the same population, because sampling introduces random scatter. So any observed difference could be a real effect or chance dressed up as signal. Researchers needed a disciplined way to decide when a difference is too large to shrug off as noise. Ronald Fisher (1890 to 1962), working at Rothamsted agricultural station in the 1920s, offered one. Assume there is no effect. Compute how improbable your data would be under that assumption. If the data are improbable enough, treat the no-effect assumption as untenable. The p-value is the engine of that inference.

What a p-value actually is

Formally, the p-value is the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is true. Read that carefully, because every clause matters. It is a probability about the data, on the supposition that the null holds. It is not a probability about the hypothesis. A p-value of 0.03 does not mean there is a 3 percent chance the null is true, nor a 97 percent chance your theory is right. Those are statements about hypotheses given data; the p-value is a statement about data given a hypothesis, and the two cannot be swapped without committing what statisticians call the inverse fallacy. To get the probability that a hypothesis is true, you would need a prior and Bayes's rule (see bayesian-updating), which classical testing deliberately refuses to supply.

Three further things a p-value is not, each a routine source of error. It is not the effect size: a trivially small difference can be highly significant if the sample is large enough, because significance rises with sample size regardless of how big the effect is. It is not a measure of importance: a drug that lowers blood pressure by an amount no patient would notice can clear p < 0.001 in a large trial. And it is not the probability that your result will replicate. A single low p-value from a single study is a weak, noisy signal, far weaker than most researchers, let alone most readers, assume.

Where 0.05 came from

The famous threshold is a convention, not a law of nature. In Statistical Methods for Research Workers (1925), Fisher wrote that it is convenient to draw the line at 0.05, a result being significant when it would occur by chance less than one time in twenty. He meant it as a rough working guide, a signal that a result was worth a second look, not a bright line between truth and falsehood, and he expected researchers to weigh p-values in context and against repeated experiments. The number stuck anyway, hardened by textbooks and journal editors into a pass or fail gate that Fisher never intended.

Two frameworks fused into one

What most people call significance testing is a hybrid of two rival systems that their creators considered incompatible. Fisher's approach treated the p-value as continuous evidence against the null, to be interpreted by judgment. Jerzy Neyman (1894 to 1981) and Egon Pearson (1895 to 1980), in work from 1928 and 1933, built a different structure aimed at decisions rather than evidence: fix in advance an acceptable rate of false alarms (alpha, the Type I error rate) and of missed effects (beta, the Type II error rate, whose complement is statistical power), then follow a rule that controls those long-run error rates. Fisher and Neyman feuded bitterly over which was right. Textbooks quietly merged them, reporting an exact p-value (Fisher) while comparing it to a fixed 0.05 cutoff and talking of rejecting hypotheses (Neyman-Pearson). The incoherence of this fusion is one root of the confusion around the whole enterprise.

Statistical versus practical significance

The single most useful distinction the concept forces is between statistical and practical significance. Statistical significance asks whether an effect is distinguishable from zero given the noise. Practical significance asks whether the effect is large enough to matter. These come apart constantly. Increase the sample and even a negligible effect becomes statistically significant, because the test grows more able to detect ever-smaller departures from the null. This is why the effect size, how big the difference is, and the confidence interval, the range of effects the data are compatible with, carry information a lone p-value cannot. "Significant" in ordinary English means important. In statistics it means detectable. Conflating the two is the error built into the word.

Lineage

Significance testing sits atop the older machinery of probabilistic thinking and the normal distribution, which supply the sampling distributions against which "extreme" is measured. Its immediate ancestors are Fisher, who gave it the p-value and the 0.05 convention along with analysis of variance, and Neyman and Pearson, who gave it error rates and power. It descends more distantly from Karl Pearson (1857 to 1936), whose chi-square test of 1900 was an early significance test, and from William Sealy Gosset (1876 to 1937), the Guinness brewer who published the t-distribution in 1908 under the pen name "Student." The whole apparatus is the frequentist tributary of statistics, defined by its refusal to assign probabilities to hypotheses, the move that separates it from the Bayesian tradition of bayesian-updating.

The strongest case for it

Significance testing endures because it does real work. It offers a common, portable standard: a p-value is computed the same way across fields and read by anyone who knows the rules, which lets strangers evaluate each other's claims without sharing priors. It provides an explicit, if crude, guard against fooling yourself with noise, forcing the researcher to ask before celebrating a pattern whether chance alone could have produced it. Fisher's original use, screening many agricultural treatments to decide which deserved further study, is genuinely sound: as a filter that flags results worth pursuing, the p-value is cheap and effective. In the Neyman-Pearson framing, controlling long-run error rates is exactly what you want when you must make repeated automated decisions, as in industrial quality control. The tool is not broken. It is a decision aid that has been mistaken for a truth machine.

The strongest case against it

The case against is old, deep, and largely conceded even by defenders; the disputes are over what to do, not whether there is a problem.

The oldest charge is that significance testing answers a question no one is asking. Jacob Cohen (1923 to 1998), in "The Earth Is Round (p < .05)" (1994), argued that researchers want to know the probability their hypothesis is true given the data, while the p-value gives the probability of the data given the null, and that the ritual persists because people misread the second as the first. Cohen had spent decades documenting how underpowered psychology studies were and how routinely the logic was inverted.

A second charge concerns incentives. John Ioannidis, in "Why Most Published Research Findings Are False" (2005), showed formally that when studies are small, effects are modest, and researchers have many analytic choices, the majority of statistically significant published findings can be false positives. The 0.05 gate, by rewarding significance with publication, breeds a literature dredged for chance results. This connects directly to p-hacking-and-harking, tweaking analyses until p drops below 0.05 and inventing hypotheses after seeing the data, and to the-replication-crisis these pressures helped cause.

The critique went institutional. In 2015 the journal Basic and Applied Social Psychology banned p-values from its pages outright. In 2016 the American Statistical Association issued the first formal statement in its history on a statistical method, warning in six principles that p-values do not measure the probability a hypothesis is true, that decisions should not be based on whether a p-value crosses a threshold, and that a p-value alone does not measure the size or importance of an effect. In 2019 a Nature comment by Valentin Amrhein, Sander Greenland, and Blake McShane, co-signed by more than 800 scientists, called to "retire statistical significance," abolishing the dichotomy between significant and non-significant and treating p-values as one continuous piece of evidence among many.

The defenders of testing, statisticians such as Deborah Mayo, reply that the fault lies in ritual and misuse, not in the logic, and that a properly understood severe test remains a cornerstone of honest inquiry. This is the significance-testing war, and it is unresolved. What almost no one now defends is the reflex that a p just below 0.05 certifies a discovery and one just above it certifies nothing.

Where it stands now

The p-value survives, chastened. It is still the default output of nearly every statistical package and the currency of most empirical journals, and no rival has replaced it wholesale. But the surrounding norms have shifted. Reporting effect sizes and confidence intervals alongside p-values is now expected in many fields. Preregistration, which fixes hypotheses and analyses before data collection, blunts the freedom that made significance easy to manufacture. Some methodologists push toward Bayesian estimation, others toward simply describing effects and their uncertainty without any threshold at all. A serious proposal by a large group of statisticians in 2018 would lower the default cutoff for new discoveries to 0.005. The most durable lesson of a century of argument is the one Fisher already half-knew: a single p-value is a modest, local, easily abused signal, and treating "significant" as a synonym for "true" or "important" is a category error the word itself invites.

Test yourself

Think of a headline you believed because a study "found a significant effect." Ask three questions the p-value cannot answer on its own. How big was the effect, in units you can feel? How wide was the range of effects the data were actually compatible with? And has anyone reproduced it? If you cannot answer these, you learned that the result was unlikely under the assumption of no effect, and nearly nothing else. That is less than it sounded like, and noticing the gap is the whole point.