Statistical Power

In brief

Statistical power is the probability that a study will detect an effect that is really present. It is the flip side of the Type II error, the false negative: if power is 0.5, then even when the effect exists, half of studies looking for it will miss. Power depends on three things a researcher can reason about before collecting any data: how large the true effect is, how much the measurements vary, and above all how many observations are gathered. The concept comes from the Neyman-Pearson framework of the 1930s, but its most consequential champion was Jacob Cohen, who spent decades pointing out that psychology largely ignored it. When a field runs on underpowered studies, the damage is double. It misses real effects, and, less obviously, the significant results it does publish are inflated and fragile. Power is now understood to be one of the engines of the replication crisis.

The full treatment

The two ways to be wrong

Every study that tests a hypothesis can err in two directions, a distinction sharpened by Jerzy Neyman (1894 to 1981) and Egon Pearson (1895 to 1980) in a series of papers culminating in 1933. A Type I error is a false positive: concluding there is an effect when there is none. Its rate is the significance level, conventionally set at 0.05. A Type II error is a false negative: concluding there is no effect when one is really there. Its rate is written as beta. Power is simply one minus beta, the probability of correctly detecting a true effect. The two error types trade against each other, but they are not symmetric in how the culture treats them. Journals, referees, and researchers have long policed Type I errors with the 0.05 threshold while leaving Type II errors, and therefore power, almost unexamined. A study can be designed with great care about the 5 percent and no thought at all about whether it has any real chance of finding what it seeks.

What power depends on

Three levers set the power of a test. The first is the true effect size: a large difference is easy to see, a subtle one is easy to miss. The second is measurement noise: the more variable the data, the harder any signal is to pull out. The third, and the only one usually under the researcher's direct control, is the sample size. Bigger samples average out noise and make even small true effects detectable. This is why power analysis, done properly, happens before data collection: you specify the smallest effect worth caring about and the power you want (0.80 is the usual target), and you solve for the number of subjects required. Skip that step and you are gambling that your sample happened to be large enough, without ever having checked the odds.

Cohen's indictment

The person who forced psychology to look at this was Jacob Cohen (1923 to 1998). In 1962 he surveyed a full volume of the Journal of Abnormal and Social Psychology and computed the power of the studies in it. For a medium-sized effect, the average study had roughly a 50 percent chance of detecting it: a coin flip. For small effects the odds were far worse. His point was withering: the field was running experiments that, even when their hypotheses were correct, would fail about half the time, and then interpreting the failures as evidence of no effect. He returned to the theme for the rest of his career, most fully in Statistical Power Analysis for the Behavioral Sciences (1969, second edition 1988), which supplied the tables and the now-standard small, medium, and large effect-size conventions, and in a 1992 article he pointedly titled "A Power Primer." Decades of exhortation moved the field slowly. Reviews long after his first survey kept finding average power stuck near the same low value.

The winner's curse

The subtle half of the story is that low power does not only cause misses. It corrupts the hits. Suppose an underpowered study, against the odds, does clear the 0.05 bar. To have reached significance with a small sample, the observed effect must have been unusually large, larger than the truth. So the studies that get published are precisely the ones that overstate the effect, a selection effect statisticians call the "winner's curse." Andrew Gelman and John Carlin formalized this in 2014 as the Type M (magnitude) error, the factor by which a significant estimate exaggerates the real effect, alongside the Type S (sign) error, the chance it even points the wrong way. In a badly underpowered design, a published, significant, seemingly clean result can be inflated several times over, and the next lab has little chance of reproducing it. This is the mechanism, not fraud and not bad luck alone, that makes a literature full of exciting findings quietly unreplicable.

A distinction worth keeping

Power is about detection before the fact, not interpretation after it. "Post hoc power," computed from the effect actually observed in a completed study, is a known confusion: it adds nothing beyond the p-value and can mislead. And a non-significant result from an underpowered study is not evidence of absence, only absence of evidence. To claim a true null, you need a design that had real power to find an effect if one were there.

Lineage

Power is inseparable from the Neyman-Pearson approach to hypothesis testing, which framed testing as a decision between two hypotheses with two error rates to be balanced. This was a deliberate break from Ronald Fisher (1890 to 1962), whose significance testing centered on the p-value alone and who never gave power a comparable role; the hybrid taught in most textbooks awkwardly welds the two traditions together. The idea sits close to signal detection theory, which likewise separates a real signal from noise and treats a "hit" and a "miss" as two distinct outcomes. Cohen carried the framework from mathematical statistics into everyday research practice, translating beta and power into effect-size rules of thumb that a working psychologist could apply.

The strongest case for it

Power analysis is the rare methodological reform that is both principled and practical. It converts a vague hope ("I ran a study and hope to find something") into an explicit, checkable claim about the odds of success, made before any data exist and therefore immune to hindsight. It protects against the most demoralizing waste in science: running an experiment that could not have detected its target even in principle. It makes null results meaningful only when a design had a genuine chance to fail the null. And it exposes the hidden cost of small studies, showing that they are not merely weaker versions of large ones but actively misleading, because their surviving positives are inflated. As a diagnostic for a whole literature it has proven accurate: fields with low average power have turned out, on later inspection, to be exactly the fields whose findings do not replicate.

The strongest case against it

The concept is not seriously disputed, but its standard implementation draws real criticism. The most common charge is that power analysis, as practiced, rests on a number nobody knows: the true effect size. Researchers routinely plug in an effect estimated from a small pilot study or a published paper, and both are inflated by the very winner's curse power is meant to guard against. The result is a calculation that certifies a sample far too small, dressing a guess in the authority of arithmetic. Critics call this "sample size samba," reverse-engineering the desired subject count. A second line of attack targets the whole Type I/Type II machinery. Advocates of estimation, such as Geoff Cumming, argue that fixating on a binary "detect or not" decision, and on power as the probability of clearing a threshold, is the wrong frame entirely; the useful questions are how big the effect is and how precise the estimate, answered with confidence intervals rather than a pass-fail verdict. Bayesians press further, denying that the fixed error rates of the Neyman-Pearson scheme are the right currency for scientific belief at all. Even sympathetic reformers note that "aim for 0.80 power" can harden into ritual, another arbitrary threshold beside the arbitrary 0.05, invoked without thought about the effect size that would actually matter. The reply from power's defenders is that these critiques argue for doing power reasoning better, with honest, conservative effect sizes and attention to precision, rather than abandoning the underlying insight that a study too small to see its quarry should not be run.

Where it stands now

Power moved from a specialist's footnote to the center of methodological reform. The turning point was the reproducibility upheaval of the 2010s. John Ioannidis's 2005 essay "Why Most Published Research Findings Are False" listed low power among the structural reasons a positive result is likely wrong. In 2013 Katherine Button and colleagues, surveying neuroscience in Nature Reviews Neuroscience under the title "Power Failure," estimated the median study's power at around 20 percent, an astonishingly low figure that made Cohen's decades-old warning concrete. Two years later the Open Science Collaboration's large replication project found that a majority of tested psychology results did not reproduce, and underpowered original studies were among the usual suspects. Registered reports, preregistration, and journal policies now often demand a power analysis or a target sample size in advance, and adequately powered replications and "many-labs" collaborations have become a recognized form of research. The lesson Cohen preached to a field that would not listen has, at last, been learned.

Test yourself

Think of a striking finding you have believed on the strength of a single study, in the news or in your own field. Ask a question you were probably never given the means to ask: how many people were in it, and was that enough to have found the effect if it were real? A dramatic result from a tiny sample is not strong evidence made vivid. It is weak evidence that, precisely because it is weak, had to look dramatic to get published at all.