Reliability and Validity

In brief

Psychology tries to measure things no one can see: intelligence, anxiety, extraversion, prejudice. None is observed directly. Each is a latent construct, inferred from behavior we can observe, such as answers on a questionnaire or reaction times on a screen. Turning an invisible trait into a number is called operationalization, and it raises two separate questions. First, is the number stable? If you measured the same person again, on a different day or with a different item set or a different rater, would you get roughly the same value? That is reliability. Second, does the number mean what its name promises? A depression inventory that reliably captures how tired someone was that week is consistent but wrong. That is validity. The distinction has a sharp asymmetry: a measure can be highly reliable and still measure the wrong thing, but a measure that is not reliable cannot be valid, because a number that will not sit still cannot faithfully track anything.

The full treatment

The problem of the unobservable trait

The founding difficulty of psychometrics is that its objects do not exist as physical quantities waiting to be read off an instrument. A thermometer measures temperature, which has an independent definition; a personality inventory measures conscientiousness, which is defined largely by the inventory itself. Charles Spearman, working in London from 1904, noticed that a person's scores across unrelated mental tasks tended to correlate, and he proposed an underlying general factor he called g to explain the pattern (see theories-of-intelligence). But g is not seen. It is posited to account for observed correlations, and every latent construct carries this same status. The measurement problem is therefore not merely technical. It is the problem of justifying the claim that a pattern of visible responses is evidence for an invisible cause.

Reliability: the consistency axis

Reliability is repeatability, and it comes in several forms because a score can vary along several axes. Test-retest reliability asks whether the same test given to the same people on two occasions yields stable scores; a trait measure should show this, though a mood measure should not, since mood genuinely changes. Inter-rater reliability asks whether two independent judges scoring the same behavior agree, which matters wherever a human codes an open response or rates a clinical interview. Internal consistency asks whether the items within a single test hang together, whether they seem to be tapping one thing. Its most reported index is coefficient alpha, introduced in its general form by Lee Cronbach in 1951, which roughly captures how strongly a set of items intercorrelate. Underlying all of these is classical test theory's simple model: an observed score is the sum of a true score and random error, and reliability is the proportion of score variance that is true rather than error. High reliability means the error term is small. It says nothing about whether the true score is the true score of the right thing.

Validity: the meaning axis

Validity is the case that a test measures what it claims. Three traditional facets are still taught. Content validity asks whether the items cover the domain: a mathematics exam that omits geometry has a content gap. Criterion validity asks whether scores relate to an outcome they should predict, such as whether an aptitude test forecasts later job or school performance; this splits into concurrent validity (agreement with a criterion measured now) and predictive validity (forecasting a future one). Construct validity, the deepest, asks whether the test tracks the theoretical trait itself. Lee Cronbach and Paul Meehl set out the idea in 1955, arguing that a construct is defined by its place in a nomological network, the web of lawful relations it is expected to have with other variables. You validate a measure of anxiety by showing it behaves as anxiety should: it rises under threat, correlates with physiological arousal, distinguishes clinical from non-clinical groups, and does not merely duplicate a measure of general distress.

Why a measure can be reliable but invalid

This is the crux, and it is where intuition fails. Reliability is necessary for validity but nowhere near sufficient. A bathroom scale that always reads twelve pounds heavy is perfectly reliable and consistently wrong. A questionnaire in which every item secretly measures verbal fluency will show beautiful internal consistency, a high alpha, and stable retest scores, while measuring nothing like the trait on its label. Consistency guarantees only that the instrument is systematic, not that the system points at the intended target. This is why test builders treat a high reliability coefficient as a floor, not a verdict, and why Donald Campbell and Donald Fiske's 1959 multitrait-multimethod matrix demanded two things at once: convergent validity, meaning the measure agrees with other measures of the same trait, and discriminant validity, meaning it diverges from measures of different traits. A test that correlates with everything measures nothing in particular.

Construct validity as the frame

By the late twentieth century the field had largely folded content and criterion validity into construct validity. Samuel Messick argued in 1989 that validity is a unified, evolving judgment: not a property a test has or lacks, but the degree to which evidence and theory support the interpretations and uses made of its scores. On this view validation is never finished, and it explicitly includes the social consequences of using a test. A measure is not validated in the abstract. It is validated for a purpose.

Lineage

The apparatus grew out of the correlational tradition Francis Galton began and Karl Pearson formalized, then Charles Spearman applied to mental testing from 1904, deriving early reliability corrections in the same decade. Classical test theory, with its true-score-plus-error model, was consolidated by Harold Gulliksen in 1950. Cronbach's 1951 alpha generalized earlier internal-consistency work by G. Frederic Kuder and Marion Richardson. The conceptual turn came with Cronbach and Meehl in 1955, who moved validity from "does it predict a criterion" to "does it fit a theory," and with Campbell and Fiske in 1959, who gave that turn a method. Messick's unification in 1989 completed the arc. A later strand, item response theory (from the work of Frederic Lord and the Danish statistician Georg Rasch in the 1950s and 1960s), reframed measurement around the relation between a person's latent trait level and the probability of each item response, largely superseding classical test theory in high-stakes testing while leaving the reliability and validity questions intact.

The strongest case for it

The distinction is the discipline's first line of defense against fooling itself. Without it, any internally consistent instrument could pass as a measure of whatever its author named it, and psychology would drown in reliable measures of nothing. Separating consistency from meaning forces a second, harder question that no amount of statistical polish can answer on its own. The framework also travels: the same logic disciplines medicine (does a biomarker track the disease or an incidental correlate), education (does an exam measure learning or test-taking skill), and hiring. And construct validity, understood as an ongoing argument rather than a one-time certificate, gives the field a way to revise or retire a measure as evidence accumulates.

The strongest case against it

The critiques come from inside psychometrics as much as outside it. The most pointed target is the reflex to treat coefficient alpha as proof of quality. Klaas Sijtsma argued in 2009 that alpha is routinely misreported and misunderstood: it is neither a measure of a test's one-dimensionality nor a lower bound worth trusting under the conditions in which it is usually invoked, and reporting it has become a ritual that substitutes for real reliability analysis. Many methodologists now recommend alternatives such as McDonald's omega. The deeper worry is that a field can accumulate excellent reliability while its validity rests on thin evidence, because reliability is easy to compute and validity is a slow, arguable, theory-laden case that many published measures never really make.

The construct-validity program has its own critics. Because a construct is validated against a nomological network that the researcher also proposes, the enterprise can turn circular: the theory defines the trait, the trait is measured to test the theory, and disconfirming results can be absorbed by adjusting either. Joel Michell has pressed a more radical charge, that psychology never established its variables are quantitative in the first place, so the routine assumption that traits have interval structure is an unexamined article of faith rather than a finding.

The tensions are sharpest in a live case: the Implicit Association Test (see implicit-bias). The IAT, introduced by Anthony Greenwald, Debbie McGhee, and Jordan Schwartz in 1998, aims to measure automatic attitudes from reaction-time differences. Its test-retest reliability is modest, with reported values often around 0.5, well below what individual-level diagnosis would require, and its predictive validity for actual discriminatory behavior is contested: a 2013 meta-analysis by Frederick Oswald and colleagues found the IAT a weak predictor of behavior, while a 2009 meta-analysis by Greenwald and colleagues reported small but nonzero prediction, and the two camps still disagree on what the scores mean. The case shows the two axes doing exactly the work they were built for: a measure can be famous, theoretically motivated, and widely deployed, and still face open questions on both whether it is consistent enough to score a person and whether it tracks the construct it names.

Where it stands now

The reliability-versus-validity distinction is settled bedrock, taught in every measurement course and required by every serious test manual. What has moved is the sophistication around it. Alpha is on the retreat as the default reliability index; item response theory dominates large-scale and adaptive testing; and validity is now generally understood in Messick's unified, consequential sense rather than as a checklist of three separate types. The replication crisis (see the-replication-crisis) sharpened attention on measurement, since a study cannot replicate a finding about a construct it never measured well, a concern sometimes called a "measurement crisis" underlying the more famous one. The core lesson has only hardened: a number that sits still is not yet a number that means something, and establishing meaning is the slow, arguable work that the discipline can never finish once and for all.

Test yourself

Think of a number that is used to judge you or someone you know: a performance rating, a standardized score, a customer satisfaction figure. Ask the two questions separately. Is it reliable, would you get nearly the same number on a different day, from a different rater, using a slightly different set of questions? Then ask the harder one: even if it is perfectly consistent, is there good reason to believe it measures the quality it claims to, rather than something merely correlated with it that is easier to count? The gap between the two questions is where most bad measurement lives.