Variance and spread

Intuitive problem

Two suppliers quote the same average delivery time: ten days. On the average alone they are interchangeable, yet their records differ in a way the mean cannot see. Supplier A's deliveries land within a day of ten on every order in the file. Supplier B's alternate between arriving about a week early and about a week late, and the earliness and lateness cancel so neatly that the mean comes out at ten days for both. A planner who must promise a customer a date cares intensely about the difference, and the mean offers no help. What is wanted is a second number measuring how far the outcomes typically sit from their own average, and the question is what that number should be and why it takes the form it does.

Definition

Take a quantity with recorded outcomes x_1, x_2, up to x_n, where n is the number of outcomes. Their mean m is the sum of the outcomes divided by n. The variance is the average of the squared deviations from the mean: square each difference x_i minus m, sum the squares, and divide by n. The standard deviation, written s, is the square root of the variance, which brings the measure back into the original units, days or milliliters. For a random quantity X described by probabilities rather than a data file, the same definition reads through the expected value E: the variance of X is E[(X minus E[X]) squared], the probability-weighted average of the squared deviation.

One convention is fixed here: this entry divides by n, the so-called population divisor, treating the data in hand as the whole story. The alternative divisor n minus 1 exists and matters for sample data; it is named in Common mistakes and left there.

Derivation

The definition above deserves to be earned rather than announced, because the squaring looks arbitrary until you see what forces it.

Start with the obvious first attempt: average the deviations themselves. It fails identically. The sum of the deviations is the sum of the x_i minus n copies of m, and since n times m is by definition the sum of the x_i, the two cancel and the average signed deviation is exactly zero for every data set. Overshoots and undershoots erase each other by construction. So any usable spread measure must first kill the signs, and the two natural ways to do that are taking absolute values or squaring. Both give legitimate measures; two properties pick out the square.

First, the mean is the unique center that minimizes the average squared deviation. Measure spread around an arbitrary center c instead of m, and expand each term: x_i minus c equals (x_i minus m) plus (m minus c). Squaring and averaging gives three pieces: the variance itself, plus twice (m minus c) times the average signed deviation, plus (m minus c) squared. The middle piece vanishes by the cancellation just derived, leaving the variance plus (m minus c) squared. That extra piece is a square, so it is positive whenever c differs from m and zero when c equals m. Squared deviation is smallest around the mean and around nothing else, so mean and variance belong together as center and spread.

Second, variances add across independent quantities. Write the centered versions of two independent random quantities X and Y, that is, each minus its own expected value. The variance of X plus Y expands into the variance of X, plus the variance of Y, plus twice the expected product of the two centered quantities. For independent quantities the expected value of a product is the product of the expected values, so that cross term is E of centered X times E of centered Y, and each factor is zero by the cancellation above. Two lines, and the conclusion: the variance of a sum of independent quantities is the sum of their variances. The absolute-value measure has neither the minimizing property nor this additivity, and that is what picks the square.

The definition also yields a shortcut worth deriving. Expand E[(X minus E[X]) squared] term by term: E[X squared] minus twice E[X] times E[X], plus E[X] squared, which collapses to E[X squared] minus (E[X]) squared. In words, the variance is the mean of the squares minus the square of the mean. It saves arithmetic by hand, and it carries a hazard: when the mean is large compared to the spread, the two terms are nearly equal large numbers, and subtracting them cancels the leading digits, a loss of precision treated in the entry on approximation error and numerical stability. Use the shortcut for hand-sized numbers and distrust it in floating-point code.

Worked example

Return to the suppliers, with four recorded delivery times each. Supplier A: 9, 10, 10, 11 days. Supplier B: 3, 17, 3, 17 days. Each set sums to 40, so each mean is 10 days, and the averages are indistinguishable.

Now the spread, with the population divisor n equal to 4. Supplier A's deviations from 10 are minus 1, 0, 0, and 1; the squares are 1, 0, 0, 1; their sum is 2; the variance is 2 divided by 4, or 0.5 days squared, and the standard deviation is the square root, about 0.71 days. Supplier B's deviations are minus 7, 7, minus 7, 7; every square is 49; the variance is 49 days squared and the standard deviation is exactly 7 days. Equal means, and spreads a factor of ten apart in standard deviation. The shortcut form checks B: the mean of the squares is (9 plus 289 plus 9 plus 289) divided by 4, which is 149, minus the squared mean of 100, giving 49, as before.

Limits and boundary conditions

Squaring amplifies whatever is already large, so variance is sensitive to outliers: one wild observation can dominate the sum of squares, so the estimate can say more about one bad day than about the process. Pushed to the extreme, the sensitivity becomes non-existence: for some heavy-tailed distributions the probability-weighted sum defining E[(X minus E[X]) squared] diverges, and the variance is undefined outright, so the quantity cannot be assumed to exist just because a distribution does.

The standard deviation is fully descriptive only for roughly normal, bell-shaped data, where fixed fractions attach to it: about 68 percent of outcomes within one standard deviation of the mean and about 95 percent within two. For skewed or lumpy data those fractions fail, and the standard deviation shrinks back to a summary number without that interpretive layer.

And one honest admission a level up. This entry charges the mean with blindness, and the same charge lands on variance in turn: two processes can share both mean and variance and still carry different risk, because variance squares the deviations and thereby discards their direction. A process that rarely deviates but overshoots enormously when it does, and a mirror process that undershoots the same way, can produce identical means and variances while exposing you to opposite dangers. Spread is the second summary number, and it has its own residue, which is why distributions carry more information than any short list of summaries.

Common mistakes

Four errors recur. First, inferring spread from size: a larger mean does not imply a larger variance; a process can run at a high level steadily or at a low level erratically, and the two numbers vary independently. Second, treating the standard deviation as the average absolute deviation. They agree when every deviation has the same magnitude, as with supplier B above, where both equal 7, but in general they differ: supplier A's average absolute deviation is 0.5 days while its standard deviation is about 0.71, because squaring weights the larger deviations more heavily. Third, quoting variance in the original units. Variance carries squared units, days squared or milliliters squared, and the standard deviation exists precisely to return to units a decision can be stated in. Fourth, the divisor. Software and textbooks often divide the sum of squares by n minus 1 rather than n when the data are a sample drawn from something larger; this is a convention difference the reader will meet, it slightly enlarges the result, and it changes nothing in the logic above. This entry computes with n throughout.

Build with it

Two bottling machines fill bottles labeled 500 milliliters. Five measured fills from machine P: 498, 500, 500, 502, 500. Five from machine Q: 490, 512, 495, 505, 498. Verify first that both means equal 500, so the average alone cannot choose between them.

Then, by hand and with the population divisor n as defined in this entry, compute each machine's variance and standard deviation. State the interval from the mean minus two standard deviations to the mean plus two standard deviations for each machine, and check what fraction of that machine's five fills the interval actually contains; report the fraction you find rather than the fraction the normal-shape rule predicts, since five points is a coarse check. Finally, make the reliability call: a contract requires every bottle within 5 milliliters of the label. Decide which machine to run and defend the decision from the spreads, comparing each machine's two-standard-deviation half-width against the 5 milliliter tolerance.

Success is three things: both variances and standard deviations correct under the divisor n, the two intervals stated and checked against the data with the fractions reported, and a machine choice defended by the spread arithmetic rather than by the means, which were built to tie.