Sampling Distributions

All of inference is built on one idea: any number you calculate from a sample — its mean, its median, its standard deviation — is itself a random quantity. Take a different sample and you'd get a slightly different value. The distribution of those values over many samples is a sampling distribution.

One sample gives one estimate

Suppose the truth is fixed but unknown: the average reaction time of an entire population. You can't measure everyone, so you take a sample of n people and compute their mean. That single number is your estimate. But it came from a random draw, so it's a little high or a little low by luck.

The key step is to imagine repeating the study thousands of times. Each repetition gives one estimate; collect them all, and you get the sampling distribution of that statistic. It shows how much the estimate varies from sample to sample, and you need that to judge how much to trust a single estimate.

Two things to take away

Different statistics, different distributions

Switch the statistic from mean to median and the sampling distribution changes shape and width. Every statistic has its own sampling distribution. So you can reason about the reliability of any estimate, not just the mean.

Which estimate should you trust?

That switch raises a question. When two statistics both estimate the middle of a population, which one should you use? Statisticians judge estimators on three properties, and the widget shows all three.

An estimator is unbiased when its sampling distribution is centered on the true value, so the "Average estimate" readout settles on the population value. It is consistent when the whole distribution narrows onto that value as n rises; the sample-size slider shows this. Efficiency compares estimators: of two unbiased estimators, the more efficient one has the smaller standard error, so it gets more precision out of the same participants. Methods sections use all three words without defining them, and the glossary has a short entry for each.

Efficiency has practical consequences. When you sample from a normal population, the median's standard error is larger than the mean's by a factor of √(π/2), about 1.253. To match the mean's precision with the median you therefore need π/2, roughly 1.57 times as many people. The median ignores the sizes of the values, which makes it robust to outliers, but on normal data it needs a larger sample.

The three populations in the widget show the same trade-off. Each row below is 400,000 simulated draws at n = 25:

The bimodal row is the extreme case, and the population panel shows why. Very few values in that population are near 50, so a sample's middle value usually comes from one hump or the other: 74% of sample medians fall outside the 40-to-60 band. The mean uses every value and varies much less.

For the skewed row, efficiency is the wrong question. That population's mean is about 16 while its median is 11.1, so the two statistics are estimating different numbers. Set the widget to Skewed and Sample median and the "Average estimate" readout moves toward 11, not 16. Neither statistic is wrong. They answer different questions, and only for a symmetric population do they estimate the same value. Skew is also why the median is the usual summary for income or reaction times, and why a few extreme values change one statistic far more than the other.

One practical problem: the median has no formula as simple as SE = σ/√n. Its standard error depends on how densely the population's values are packed around its median, which you never know. This is one reason to use the bootstrap: resampling your own data estimates the width of any sampling distribution without a formula.

Problem 6 of the practice problems works the standard error in both directions: what it is for 25 drivers, what happens at 100, and how many you would need to reach a target precision.