Sampling Distributions

One idea powers all of inference: any number you calculate from a sample — its mean, its median, its standard deviation — is itself a random quantity. Take a different sample and you'd get a slightly different value. The pattern those values make, over many samples, is a sampling distribution.

One sample gives one estimate

Suppose the truth is fixed but unknown — the average reaction time of an entire population. You can't measure everyone, so you take a sample of n people and compute their mean. That single number is your estimate. But it came from a random draw, so it's a little high or a little low by luck.

The crucial move is to imagine repeating the study thousands of times. Each repetition gives one estimate; collect them all, and you get the sampling distribution of that statistic. It tells you how much your estimate bounces around — and that's exactly what you need to judge how trustworthy a single estimate is.

Two things to take away

Different statistics, different distributions

Switch the statistic from mean to median and the sampling distribution changes shape and width. Every statistic has its own sampling distribution. That's a big deal: it means we can reason about the reliability of any estimate, not just the mean.

Which estimate should you trust?

That switch raises a question the bottom panel can settle. When two statistics both aim at the middle of a population, which one deserves your sample? Estimators get graded on three properties, and the widget shows all three.

An estimator is unbiased when its sampling distribution sits centered on the truth, so the "Average estimate" readout settles on the population value instead of beside it. It is consistent when the whole distribution tightens onto that value as n rises, which is what the sample-size slider demonstrates. Efficiency is the comparative one: of two unbiased estimators, the efficient one has the smaller standard error, so it gets more precision out of the same participants. Methods sections use all three words without stopping to define them, and the glossary keeps a short entry for each.

Efficiency carries a price tag. Sampling from a normal population, the median's standard error runs larger than the mean's by a factor of √(π/2), about 1.253. Matching the mean's precision with a median therefore costs π/2, roughly 1.57 times as many people. Discarding the magnitudes buys robustness, and you pay for it in sample size.

The three populations in the widget make that trade visible. Each row below is 400,000 simulated draws at n = 25:

The bimodal row is the extreme case, and the population panel explains it. Hardly anyone sits near 50 in that population, so the middle value of a sample has nothing solid to settle on and swings between the two humps: 74% of sample medians land outside the 40-to-60 band. The mean uses every value and stays put.

The skewed row tells a different story, and the honest answer there is that efficiency is the wrong question. That population's mean is about 16 while its median is 11.1, so the two statistics are estimating different numbers. Set the widget to Skewed and Sample median and the "Average estimate" readout drifts toward 11 rather than 16. Neither is wrong; they answer different questions, and only for a symmetric population are they competing for the same job. Skew is also why the median is the usual summary for income or reaction times, and why a few extreme values change one statistic far more than the other.

One practical wrinkle: the median has no formula as tidy as SE = σ/√n. Its standard error depends on how densely the population packs around its own middle, which you never know. That is one of the reasons the bootstrap exists, since resampling your own data estimates the width of any sampling distribution without needing a formula for it.

Problem 6 of the practice problems works the standard error in both directions: what it is for 25 drivers, what happens at 100, and how many you would need to reach a target precision.