Sampling Methods

You almost never measure the whole population. You measure a sample and hope it stands in for the rest, and whether it can depends entirely on how you picked it. A good sampling method makes the sample representative on average; a bad one bakes in a bias that no amount of extra data will ever wash out. This is the difference between an estimate that homes in on the truth and one that confidently misses.

Probability samples: everyone has a known chance

These use chance to choose who's in, so the sample is representative in expectation and you can quantify its uncertainty (this is what sampling distributions describe).

• Simple random sampling (SRS): every person has an equal chance; draw names from a hat. Unbiased, and the benchmark, though it needs a complete list and can, by luck, miss small subgroups.
• Stratified sampling: split the population into meaningful groups (strata: age bands, regions, faculties) and sample within each, usually in proportion. Guarantees every group is represented and, by removing the between-group variation, gives a more precise estimate than SRS for the same sample size.
• Cluster sampling: split into many natural groups (clusters: schools, city blocks), randomly pick a few whole clusters, and measure everyone in them. Cheap when the population is spread out, but people within a cluster resemble each other, so the estimate is far noisier. Pick the wrong clusters and it swings you badly.
• Systematic sampling: order the list and take every kth person from a random start. As good as SRS if the ordering is unrelated to what you're measuring, but dangerous if the list has a hidden periodicity that lines up with your step.

Non-probability samples: convenient, and treacherous

Convenience sampling (whoever's easy to reach: your class, passers-by, an online panel) and snowball sampling (participants recruit their friends) don't give everyone a chance of selection, so there's no guarantee of representativeness, and usually no hope of it. They're sometimes unavoidable for hard-to-reach groups, but the people who are easy to reach differ systematically from those who aren't. Quota sampling is the one that fools people: it sets targets that mirror the population (50 men, 50 women, 30 of them over 60) and then lets the recruiter fill each target with whoever turns up. The final table of counts looks exactly like a stratified sample. The selection inside each quota was never random, so the guarantee that comes with stratification is not there.

Sample it five ways and watch

This is a stylized city whose opinion on some issue varies by neighborhood: the campus district (top-left) is overwhelmingly in favor, the south is against. The true city-wide support is marked. Sample the city each way, draw many times, and compare where each method's estimates land.

One pattern runs through all five methods. Bias is where the cloud is centered; variance is how wide it is. Convenience sampling is biased: its cloud sits far from the truth and merely gets tighter (a more precise wrong answer) as n grows. The probability methods are all roughly unbiased, centered on the truth, but they differ in variance. Stratified is tightest (and, on this neighborhood-ordered list, systematic behaves like it), simple random is wider, and cluster is widest of all. More data fixes variance; only a better method fixes bias.

Putting a number on the cluster penalty

"Cluster sampling is noisier" deserves a number, and survey statisticians have one. The design effect is the ratio of your design's variance to the variance a simple random sample of the same size would have produced. A design effect of 1 means the method cost you nothing. A design effect of 3 means your estimate wobbles three times as widely as SRS would have, so two thirds of your fieldwork bought no precision at all. Divide by it and you get the number that matters for planning, the effective sample size: n_eff = n / deff, the count of independent people your clustered sample is genuinely worth.

For clusters of roughly equal size, Kish's approximation writes it in terms of two quantities you can estimate before you start: deff ≈ 1 + (m − 1)ρ, where m is how many people you measure per cluster and ρ is the intraclass correlation, the share of the total variation that sits between clusters rather than within them. Measuring one person per cluster costs nothing, since m = 1 makes the design effect exactly 1. And a correlation small enough to look negligible stops being negligible once it is multiplied by a large cluster.

The power lesson poses the case and leaves the arithmetic open: 30 classes of 20 pupils is 600 children, but how many independent ones?

At ρ = .10, which is unremarkable for children who share a teacher, roughly two thirds of the sample has evaporated before you analyze anything. That is a calculation to run while you are still choosing schools.

The city above is the same arithmetic at its extreme. Its four neighborhoods disagree so sharply, 85% support by the campus against 20% in the south-east, that ρ works out at about .24, and the cluster method draws the whole sample from one neighborhood. Take 80 people that way and the estimates land with a standard deviation near .25, against .05 for a simple random sample of the same 80. That is a design effect of about 24, so those 80 answers carry roughly the information of 3 independent ones. The width of that cloud is not a quirk of the animation; it is a number you can compute in advance.

Frame, population, and the people who don't answer

Even a perfect random method samples from your sampling frame (the actual list you draw from), not the true population. If your frame is "students with a campus email," everyone without one is invisible before you begin (coverage error). Then nonresponse bias strikes: if the people who ignore your survey differ from those who answer, as they usually do, your realized sample drifts from your intended one. A famously overlooked flavor is the WEIRD problem: a huge share of published psychology rests on Western, Educated, Industrialized, Rich, Democratic undergraduates, who are a strange sample of humanity to generalize from.

Problem 39 of the practice problems is a council survey with 4,300 responses and no usable answer, which is the clearest demonstration there is that sample size cannot rescue a sampling method.