Sampling Methods

You almost never measure the whole population. You measure a sample and hope it represents the rest, and whether it can depends entirely on how you picked it. A good sampling method makes the sample representative on average. A bad one builds in a bias that no amount of extra data will remove.

Probability samples: everyone has a known chance

These use chance to choose who's in, so the sample is representative in expectation and you can quantify its uncertainty (this is what sampling distributions describe).

• Simple random sampling (SRS): every person has an equal chance; draw names from a hat. Unbiased, and the benchmark, though it needs a complete list and can, by luck, miss small subgroups.
• Stratified sampling: split the population into meaningful groups (strata: age bands, regions, faculties) and sample within each, usually in proportion. Guarantees every group is represented and, by removing the between-group variation, gives a more precise estimate than SRS for the same sample size.
• Cluster sampling: split into many natural groups (clusters: schools, city blocks), randomly pick a few whole clusters, and measure everyone in them. Cheap when the population is spread out, but people within a cluster resemble each other, so the estimate is far noisier. An unlucky choice of clusters can put the estimate far from the truth.
• Systematic sampling: order the list and take every kth person from a random start. As good as SRS if the ordering is unrelated to what you're measuring, but dangerous if the list has a hidden periodicity that lines up with your step.

Non-probability samples: convenient, and treacherous

Convenience sampling (whoever's easy to reach: your class, passers-by, an online panel) and snowball sampling (participants recruit their friends) don't give everyone a chance of selection. Nothing guarantees that the sample is representative, and it usually isn't. They're sometimes unavoidable for hard-to-reach groups, but the people who are easy to reach differ systematically from those who aren't. Quota sampling is easy to mistake for a probability method. It sets targets that mirror the population (50 men, 50 women, 30 of them over 60) and then lets the recruiter fill each target with whoever turns up. The final table of counts looks exactly like a stratified sample. But the selection inside each quota was never random, so the guarantee that comes with stratification is not there.

Sample it five ways and watch

This is a stylized city whose opinion on some issue varies by neighborhood: the campus district (top-left) is overwhelmingly in favor, the south is against. The true city-wide support is marked. Sample the city each way, draw many times, and compare each method's estimates with the truth.

Bias is where the cloud is centered, and variance is how wide it is. Convenience sampling is biased: its cloud is centered far from the truth and only gets narrower (a more precise wrong answer) as n grows. The probability methods are all roughly unbiased, centered on the truth, but they differ in variance. Stratified is tightest (and, on this neighborhood-ordered list, systematic behaves like it), simple random is wider, and cluster is widest of all. More data reduces variance. Only a better method reduces bias.

Putting a number on the cluster penalty

Survey statisticians measure how much noisier cluster sampling is. The design effect is the ratio of your design's variance to the variance a simple random sample of the same size would have produced. A design effect of 1 means clustering added no variance. A design effect of 3 means your estimate's variance is three times what SRS would have given, so two thirds of your fieldwork added no precision. Dividing your sample size by it gives the number that matters for planning, the effective sample size: n_eff = n / deff. It is the number of independent people your clustered sample is worth.

For clusters of roughly equal size, Kish's approximation uses two quantities you can estimate before you start: deff ≈ 1 + (m − 1)ρ. Here m is the number of people you measure per cluster, and ρ is the intraclass correlation, the share of the total variation that is between clusters rather than within them. With one person per cluster there is no penalty, since m = 1 makes the design effect exactly 1. But even a correlation small enough to look negligible matters once it is multiplied by a large cluster.

The power lesson asks the question without working it out: 30 classes of 20 pupils is 600 children, but how many independent ones?

At ρ = .10, which is typical for children who share a teacher, roughly two thirds of the sample's information is lost before you analyze anything. Run this calculation while you are still choosing schools.

The city above is an extreme case. Its four neighborhoods disagree sharply, 85% support by the campus against 20% in the south-east, so ρ is about .24, and the cluster method draws the whole sample from one neighborhood. Take 80 people that way and the estimates have a standard deviation near .25, against .05 for a simple random sample of the same 80. That is a design effect of about 24, so those 80 answers hold roughly the information of 3 independent ones. You can compute the width of that cloud in advance.

Frame, population, and the people who don't answer

Even a perfect random method samples from your sampling frame (the actual list you draw from), not the true population. If your frame is "students with a campus email," everyone without one is excluded before you begin (coverage error). Then there is nonresponse bias: if the people who ignore your survey differ from those who answer, as they usually do, your realized sample differs from your intended one. A well-known example is the WEIRD problem: a very large share of published psychology is based on Western, Educated, Industrialized, Rich, Democratic undergraduates, who are an unusual sample of humanity to generalize from.

Problem 96 of the practice problems is a council survey with 4,300 responses and no usable answer, because a large sample cannot fix a bad sampling method.

Stats 1 covers this material at course level in Producing Data & Sampling Design. Its widget runs four of these designs on the same population and compares each one with a known truth.