Producing Data & Sampling Design
Every statistical method in this course assumes the data arrived honestly. Where the numbers came from decides what they are allowed to prove, and no amount of careful analysis afterwards can repair a sample that was gathered the wrong way.
Population, sample, parameter, statistic
The population is the whole group you want to describe: every student at the university, every patient with the diagnosis, every trial a participant could have run. A sample is the part of it you actually measure. The number you want is a parameter, a fixed property of the population, and the number you get is a statistic, computed from your sample and different every time you repeat the study.
Those four words are the whole vocabulary of inference. Greek letters conventionally mark parameters (μ, pronounced mu, for a population mean; σ, sigma, for its standard deviation) and Latin letters mark statistics (x̄ for the sample mean, s for the sample standard deviation). You will meet that convention again in every formula in Section 1.11 onward.
Observational study or experiment?
An observational study measures people as you find them. You record who exercises and who does not, then compare their blood pressure. An experiment assigns the condition yourself: you decide, by a chance mechanism, which participants get the exercise program.
The difference decides what you may claim. In an observational study the groups differ in the thing you measured and in everything that made them sort themselves that way, so a difference in outcome has more than one available explanation. That extra variable is a confounder. Random assignment is what breaks the link, because it makes the groups comparable on every variable at once, the ones you thought of and the ones you did not. Observation supports a claim about association; a randomized experiment supports a claim about cause.
🎮 Four Ways to Choose 100 Students
A campus of 800 students is asked whether the university should run a late-night bus. On-campus students mostly say no; commuters mostly say yes. The true answer is fixed and known, so every design can be graded against it.
The 800 students. Filled dots want the bus; the block on the right commutes. One sample is ringed.
What 200 repeats of this design estimate. The line marks the truth.
What the four designs are doing
A simple random sample gives every group of n students the same chance of being the one you get. It is the reference design, and the histogram shows why: the estimates pile up on the true 40%, so the bias readout sits near zero whatever sample size you choose.
A stratified sample splits the population into groups that differ from each other, then draws a separate random sample inside each one. Here the strata are the residents and the commuters, sampled in proportion to their sizes. The estimates still center on the truth, and they cluster more tightly: at n = 100 the standard error falls from about 4.6% to about 3.9%, a 15% reduction bought with no extra participants. Stratifying pays off exactly when the groups differ on the thing you are measuring, which is why the design needs you to know something about the population first. Multistage designs extend the same idea, sampling schools and then classes and then pupils, because a national list of every pupil rarely exists.
The two remaining designs are the ones you meet in the wild. A convenience sample takes whoever is easy to reach, which on this campus means the café at lunchtime, which means residents. The histogram slides down to 25% and stays there. A voluntary response sample lets people put themselves forward, and the students who want the bus are the ones motivated to click the link, so the estimate climbs to about 67%.
A bigger bad sample is a worse bad sample. Push the slider right and watch the two flawed designs. Their spread shrinks like everyone else's, so the estimate becomes more and more precisely wrong. Sample size fixes variability; only the design fixes bias.
Randomized comparative experiments
The same logic runs the other way when you are assigning conditions rather than choosing people. A randomized comparative experiment takes the participants you have, splits them by a chance device, and gives each group a different treatment. Comparison rules out anything that would have happened anyway, such as recovery with time or practice on the task. Randomization rules out the systematic differences between the people in each group. A control group with no treatment, or a placebo, supplies the baseline that makes the comparison mean something.
Two refinements are worth naming now. Blocking is stratification applied to assignment: sort participants into similar blocks first, then randomize inside each block, so a known source of variation cannot land unevenly. A matched pairs design pushes that to its limit by pairing participants, or by measuring the same person under both conditions, which is where the paired t-test in Section 1.13 comes from.
Four ways a survey goes wrong
Undercoverage leaves part of the population off the list you sample from, so those people could never have been chosen. A phone survey misses households without a phone, and the campus café misses commuters entirely. Non-response happens after selection: you picked a proper random sample and some of them declined, and if the refusers differ from the responders your careful design has turned into a voluntary one without announcing it.
Response bias is about the answers rather than the people. Participants understate their drinking, overstate their voting, and adjust what they say to suit the interviewer in front of them. Question wording is the most easily overlooked, because a leading phrase moves an answer without anyone noticing: asking whether the university should "finally do something about late-night safety" measures the sentence rather than the opinion.
None of these show up in a standard error. A confidence interval reports how much your estimate would bounce around across repeated samples, and it is computed as though the design were sound. The interval around that 67% is honest arithmetic on a number that is 27 points from the truth.
Going deeper
This lesson is the course-level treatment. The Research Toolkit works the same ground in more depth, one design decision at a time: Sampling Methods covers probability and non-probability sampling frames and how to report a sampling strategy; Observational Designs separates cohort, case-control and cross-sectional studies; Experiments & Random Assignment goes further into allocation, blocking and blinding; and Reliability & Validity takes up whether the instrument measures what you think it measures once the sample is in the door.
Why it matters: the rest of Stats 1 assumes a random sample from the population you care about. Sampling distributions describe how a statistic varies across those repeats, and that description is only true if the sampling was random in the first place.
Common questions
What is the difference between an observational study and an experiment?
In an observational study you record the condition people are already in; in an experiment you assign it yourself, using a chance mechanism. That single difference decides what you may conclude. People who chose to exercise differ from people who did not in income, health and a dozen things nobody measured, so a blood-pressure gap has several available explanations. Random assignment makes the groups comparable on every variable at once, including the ones you never thought of, which is why only an experiment supports a claim about cause.
Is a bigger sample always better?
Only for precision. Sample size controls how much an estimate bounces around from study to study, and it does nothing at all about a design that points at the wrong number. A convenience sample of 5,000 estimates the population it can reach, more and more precisely, however large it grows. The famous case is the 1936 Literary Digest poll, which collected 2.4 million responses from car and telephone owners and called the presidential election for the wrong man. Fix the design first, then spend money on n.
How do I choose between simple random and stratified sampling?
Stratify when you know the population splits into groups that differ on what you are measuring, and when you can tell which group each person belongs to before sampling. That is what buys the precision: the between-group variation is removed from the sampling error rather than left in it. If the groups turn out to have similar means the design costs you effort and returns nothing, and if you cannot classify people in advance it is not available at all. A simple random sample is the safe default and needs no prior knowledge.