Quasi-Experiments & Natural Experiments
Often you simply can't randomize. You can't randomly assign people to smoke, to live through a policy change, or to attend one school over another. A quasi-experiment keeps the spirit of an experiment (a treatment, a comparison, an outcome) but drops the one piece that made it airtight: random assignment. The whole craft is squeezing a credible causal claim out of groups that were never equivalent to begin with.
The quasi-experimental toolkit
- Nonequivalent groups: a treated group and a comparison group that formed by choice or circumstance, not a coin flip, so they may differ in hidden ways.
- Pre/post (one group): measure before and after a treatment. Cheap, but anything else changing over time (maturation, the season, the economy) is confounded with the effect.
- Interrupted time series: many measurements before and after an intervention, so you can see whether the trend genuinely bent at the intervention.
- Matching: pair each treated unit with a comparison unit that looks similar on measured variables, which helps, but can only balance what you measured, never the unknown confounds randomization would have caught. Which variables belong in the match is a question about causal structure, not a question about which ones you happen to have.
Difference-in-differences: the workhorse
The cleverest common design combines a nonequivalent comparison group and a before/after measurement. Two naïve estimates each fail on their own — and subtracting them fixes both. Play with it:
📈 Difference-in-Differences Explorer
A treated group and a comparison group, each measured before and after. See why "compare the groups afterwards" and "compare the treated group's change" both mislead, and how the difference-in-differences estimate cancels their biases, as long as the trends would have stayed parallel.
Two biases, canceled: comparing the groups after only is fooled by the baseline gap (the groups differed to begin with); comparing the treated group's change only is fooled by the shared trend (things were moving anyway). Difference-in-differences subtracts the comparison group's change from the treated group's change, wiping out both, provided the two groups would have moved in parallel without the treatment. That last clause is the parallel-trends assumption, and the differential-trend slider shows exactly how much a violation of it leaks straight into your estimate.
Regression discontinuity, in one picture
When a treatment is handed out by a strict cutoff — a scholarship for scores ≥ 90, a program for incomes below a line — you can compare people just above and just below the threshold. They're nearly identical by luck of landing on either side, so a sudden jump in the outcome right at the cutoff is hard to explain as anything but the treatment. It's the closest a quasi-experiment gets to a local randomized trial.
Natural experiments take this further: nature or policy does the "assigning" for you, whether a lottery, an arbitrary border, or a law that changed in one region but not the neighboring one. When the assigning event is genuinely as-good-as-random with respect to the outcome, a natural experiment can rival a true experiment for causal credibility, on data you could never have assigned yourself.
Why it matters: most of the questions that matter most (about policies, diseases, life events) can't be randomized for practical or ethical reasons. Quasi-experiments are how research still makes progress on them. But every one leans on an assumption that randomization would have made unnecessary, so name that assumption out loud, and design your comparison to make it as believable as you can.
Problem 38 of the practice problems is a self-selected comparison dressed up as a result, and asks which quasi-experimental design would have made the claim defensible.
Common questions
When does regression discontinuity fail?
It fails when the cutoff is not as sharp as it looks. The design leans on one idea, that people just above and just below the threshold are alike apart from the treatment, and anything that lets them choose their side breaks it. The usual tell is bunching: plot the distribution of the running variable and look for a pile-up on the favorable side of the line, which means somebody was retaking the test, rounding the score, or granting exceptions. Two more limits worth stating in advance. The estimate is local, valid near the cutoff and not for people far from it, so a scholarship effect measured at a score of 90 says little about students at 60. And the answer can depend on how wide a window you look at, which is why an honest analysis shows the estimate across several bandwidths rather than the one that worked.
What is the parallel-trends assumption in difference-in-differences?
Difference-in-differences estimates a treatment effect by subtracting the comparison group's before-to-after change from the treated group's change. That subtraction is valid only if, absent the treatment, both groups would have moved by the same amount — the parallel-trends assumption. It allows the groups to start at different levels (a baseline gap is fine) but requires their trends to match. If the treated group was already on a steeper trajectory, that differential trend gets counted as 'effect,' biasing the estimate.
What is a natural experiment?
A natural experiment is a quasi-experiment where some outside force (a lottery, a law change, an arbitrary cutoff or border) assigns the 'treatment' in a way that is as-good-as-random with respect to the outcome. The researcher doesn't manipulate anything; they exploit the accident. When the assigning event really is unrelated to who would have done well anyway, a natural experiment can approach the causal credibility of a randomized trial on questions you could never ethically or practically assign yourself.