Quasi-Experiments & Natural Experiments
Often you simply can't randomize. You can't randomly assign people to smoke, to live through a policy change, or to attend one school over another. A quasi-experiment keeps the structure of an experiment (a treatment, a comparison, an outcome) but drops random assignment, the piece that made the causal claim secure. The work is then to build a credible causal claim from groups that were never equivalent to begin with.
The quasi-experimental toolkit
- Nonequivalent groups: a treated group and a comparison group that formed by choice or circumstance, not a coin flip, so they may differ in hidden ways.
- Pre/post (one group): measure before and after a treatment. Cheap, but anything else changing over time (maturation, the season, the economy) is confounded with the effect. The analysis is the ordinary within-subjects comparison. What is missing is a group that did not get the treatment.
- Interrupted time series: many measurements before and after an intervention, so you can see whether the trend changed at the intervention.
- Matching: pair each treated unit with a comparison unit that looks similar on measured variables. This helps, but it can only balance what you measured, never the unknown confounds randomization would have balanced. Which variables belong in the match depends on the causal structure, not on which ones you happen to have. Matching on a variable and adjusting for it statistically are two ways to remove the same imbalance, and neither can account for a variable you never recorded.
Difference-in-differences: the workhorse
A common and effective design combines a nonequivalent comparison group and a before/after measurement. Two simple estimates each fail on their own, and subtracting one from the other fixes both. Try it:
📈 Difference-in-Differences Explorer
A treated group and a comparison group, each measured before and after. See why "compare the groups afterwards" and "compare the treated group's change" both mislead, and how the difference-in-differences estimate cancels their biases, as long as the trends would have stayed parallel.
Comparing the groups after only is biased by the baseline gap (the groups differed to begin with). Comparing the treated group's change only is biased by the shared trend (things were moving anyway). Difference-in-differences subtracts the comparison group's change from the treated group's change, which removes both biases, provided the two groups would have moved in parallel without the treatment. That condition is the parallel-trends assumption. The differential-trend slider shows that a violation adds its full size to your estimate. In a regression the whole estimate is one coefficient, the treated × post interaction term.
That coefficient has a lesson of its own. Difference-in-differences in the Statistics Core covers what each of the four coefficients means, which three not to interpret, and the two-way fixed-effects form papers write it in. It also explains why the standard errors have to be clustered, and how an event study tests parallel trends one period at a time.
Regression discontinuity, in one picture
When a treatment is handed out by a strict cutoff (a scholarship for scores ≥ 90, a program for incomes below a line), you can compare people just above and just below the threshold. They're nearly identical, and which side of the cutoff each one ended up on is close to chance. So a sudden jump in the outcome right at the cutoff is hard to explain as anything but the treatment. It's the closest a quasi-experiment gets to a local randomized trial.
To estimate the jump, see Regression discontinuity design in the Statistics Core. It shows how to center the running variable so that one coefficient is the jump. It explains why the two sides need their own slopes and how wide a window to fit. It also covers three checks for judging a published study: balanced covariates, a smooth density and placebo cutoffs.
Natural experiments take this further: nature or policy does the "assigning" for you, whether a lottery, an arbitrary border, or a law that changed in one region but not the neighboring one. When the assigning event is as-good-as-random with respect to the outcome, a natural experiment can match a true experiment for causal credibility, using treatments you could never have assigned yourself. When there is no assigning event at all and you are simply watching who does what, you have moved into observational territory, where you have to build the comparison yourself.
A lottery of that kind is also the cleanest example of an instrument. This lesson stops at the design. Instrumental variables and 2SLS in the Statistics Core covers the estimator. It shows which regression to run on which variable, why the answer is a ratio of two of them, and whose effect the resulting number averages.
Most of the time you will meet these designs in somebody else's paper rather than your own data. Reading a Causal-Inference Paper is the guide for that: how to find the identifying assumption, judge the defense offered for it, and work out whose effect the estimate belongs to.
Why it matters: most of the questions that matter most (about policies, diseases, life events) can't be randomized for practical or ethical reasons. Quasi-experiments are how research still makes progress on them. But each one depends on an assumption that randomization would have made unnecessary. Name that assumption, and design your comparison to make it as believable as you can.
Problem 95 of the practice problems is a self-selected comparison presented as a result. It asks which quasi-experimental design would have made the claim defensible.
Common questions
When does regression discontinuity fail?
It fails when the cutoff is not as sharp as it looks. The design assumes that people just above and just below the threshold are alike apart from the treatment, and anything that lets them choose their side breaks that assumption. The usual sign is bunching: plot the distribution of the running variable and look for a pile-up on the favorable side of the line. A pile-up means somebody was retaking the test, rounding the score, or granting exceptions. The design has two more limits. The estimate is local, valid near the cutoff and not for people far from it, so a scholarship effect measured at a score of 90 says little about students at 60. And the answer can depend on how wide a window you look at, so report the estimate across several bandwidths, not only the one that worked.
What is the parallel-trends assumption in difference-in-differences?
Difference-in-differences estimates a treatment effect by subtracting the comparison group's before-to-after change from the treated group's change. That subtraction is valid only if, absent the treatment, both groups would have moved by the same amount — the parallel-trends assumption. It allows the groups to start at different levels (a baseline gap is fine) but requires their trends to match. If the treated group was already on a steeper trajectory, that differential trend gets counted as 'effect,' biasing the estimate.
What is a natural experiment?
A natural experiment is a quasi-experiment where some outside force (a lottery, a law change, an arbitrary cutoff or border) assigns the 'treatment' in a way that is as-good-as-random with respect to the outcome. The researcher doesn't manipulate anything and instead studies an assignment that happened by accident. When the assigning event is unrelated to who would have done well anyway, a natural experiment can approach the causal credibility of a randomized trial. It can do that on questions you could never ethically or practically assign yourself.