Difference-in-Differences

Of all the causal designs, this is the one whose output looks least like an achievement. A policy arrived in some places and not others, you have records from before and after, and the entire answer is a single coefficient on a product of two dummy variables. Quasi-experiments in the Research Toolkit shows why that subtraction works: comparing the groups afterward is fooled by whatever gap they started with, comparing the treated group's own change is fooled by whatever was moving anyway, and taking the difference of the two differences cancels both. This lesson starts where that one stops, with the regression that does the canceling, the standard errors it needs, and the assumption holding the whole thing up.

One coefficient, four cells

Here is the setup in its smallest form. A regional education authority put a peer-tutoring scheme into 25 of its 50 schools from the fifth term of an eight-term record, and the outcome is each school's mean end-of-term score. Average the terms before the switch and the terms after it, for each group, and you have four numbers:

The bottom-right cell is the estimate, and it can be read two ways that give the same number: the tutoring schools gained 6.5 and the comparison schools gained 3.9, so 2.6 points of the gain are left over; or the gap between the groups widened from 2.2 to 4.8, and 2.6 is how much it widened. Those are the two differences the design is named after, and it does not matter which you difference first.

The regression that does it

Stack the data so that every school contributes one row per period, and build two indicators. Treated is 1 for a school in the scheme and 0 otherwise, in every period, including the ones before the scheme existed. Post is 1 in the periods after the switch and 0 before it, for every school, including the ones that never joined. Then fit

Y = b₀ + b₁Treated + b₂Post + b₃(Treated × Post) + e

and the four coefficients are the four cells of that table, rearranged. b₀ is the comparison group before the switch, 61.9. b₁ is the baseline gap between the groups, 2.2. b₂ is the comparison group's own change over time, 3.9. And b₃, the coefficient on the interaction, is 2.6, the estimate. Add all four together and you get 70.6, the treated group after, which is the check that the model has spent every number on something.

Two of those coefficients get misread often enough to be worth naming. b₁ is not an effect of the scheme; it is the difference between two kinds of school that existed before any scheme did, and the whole design is built around not believing it. b₂ is not an effect of time in any interesting sense either; it is whatever else happened to everybody between the two periods, which is exactly the thing being subtracted out. Only b₃ is an answer, and a results table that reports all four with stars beside them invites a reader to interpret three quantities that were never meant to carry meaning.

The same model, written the way papers write it

With two groups and two periods the specification above is complete. Real data usually have more of both, and then the standard form replaces each indicator with a full set of dummies:

Y_it = a_i + g_t + bD_it + e_it

where a_i is a dummy for each unit, g_t is a dummy for each period, and D_it is 1 when unit i is being treated at time t. This is two-way fixed effects, and the unit dummies absorb anything about a school that never changes while the period dummies absorb anything that hit every school at once. It is the same estimator: with two groups and two periods, a dummy per unit and a dummy per period span exactly the same space as Treated, Post and an intercept, so b and b₃ are the identical number. The fixed-effects form is how it appears in almost every paper you will read, and it is what lets the design handle fifty schools rather than two.

What changes with the longer panel is the standard error. A school observed in eight terms contributes eight rows, and those eight rows are not eight independent facts about the world; a school that is unusually good is unusually good in all of them. Ordinary least squares takes the row count at face value and returns standard errors that are far too small, sometimes by a factor of two or three. The standard answer is clustered standard errors, grouped at the level the treatment was assigned at, which here is the school, so that each school counts as one draw rather than eight. In the small case this has an exact and reassuring form: reduce each school to a single number, its own after-minus-before change, and compare the 25 changes in one group with the 25 in the other with an independent-samples t-test. That is the clustered standard error, computed by hand. In the tutoring data it gives 2.60 points, SE = 0.63, t(48) = 4.13, p < .001, 95% CI [1.33, 3.87].

Parallel trends, and why no test reaches it

The subtraction works only if the comparison group's change is the right stand-in for what the treated group would have done untreated. That is the parallel-trends assumption, and reading it carefully is the whole skill. It permits the two groups to sit at different levels forever, which is why b₁ is allowed to be large. It does not permit them to be moving at different speeds. And it is a statement about a path nobody observed, the treated group's outcome in a world where the scheme never arrived, so there is no column in any dataset to test it against. Whatever a paper offers, it is offering evidence about a world it cannot see.

What it can offer is the periods before the treatment, where both groups were untreated and any divergence between them is a divergence the design cannot explain. The formal version is an event study: instead of one before-and-after contrast, estimate a separate coefficient for each period, measured against a chosen reference period just before the switch. In the tutoring data, with term 4 as the reference, that means seven coefficients for terms 1, 2, 3, 5, 6, 7 and 8, each one the gap between the groups in that term minus the gap in term 4. With only two groups you can compute them without any regression at all, which is what the widget above does.

Three of those coefficients sit before the scheme existed, and under parallel trends they should be noise around zero. Suppose instead that they come back at −1.8, −1.2 and −0.6, marching steadily toward the reference. That is a differential trend of 0.6 points a term, visible three terms early, and it does not stop at term 4 simply because the scheme started there. The average term before the switch sits 2.5 terms back and the average term after it sits 1.5 terms forward, four terms apart, so four terms of a 0.6-point drift is 2.4 points sitting inside the estimate. Of the 2.60 the table reported, 0.2 would be left. The 2 × 2 table cannot show you this. It has only four numbers in it, and four numbers have no room to hold a slope.

Pre-trends are evidence rather than proof, and the honest version of that sentence has teeth. A pre-trend test can pass because the trends really are parallel, or because the pre-period is short, or because the outcome is noisy enough that a real drift cannot be distinguished from zero. Turn the noise slider up in the widget with a differential trend already set and watch the coefficients before term 5 widen until they cover zero comfortably, while the estimate stays exactly as wrong as it was. A design with three quiet pre-periods and wide error bars has not been tested; it has been left untested more politely. Say how many pre-periods there were and how precisely each was estimated, and a reader can judge for themselves.

Many periods, and treatments that start at different times

Two extensions turn up constantly. The first is easy: more than two periods, with everyone treated at the same moment. The two-way fixed-effects regression handles that unchanged, the event study becomes the natural way to report it, and the only new question is whether the effect grows, fades or holds across the post-treatment periods, which the event study answers by showing each one separately.

The second is not easy. Under staggered adoption, units join the treatment at different dates: a few districts in 2015, more in 2017, the rest in 2019. Running the same two-way fixed-effects regression feels like the obvious move, and for two decades it was the obvious move. It can be badly wrong. The reason is that with staggered timing the regression is silently averaging many small comparisons, and some of them use already-treated units as the comparison group for later-treated ones. When the effect changes over time, those comparisons subtract one group's treatment effect from another's, and they can enter the average with negative weight, which is enough to flip the sign of an estimate whose every underlying effect is positive.

A decomposition published by Goodman-Bacon in 2021 made that arithmetic explicit by writing the two-way estimate as a weighted average of every two-group, two-period comparison inside the data, weights and all, and it is now standard practice to report it. The replacements avoid the bad comparisons by construction: the Callaway and Sant'Anna estimator computes a separate effect for each adoption cohort at each period against units not yet treated, and the Sun and Abraham interaction-weighted estimator does the same repair inside an event study. Both are one command in R or Python. The reason to know their names now is defensive, since a paper running plain two-way fixed effects on staggered timing without addressing the issue is a paper with a question to answer.

What to report

A difference-in-differences result is a coefficient, and the coefficient is the easy part. What makes it readable is the sentence around it: which units were treated and when, what the comparison group is and why it is plausible, how the standard errors were clustered, how many pre-periods were available and what the event study did in them. Three of those five are about the design rather than the arithmetic, which is the right proportion. Like the instrument and the cutoff before it, this design buys its credibility with an assumption, and the assumption is the part a reader is entitled to see defended.