Reading a Causal-Inference Paper

Advanced courses often end with a journal club. Someone hands out a paper that claims a cause, everyone reads it, and an hour later you are expected to say something useful about it. What is being tested is not whether you can run the analysis but whether you can referee a design, and refereeing has its own order of steps. This guide gives that order as seven questions. It then applies them to an invented study whose numbers you can check, and ends with a checklist short enough to take into the room.

1 · What is the causal question?

Start by writing the paper's question in your own words as a comparison between two states of the world. Write a comparison, not a topic or a pair of variables: these units, under this treatment, against the same units without it, over this period. A paper that estimates the effect of a tutoring program on test scores is asking what a given child's score would have been without the program. Causal designs exist because that second score is never observed. Causal DAGs & Confounding sets out this vocabulary.

Read the abstract's wording closely. "We examine the relationship between" is a weaker claim than "we estimate the effect of", and papers sometimes make the first claim in the abstract and the second in the conclusion. Note which claim the paper makes. Everything after this step tests whether the design supports the stronger one.

Your sentence needs four things: the treatment, described precisely enough that you could administer it, the units, the comparison, and the time horizon. If the paper leaves one out, you have to work it out yourself. A policy measured after six months and the same policy measured after six years answer different questions.

2 · Which of the three designs is it?

Almost every causal claim from observational data in this literature rests on one of three designs, and the methods section usually describes it in a single paragraph. An instrumental variable is a variable that the authors claim shifts the treatment but has no direct effect on the outcome. A regression discontinuity uses a strict rule with a cutoff on a continuous score. A difference-in-differences uses a policy that arrived in some places at a known time and not in others.

Find that paragraph before you read anything else, because it tells you which assumption to spend the hour on. If you cannot find it, the paper may be doing plain covariate adjustment and calling the result an effect. That is a fourth and much weaker approach, and then the question is which controls are included and which are missing.

3 · Where the variation comes from

Each of these designs relies on some part of the treatment that was assigned for a reason unrelated to the outcome. Find that part. Which units were treated differently, at which moment, and why? The answer should be a short account of something that happened: a lottery was drawn, a rule was written, a budget ran out, a border moved, a legislature chose a date for reasons of its own.

Then ask who could have influenced that reason, and whether they had a motive to. A rule written by an agency whose staff also deliver the treatment is not the same as a rule written into a statute years earlier. An eligibility score a caseworker can round up is not the same as an exam mark typed in by a machine. A policy date chosen because outcomes were already deteriorating is the worst case of all, because the choice was made using the very thing the design is trying to measure.

Also ask who is not in the comparison. Designs like these usually discard most of the sample: everyone far from the cutoff, everyone the instrument did not move, everyone in a region with no policy change. That is legitimate, but the paper should state it rather than leave you to infer it from a sample-size row.

4 · The assumption, and its defense

Spend most of the hour on this step. Write the identifying assumption in one plain sentence, in the paper's own setting, with no symbols. For the homework-club study further down, it reads: living near a club site changes a child's test score only by getting them into the club, and in no other way. Once it is in words, everyone in the room can argue about whether it is true.

The assumption cannot be tested. It is a statement about an outcome that was never observed, so no dataset contains the column that would settle it, and no diagnostic, goodness-of-fit measure or residual plot says anything about it. All three lessons make this point in their own terms. It is why a causal paper has to be refereed and cannot simply be checked.

Instead, the paper should put the assumption where it could visibly fail. There are four ways to do this, so note which ones the paper uses.

• An account of the variation good enough that a skeptic can attack the story rather than the statistics.
• Balance on things fixed before the treatment existed, which the treatment cannot have caused.
• A falsification: a sample, a period or a cutoff where the assumption implies nothing should happen.
• A sensitivity calculation saying how large a violation would have to be before the conclusion changes.

Then judge each defense. A falsification test that comes back non-significant tells you something only if it could have caught the violation that matters. Work out the smallest effect it could detect, compare that with the size of violation that would overturn the result, and you will often find the test could not have failed. The worked study below is such a case, and the arithmetic takes one line.

Balance tables are open to a related misreading. A row with a large standard error and a small t does not show that the difference is zero. It shows that the data cannot tell. Read the point estimates, ask whether the imbalances point the same way, and ask which way they would push the headline estimate. An imbalance that works against the paper's finding matters much less than one that works in its favor, and papers rarely say which kind they have.

5 · Reading the tables

Read the tables in a different order from the one the paper uses. The headline table is the one the abstract describes. The tables that tell you whether to believe it usually come before it or are in an appendix. Read those first, and come to the headline coefficient last.

An instrumental-variables paper should show three things. First, the first stage with its F statistic, which tests relevance and nothing else. Second, the reduced form, the plain regression of the outcome on the instrument, which contains the whole finding before any division. Third, the ordinary least squares estimate beside the two-stage one, so you can see how far apart they are and in which direction. A paper that reports only the two-stage coefficient hides the arithmetic. A ratio can be large because its numerator is large or because its denominator is small, and those lead to very different conclusions.

Balance and placebo tables are where the assumption is tested in public, so read them closely. For robustness columns, work out what changes from one column to the next, then ask whether the specification closest to the identifying concern is also the weakest one. That pattern is common and rarely pointed out, because authors seldom draw attention to the column that weakens their result.

Two more checks cover most of the rest. Check that the sample size is the same across columns that claim to be comparable, because a specification that drops a third of the data without comment has changed more than its controls. And check whether the standard errors were clustered at the level where the treatment was assigned. For a policy that arrived by region, that means the region, not the person. Difference-in-Differences explains why that choice can change a standard error by a factor of two or three.

6 · The magnitude, and whose it is

Read the coefficient aloud as a sentence, in the units of the outcome. Then convert it into something easier to picture: a share of the outcome's standard deviation, a fraction of a year of progress, a percentage of the baseline mean. Significance stars reflect the standard error and say nothing about whether the effect is large enough to matter.

Treat an implausibly large estimate as a warning sign. Designs like these often produce large numbers, partly because they estimate the effect for a particular subgroup and partly because a noisy estimate has to be large to reach significance. If an after-school club appears to raise scores by two thirds of a standard deviation, suspect the design before the club.

Then say whose effect it is, because none of these designs estimates the average effect in the population. An instrument recovers the average among the compliers, the units it actually moved. A cutoff recovers the effect for units at the cutoff. A difference-in-differences recovers the effect on the treated units in the periods they were treated. The paper should contain a sentence of the form "this is the effect of X on Y, among Z". If it does not, write that sentence yourself before the seminar and see whether anyone disagrees.

7 · What would change your mind

Before you speak, name the one finding that would reverse your verdict. If you believe the paper, name the result that would change your mind: a covariate that jumps at the cutoff, a pre-trend, a comparison group that gives a different answer. If you doubt it, name the result that would convince you.

This turns a summary into a discussion. It also shows you when nothing would change your mind. In that case you are not refereeing the paper but reporting a prior belief, and you should tell the room so. And it gives the authors, if they are ever in the room, something they can act on, which a list of complaints does not.

The standing objections, by design

Ask these as questions, not accusations. Each has a good answer available, and asking gives the presenter the chance to give it.

Instrumental variables

• Where did the instrument come from, and who decided it?
• What is the first stage, and what does the reduced form look like on its own?
• Which direct routes from the instrument to the outcome have you ruled out, and on what evidence?
• Who are the compliers here, and roughly what share of the sample are they?
• If the instrument is a policy, is anything else in that policy also touching the outcome?

Regression discontinuity

• What does the density of the running variable do at the cutoff?
• Which pre-treatment covariates jump there, and how precisely were they measured?
• How does the estimate move across bandwidths, and does the specification force one slope across the cutoff?
• Is this threshold used by any other rule or program?
• How many distinct values does the running variable take near the cutoff?

Difference-in-differences

• How many pre-treatment periods are there, and how precisely was each one estimated?
• Why this comparison group, and what happens with a different one?
• Did units adopt at different dates, and if so which estimator was used?
• Levels or logs, and does the answer survive the other choice?
• What else happened at that date in the treated units alone?

The route walked once

The paper. A city has run free after-school homework clubs for decades, sited in the 1970s under a population rule and unchanged since. Using administrative records on 6,000 children aged 8 to 11, the authors ask whether attending a club raises the end-of-year test score. Attendance is voluntary, so they instrument it with living within 800 meters of a club site. The outcome is scored out of 100 with a standard deviation of 15 in this sample.

Question 1. The comparison is a child who attended a club against the same child in a year where they did not, measured at the end of that school year. The paper says "effect of attendance", so it makes the stronger claim.

Question 2. Instrumental variables. The 800-meter cutoff might look like a discontinuity, but no rule allocates anything at 800 meters. It is the authors' own dividing line on a continuous distance, used to define the instrument.

Question 3. The variation comes from how far a family happens to live from buildings sited half a century ago. The sites were chosen in the 1970s, not by anyone involved today, and that helps the design. But families do choose where to live, and that is the concern.

Question 4. The assumption in words: being near a site changes a child's score only by getting them into the club. Here is the paper's balance table, its main defense.

The paper reports that the previous year's score is balanced, t = 0.86, and moves on. Look at the number, not the verdict. Children near a site scored 0.34 points higher before the club year began, and the income gap is the right size to produce that: 1.10 thousand at a gradient of roughly 0.30 points per thousand gives 0.33. So the row the paper finds most reassuring is consistent with a direct path from the instrument to the outcome worth about a third of a point. Keep that number in mind.

The paper's falsification test is well chosen. Older siblings in the same households, aged 13 to 15 and never eligible for the clubs, show a near-versus-far score difference of +0.21 with a standard error of 0.58, t = 0.36. If distance were a proxy for neighborhood quality, it should move their scores too, and it does not. But the test is weaker than it looks. To reach significance it needed a gap of about 1.14 points, and the violation that matters is 0.34, which would have shown up as t = 0.59. The test passes, but it could not have failed.

Question 5. Next, the tables, read in the order given in step 5.

The first stage is very strong, but it addresses only the easiest of the three conditions, relevance. An F of 558 does not validate the design, and treating it as if it did confuses relevance with exclusion. The finding itself is in the reduced form: 1.12 points with a standard error of 0.395, a modest but statistically clear result, t(5998) = 2.83.

Now use the 0.34 from the balance table. If a third of a point of that 1.12 came by a route other than attendance, the numerator should be 0.78, and dividing by the same first stage gives 2.79 rather than 4.00. The bias is 0.34 divided by 0.280, or 1.21 points. A weaker instrument would have made it larger: against a first stage of .08, the same leak would have been worth 4.25 points, more than the entire reported effect. Bring that calculation to the seminar.

Read the specifications in order of how directly each addresses the concern, not from top to bottom. Adding household controls moves the estimate down toward the 2.79 that the sensitivity calculation predicted, and that supports the diagnosis. Neighborhood fixed effects lower it further and widen the interval enough to include zero. The paper's abstract quotes specification 1.

Question 6. Four points on a test with a standard deviation of 15 is 0.27 SD. The paper describes a year of ordinary progress on this test as about 11 points, so the club adds about a third of a school year. That is a substantial claim but a plausible one. The naive comparison of 9.00 points, 0.60 SD, would be implausibly large, and that shows how much selection into an optional program inflates a simple comparison.

The first stage also shows whose effect it is. Seventeen percent of children attend wherever they live, 55% do not attend at all, and the 28% in between are the compliers. These are children who went because a site was close and would not have gone otherwise. The 4.00 applies to those 1,680 children only. A city deciding whether to build more sites is asking about that group, so here it is the policy-relevant number. It does not tell you what clubs do for children in general.

Living within 800 meters of a club site raised attendance by 28.0 percentage points, first-stage F = 557.6, and was associated with a 1.12-point difference in end-of-year scores, t(5998) = 2.83, p = .005. The implied effect of attendance is 4.00 points, SE = 1.41, 95% CI [1.24, 6.76], or 0.27 standard deviations, among the 28% of children whose attendance the proximity of a site changed.

Question 7. Two results would change the verdict. A falsification test with the power to detect a third of a point, on a sample where the assumption implies an effect of zero, would make the imbalance unlikely to matter. And a neighborhood-fixed-effects estimate that stayed near 4.00, rather than falling to 2.85, would show that the income gradient is not driving the result. In the other direction, an estimate that moved toward zero as the distance threshold tightened would point to the neighborhood and not the club, since a tighter ring should be more like its surroundings, not less. The verdict to give the room: the design is plausible, the finding is probably real, and the reported effect is probably too large by somewhere between a quarter and a half.

Keep going

• Instrumental Variables & 2SLS: relevance, independence and exclusion, the Wald ratio, weak instruments and compliers.
• Regression Discontinuity Design: the running variable, the bandwidth decision, sharp and fuzzy cutoffs, and the three checks.
• Difference-in-Differences: four cells, the interaction coefficient, clustered standard errors, event studies and staggered adoption.
• Causal DAGs & Confounding: the vocabulary of backdoor paths and colliders used throughout this guide.
• Quasi-Experiments & Natural Experiments: other designs that rely on accidents of assignment, from the Research Toolkit.
• Practice Problems: the Stats 3 set works an encouragement design, a cutoff and a two-by-two policy table by hand.
• One Study, Start to Finish: a project where you are the one making the claim.