Meta-Analysis & Forest Plots
No single study settles a question. Samples are small, contexts differ, and chance is loud. Meta-analysis treats the studies themselves as data points: pool every estimate of an effect, weight each by its precision, and out comes the field's best combined answer, plus something just as valuable: a measure of how much the studies genuinely disagree.
The one rule: weight by precision
A 400-person trial should count for more than a 30-person pilot. Meta-analysis formalizes this with inverse-variance weighting: each study's weight is 1/SE², so tight studies (small standard errors, from the usual n law) dominate and noisy ones contribute what little they know. Before any of that can happen the studies have to be on one scale, which usually means converting each result to a common effect size first; the effect-size converter handles the arithmetic when a literature reports a mix of d, r and odds ratios. The pooled effect is just the weighted average, and its standard error shrinks as evidence accumulates: the whole literature behaving like one giant study.
Reading a forest plot
The standard picture is the forest plot: one row per study, a square at its effect estimate (sized by weight), whiskers for its 95% CI, and a diamond at the bottom for the pooled effect; the diamond's width is its confidence interval. One glance tells you the effect sizes, who was precise, and whether the studies even agree.
🎮 Build Your Own Literature
Each row is a simulated study of the same true effect μ (standardized mean difference). Crank up between-study heterogeneity τ and watch the studies fan out, I² climb, and the random-effects diamond widen while the fixed-effect one stays overconfident.
Fixed vs random effects (and I²)
A fixed-effect model assumes every study estimates the same true effect, differing only by sampling noise. A random-effects model allows the true effect to vary across studies (different populations, doses, protocols) with between-study standard deviation τ. The Q statistic tests whether studies scatter more than chance allows, and I² converts it to a percentage: roughly, how much of the visible spread is real disagreement rather than sampling noise. When I² is large, the random-effects model (with its honestly wider diamond) is the defensible summary, and the interesting question becomes why the effect varies (moderator analysis, the meta-level cousin of interactions).
The elephant: publication bias
Meta-analysis can only pool what got published — and significant results get published more. If the null results are sitting in file drawers, your pooled estimate is built from a flattering sample of the evidence. Funnel plots, Egger's test, and trim-and-fill are the standard diagnostics; the deeper fixes are cultural, and they are the ones the replication crisis forced on the field: preregistration and registered reports, which get a study into the record before its result is known. A meta-analysis is only as unbiased as the literature it feeds on, the same "how did this sample come to be?" question you've asked since lesson 1.1.
Why it matters: single studies whisper; literatures speak. Meta-analysis turns a pile of noisy, conflicting results into a weighted answer with honest uncertainty, and I² tells you when "the studies disagree" is itself the headline finding.
Common questions
What does I² tell you in a meta-analysis?
The share of visible between-study variation that reflects real differences in effects rather than sampling noise. Rough bands: 25% low, 50% moderate, 75% high heterogeneity. High I² is a finding, not a defect: the effect genuinely varies across populations or protocols, the fixed-effect summary is too confident, and the interesting question becomes what moderates the effect.
How many studies do I need for a meta-analysis?
Two will produce a number, so the arithmetic is never the constraint. What breaks down with few studies is the part that matters: with fewer than about five, the between-study variance τ² is estimated so poorly that I² and the width of a random-effects interval become close to guesswork, and the pooled result inherits that uncertainty without showing it. Somewhere around ten the estimates start to behave, and moderator analyses need considerably more than that, since each moderator is effectively a regression on a sample of studies. The Hartung-Knapp adjustment is the usual defense for a small set, because it widens the interval to reflect how badly τ² is pinned down. The real constraint, though, is comparability rather than count: five studies asking the same question pool honestly, while twenty measuring subtly different things produce a precise average of nothing in particular.
What is a funnel plot and what does asymmetry mean?
Each study plotted as effect size vs. precision: big precise studies cluster at the top, small noisy ones fan out below — symmetrically, if all results reached publication. A missing lower corner (typically small null studies) suggests publication bias, testable with Egger's regression and probed with trim-and-fill. Caveat: asymmetry has innocent causes too, like small studies using different populations.