Meta-Analysis & Forest Plots
No single study settles a question. Samples are small, contexts differ, and chance is loud. Meta-analysis treats the studies themselves as data points: pool every estimate of an effect, weight each by its precision, and out comes the field's best combined answer, plus something just as valuable: a measure of how much the studies genuinely disagree.
The one rule: weight by precision
A 400-person trial should count for more than a 30-person pilot. Meta-analysis formalizes this with inverse-variance weighting: each study's weight is 1/SE², so tight studies (small standard errors, from the usual n law) dominate and noisy ones contribute what little they know. Before any of that can happen the studies have to be on one scale, which usually means converting each result to a common effect size first; the effect-size converter handles the arithmetic when a literature reports a mix of d, r and odds ratios. The pooled effect is just the weighted average, and its standard error shrinks as evidence accumulates: the whole literature behaving like one giant study.
Reading a forest plot
The standard picture is the forest plot: one row per study, a square at its effect estimate (sized by weight), whiskers for its 95% CI, and a diamond at the bottom for the pooled effect; the diamond's width is its confidence interval. One glance tells you the effect sizes, who was precise, and whether the studies even agree.
🎮 Build Your Own Literature
Each row is a simulated study of the same true effect μ (standardized mean difference). Crank up between-study heterogeneity τ and watch the studies fan out, I² climb, and the random-effects diamond widen while the fixed-effect one stays overconfident.
Fixed vs random effects (and I²)
A fixed-effect model assumes every study estimates the same true effect, differing only by sampling noise. A random-effects model allows the true effect to vary across studies (different populations, doses, protocols) with between-study standard deviation τ. The Q statistic tests whether studies scatter more than chance allows, and I² converts it to a percentage: roughly, how much of the visible spread is real disagreement rather than sampling noise. When I² is large, the random-effects model (with its honestly wider diamond) is the defensible summary, and the interesting question becomes why the effect varies (moderator analysis, the meta-level cousin of interactions).
The diamond is not a forecast
The diamond answers one question: where is the average true effect? It says nothing about where the next study will land, and readers routinely take it to mean both. Once the studies genuinely disagree, those two answers are far apart.
The second question has an interval of its own. A prediction interval covers the plausible true effect of a new study drawn from the same population of studies, so it puts the between-study spread back in:
PI = d̂ ± t(k − 2) × √(τ̂² + SE(d̂)²)
Take a literature shaped like the playground's defaults: k = 8 studies, a pooled d of 0.40 with a standard error of 0.10, and τ̂ = 0.20, which is an I² of about 50%. The diamond runs from 0.20 to 0.60, a comfortably positive average effect. The 95% prediction interval runs from −0.15 to 0.95. A new trial finding nothing whatsoever would be entirely consistent with this literature, and the forest plot as usually drawn gives no hint of that.
The interval is wider than the distribution of true effects it describes (which spans 0.01 to 0.79 here) because μ and τ are both estimated, and with 6 degrees of freedom the multiplier is 2.45 rather than 1.96. And that same shortage of studies is what makes τ̂ shaky in the first place, so a prediction interval built on five studies is honest about its own ignorance in a way that can look almost useless. That is information too: it's the width a literature this thin has earned. In R, metafor's predict() prints the interval beside the pooled estimate.
The elephant: publication bias
Meta-analysis can only pool what got published — and significant results get published more. If the null results are sitting in file drawers, your pooled estimate is built from a flattering sample of the evidence. Funnel plots, Egger's test, and trim-and-fill are the standard diagnostics; the deeper fixes are cultural, and they are the ones the replication crisis forced on the field: preregistration and registered reports, which get a study into the record before its result is known. A meta-analysis is only as unbiased as the literature it feeds on, the same "how did this sample come to be?" question you've asked since lesson 1.1.
Why it matters: single studies whisper; literatures speak. Meta-analysis turns a pile of noisy, conflicting results into a weighted answer with honest uncertainty, and I² tells you when "the studies disagree" is itself the headline finding.
Common questions
What does I² tell you in a meta-analysis?
The share of visible between-study variation that reflects real differences in effects rather than sampling noise. Rough bands: 25% low, 50% moderate, 75% high heterogeneity. High I² is a finding, not a defect: the effect genuinely varies across populations or protocols, the fixed-effect summary is too confident, and the interesting question becomes what moderates the effect.
How many studies do I need for a meta-analysis?
Two will produce a number, so the arithmetic is never the constraint. What breaks down with few studies is the part that matters: with fewer than about five, the between-study variance τ² is estimated so poorly that I² and the width of a random-effects interval become close to guesswork, and the pooled result inherits that uncertainty without showing it. Somewhere around ten the estimates start to behave, and moderator analyses need considerably more than that, since each moderator is effectively a regression on a sample of studies. The Hartung-Knapp adjustment is the usual defense for a small set, because it widens the interval to reflect how badly τ² is pinned down. The real constraint, though, is comparability rather than count: five studies asking the same question pool honestly, while twenty measuring subtly different things produce a precise average of nothing in particular.
What is a funnel plot and what does asymmetry mean?
Each study plotted as effect size vs. precision: big precise studies cluster at the top, small noisy ones fan out below — symmetrically, if all results reached publication. A missing lower corner (typically small null studies) suggests publication bias, testable with Egger's regression and probed with trim-and-fill. Caveat: asymmetry has innocent causes too, like small studies using different populations.