Post-Hoc Tests
A significant ANOVA tells you the groups aren't all equal — but not which ones differ. To find out, you compare the groups pairwise. The catch: every extra comparison is another roll of the dice for a false positive, and they add up fast. Post-hoc tests are how we go hunting for differences without fooling ourselves.
The multiple-comparisons problem
Each test at α = 0.05 has a 5% chance of a false positive when nothing's really there. Run one test and you're fine. But four groups means six pairwise comparisons; six groups means fifteen. With enough comparisons, the chance that at least one comes up "significant" by pure luck (the family-wise error rate) balloons toward certainty.
🎮 How Errors Pile Up
Every group here is truly identical — any "significant" pair is a false alarm. Each experiment tests all pairs at once: on the left every pair is tested at α = .05, on the right the same p-values must clear the stricter Bonferroni bar. Red tiles are the lies your data just told you.
How corrections fix it
The idea is to make each individual test more demanding so the overall error stays at 5%:
- Bonferroni is the simplest: just divide your α by the number of comparisons. Ten comparisons? Each must clear p < 0.005. Dead simple and always valid, but conservative (it can miss real differences).
- Tukey's HSD is purpose-built for comparing all pairs of means after ANOVA; more powerful than Bonferroni when you're doing every pairwise comparison.
- Holm is a step-down tweak on Bonferroni that's uniformly more powerful while still controlling the family-wise error.
The trade-off: corrections trade some power for protection against false alarms. Be stingier (Bonferroni) and you'll miss some real effects; be lenient and you'll cry wolf. The right choice depends on how costly a false positive is in your context.
That lost power is recoverable, but only before you collect the data. If you know you will run six comparisons, size the study for the corrected α rather than for .05: the power calculator will tell you what that costs in participants.
When 5% of your findings may be wrong
Family-wise correction answers a demanding question: what is the chance of even one false alarm anywhere in the family? For a handful of group comparisons that is exactly the right thing to protect. For a genomics screen across 20,000 genes, or a brain-imaging analysis across 100,000 voxels, insisting on it leaves almost nothing standing. So those fields changed the question they were asking.
The false discovery rate asks instead: among the results I do call significant, what share should I expect to be wrong? The Benjamini–Hochberg procedure controls it. Sort your m p-values from smallest to largest and compare the ith one against i/m × q, where q is the false-discovery rate you are willing to live with (usually .05). Find the largest i that clears its own bar, and call that result and every smaller one significant.
Ten p-values make the difference concrete. Bonferroni asks all ten to clear .005. Benjamini–Hochberg asks the smallest to clear .005, the second .010, the third .015, and so on up. Given p = .001, .008, .012 and nothing else below .02, Bonferroni returns one finding and Benjamini–Hochberg returns three, at a cost stated openly in advance: about 5% of those three are expected to be false.
Which one you want depends on what a mistake costs. Controlling the family-wise rate suits confirmatory work where one false claim discredits the paper. Controlling the false discovery rate suits screening, where the output is a shortlist that something else will check anyway. Say which you used; the two are not interchangeable, and a reader cannot tell from the p-values alone.
Planned vs. post-hoc
If you decided before seeing the data to test just a couple of specific comparisons (planned contrasts), you need little or no correction — you're not fishing. The heavy correction is the price of exploring all pairs after the fact, which is exactly what "post-hoc" (Latin for "after this") means.
Why it matters: the multiple-comparisons problem is everywhere — A/B testing, genomics, brain imaging, dredging survey data. Understanding why error inflates, and how to control it, is one of the most practically important ideas in all of statistics.
Problem 14 of the practice problems takes the significant ANOVA from Problem 13 and hunts down where the difference actually lives, by Bonferroni and by Tukey’s HSD, on the same three group means.
Common questions
Which post-hoc test should I use?
For all pairwise comparisons after a standard ANOVA, Tukey's HSD is the purpose-built choice. If you're only making a few pre-planned comparisons, Bonferroni (or better, Holm) is simple and valid. When group variances are clearly unequal, use Games–Howell, which doesn't assume homogeneity. The common thread: every option pays for extra comparisons with a stricter per-test bar.
Do I have to correct for every test in my whole paper?
No, and nobody does. A correction protects a family of tests, and defining the family is a judgment you make and then state, not something the software decides. The workable convention is that a family is a set of tests bearing on one question: all pairwise comparisons within one ANOVA, or the several outcomes you would treat as interchangeable evidence for a single claim. Tests answering genuinely separate questions belong to separate families. The rule that keeps this honest is the one on timing: decide the boundaries when you plan the analysis, because a family redrawn after seeing which p-values landed where is just p-hacking with extra arithmetic.
Do I need a significant ANOVA before running post-hoc tests?
Tradition says yes, and it's a sensible discipline against fishing. Strictly, though, tests like Tukey's HSD control the family-wise error on their own — they don't need the omnibus F as a gatekeeper, and the two can occasionally disagree (a significant Tukey pair under a non-significant F, or vice versa). Follow your field's convention, but know the protection comes from the correction, not the F.