Post-Hoc Tests

A significant ANOVA tells you the groups aren't all equal, but not which ones differ. To find out, you compare the groups pairwise. The catch: every extra comparison is another roll of the dice for a false positive, and they add up fast. Post-hoc tests are how we go hunting for differences without fooling ourselves.

The multiple-comparisons problem

Each test at α = 0.05 has a 5% chance of a false positive when nothing's really there. Run one test and you're fine. But four groups means six pairwise comparisons; six groups means fifteen. With enough comparisons, the chance that at least one comes up "significant" by pure luck (the family-wise error rate) balloons toward certainty.

A contrast asks one specific question

Six comparisons is what you pay for arriving at the data with no question. Most studies arrive with one or two. A trial of four fertilizer doses, one of which is no fertilizer at all, was designed to find out whether feeding the seedlings helps, and perhaps whether more is better. Neither of those is a pair.

A contrast asks exactly the question the design had in mind. It is a weighted combination of the group means:

ψ = a₁μ₁ + a₂μ₂ + … + a_Iμ_I,   with Σa_i = 0

ψ (psi) is the quantity being estimated, and the contrast coefficients a_i carry the question. Making them sum to zero is what turns ψ into a comparison rather than a level: if all I population means were identical, ψ would come out at zero for any zero-sum coefficients, whatever that common mean happened to be.

Two shapes cover most of what a course asks for. Coefficients of [1, −1, 0, 0] subtract the second group's mean from the first and ignore the rest, which is an ordinary pairwise comparison written as a contrast. Coefficients of [1, −⅓, −⅓, −⅓] set the first group against the average of the other three, a complex contrast asking something no single pair can ask.

That second one is also written [3, −1, −1, −1], and the two really are the same contrast. Multiplying every coefficient by 3 multiplies the estimate by 3 and its standard error by 3, so the t ratio is untouched and the p-value with it. What changes is the units the estimate reads in. The fractional version estimates the gap between the control mean and the average of the other three, in the outcome's own units. The whole-number version estimates three times that gap, which is the version SPSS wants typed into its dialog, since whole numbers are easier to enter without rounding.

Estimating it, testing it, and putting an interval on it

Swap the population means for the sample means and you have the estimate, ψ̂ = Σa_ix̄_i. Its standard error borrows the error term from the ANOVA table rather than rebuilding one:

SE_ψ̂ = √( MS_E · Σ a_i²/n_i )

and the test statistic is that estimate over its standard error,

t = ψ̂ / SE_ψ̂,   on df = N − I

on the error degrees of freedom from the same table, not on the degrees of freedom the groups in the contrast would have supplied by themselves. This is the practical dividend the omnibus table pays. A comparison between two groups of four is tested on 12 degrees of freedom rather than 6, because MS_E was estimated from all sixteen seedlings.

The confidence interval uses the same two ingredients:

ψ̂ ± t* · SE_ψ̂

with t* the two-tailed critical value at your α on those degrees of freedom. A significant t and an interval that excludes zero are one fact stated twice, since both are the same inequality rearranged. Report the interval, because it carries a size and the t does not.

Worked, on the seedlings

The four fertilizer doses in Build the ANOVA Table gave mean heights of 13, 17, 19 and 23 centimeters on four seedlings each, with MS_E = 4.00 on 12 degrees of freedom. The planned question was whether any dose beats none, so the control takes a coefficient of 1 and the three fed groups share −1 between them:

ψ̂ = 13 − (17 + 19 + 23)/3 = 13 − 19.67 = −6.67 cm
Σa_i²/n_i = (1 + ⅛ + ⅛ + ⅛)/4 = ⅓
SE = √(4.00 × ⅓) = 1.15
t = −6.67 / 1.15 = −5.77 on 12 df, two-tailed p = .00009

With t*(12) = 2.179 the interval is −6.67 ± 2.179 × 1.15, or [−9.18, −4.15] centimeters. Fed seedlings out-grow unfed ones by somewhere between four and nine centimeters on average, and the reason that reads as a plain sentence is that the fractional coefficients kept the estimate in centimeters. Run the identical numbers as [3, −1, −1, −1] and you get ψ̂ = −20.00, SE = 3.46, the same t of −5.77, and the interval [−27.55, −12.45], which is the first interval multiplied by three.

A planned contrast is also one of the few places where a one-tailed test is not a dodge. Fertilizer was never going to shrink the seedlings, the direction was written into the design, and halving the two-tailed p to .00004 is honest arithmetic. The condition is the one thing a reader cannot check: the direction has to have been chosen before the data arrived, and the write-up has to say so.

Planned or post-hoc

The lecture's distinction is about timing and count, not about the formulas, which are identical either way.

Nothing in a set of coefficients tells a reader when it was chosen. A dated analysis plan is what makes a contrast planned, ideally a pre-registered one, and without that record the honest label for any comparison is post-hoc.

Every pair, one pooled error term

Exploratory follow-ups are contrasts too, one per pair, carrying 1 and −1 on the two groups involved and zero everywhere else. Put those coefficients through the formulas above, simplify, and what falls out is the version worth memorizing before an exam:

t = (x̄_i − x̄_j) / ( s_p √(1/n_i + 1/n_j) ),   s_p = √MS_E,   df = N − I

That is not the two-sample t-test run again on two of the groups. The two-sample test builds its spread out of the two groups in front of it and tests on n_i + n_j − 2 degrees of freedom. This one borrows s_p from all I groups, so the same comparison gets a better-estimated denominator and more degrees of freedom to test it against. What pays for that is the assumption the ANOVA already made, that every group is drawing from populations of equal spread. When that assumption fails, the borrowing fails with it, which is what Games–Howell further down is for.

On the seedlings, s_p = √4.00 = 2.00 and every pair has the same standard error, 2.00 × √(¼ + ¼) = 1.41, because the groups are the same size. Six comparisons, all on 12 degrees of freedom:

The Bonferroni column is the whole idea in one column. Six comparisons at a family-wise .05 means each must clear .05/6 = .0083, and the t that cuts off .0083 in two tails on 12 degrees of freedom is t** = 3.153, against the plain t* of 2.179. Two comparisons change their answer, both of them the ones sitting at t = 2.83 between the two bars. Doubling the fertilizer from Low to High is the difference the correction refuses to certify on four seedlings a dose.

The gap between the two critical values widens as the family grows and narrows as degrees of freedom accumulate, so it is far less punishing on a real study than on sixteen seedlings. Three comparisons on 73 error degrees of freedom need t** = 2.450 where the uncorrected critical value is 1.993, a bar about a quarter higher rather than half again.

Simultaneous confidence intervals

The interval version of the same correction swaps one critical value for the other:

(x̄_i − x̄_j) ± t** · s_p √(1/n_i + 1/n_j)

Built on t** rather than t*, the whole set of intervals holds at 95% together, which is what simultaneous means here: 95% of the time, every interval in the family covers its own true difference. For None versus Low that is −4.00 ± 3.153 × 1.41, or [−8.46, 0.46], and it contains zero exactly as the corrected test declined to reject. The ordinary 95% interval on the same pair, [−7.08, −0.92], does not contain zero. Both are correct answers to different questions, and a results section that prints corrected p-values beside uncorrected intervals is answering both at once without telling the reader which is which.

Two ways to write the same decision

Bonferroni has two reporting conventions, and they look different enough on the page to confuse a reader who has only met one of them.

The first divides α and leaves the p-value alone: t(73) = 6.62, p < .017, where .017 is .05/3 and the reader is being shown which bar this p had to clear. The second leaves α at .05 and multiplies the p-value by the number of comparisons instead, capping it at 1. The seedlings' None-versus-Low p of .0152 becomes 6 × .0152 = .091 that way, and .091 against .05 delivers the same verdict t** = 3.153 delivered.

SPSS prints the second. The Sig. column of its Multiple Comparisons table under Bonferroni already carries the multiplication, so it is read against .05 like any ordinary p-value, and dividing .05 by the number of comparisons before comparing would correct the same family twice. Whichever convention you use, name it, because the family size is not recoverable from the numbers and the two cannot be converted into each other without it.

How corrections fix it

The idea is to make each individual test more demanding so the overall error stays at 5%:

• Bonferroni is the simplest: just divide your α by the number of comparisons. Ten comparisons? Each must clear p < 0.005. Dead simple and always valid, but conservative (it can miss real differences).
• Tukey's HSD is purpose-built for comparing all pairs of means after ANOVA; more powerful than Bonferroni when you're doing every pairwise comparison.
• Holm is a step-down tweak on Bonferroni that's uniformly more powerful while still controlling the family-wise error.

Two cases Tukey does not cover

Tukey's HSD assumes what the ANOVA before it assumed: every group shares one spread, so every comparison can borrow one pooled error term. When the spreads differ, Games–Howell keeps the studentized range but gives each pair its own standard error and its own Welch–Satterthwaite degrees of freedom, the same fractional df a Welch t-test reports. On the site's study-methods.csv, where the three group SDs run 7.97, 8.86 and 11.97, the two procedures disagree in both directions:

Games–Howell is often described as the conservative choice, and the middle row shows why that is too simple. Flashcards and rereading are the two tight groups, so a comparison between them no longer has to carry the third group's larger variance, and the p-value improves fivefold. The comparison that does involve the noisy group gets more expensive instead. Pooling is an averaging, and Games–Howell stops averaging. The assumption-checks sheet carries the pairing on one printable page.

The second gap is a design, not an assumption. Some studies compare several treatments against one control and have no interest in whether the treatments differ from each other. Testing all pairs then corrects for comparisons nobody wanted, and Dunnett's test corrects for the ones you actually made, using the fact that every comparison shares the control group. With four groups and 20 error degrees of freedom, the two-sided .05 cutoff for a t-like statistic is 2.54 under Dunnett, against 2.61 under Bonferroni and 2.80 under Tukey. The gap widens as groups are added. Whichever you use, say so in the write-up: a reader cannot recover the correction from the p-values.

When 5% of your findings may be wrong

Family-wise correction answers a demanding question: what is the chance of even one false alarm anywhere in the family? For a handful of group comparisons that is exactly the right thing to protect. For a genomics screen across 20,000 genes, or a brain-imaging analysis across 100,000 voxels, insisting on it leaves almost nothing standing. So those fields changed the question they were asking.

The false discovery rate asks instead: among the results I do call significant, what share should I expect to be wrong? The Benjamini–Hochberg procedure controls it. Sort your m p-values from smallest to largest and compare the ith one against i/m × q, where q is the false-discovery rate you are willing to live with (usually .05). Find the largest i that clears its own bar, and call that result and every smaller one significant.

Ten p-values make the difference concrete. Bonferroni asks all ten to clear .005. Benjamini–Hochberg asks the smallest to clear .005, the second .010, the third .015, and so on up. Given p = .001, .008, .012 and nothing else below .02, Bonferroni returns one finding and Benjamini–Hochberg returns three, at a cost stated openly in advance: about 5% of those three are expected to be false.

Problem 14 of the practice problems takes the significant ANOVA from Problem 13 and hunts down where the difference actually lives, by Bonferroni and by Tukey’s HSD, on the same three group means. Problem 25 works a control-versus-treatments contrast from four groups’ standard deviations through to its confidence interval, and Problem 26 runs three pairwise comparisons on one pooled error term and writes the corrected result up.