Post-Hoc Tests

A significant ANOVA tells you the groups aren't all equal, but not which ones differ. To find out, you compare the groups pairwise. But every extra comparison is another chance of a false positive, and the chances add up fast. Post-hoc tests make those comparisons while keeping the overall false-positive rate under control.

The multiple-comparisons problem

Each test at α = 0.05 has a 5% chance of a false positive when there is no real difference. With one test, that 5% is the whole risk. But four groups means six pairwise comparisons, and six groups means fifteen. With enough comparisons, the chance that at least one comes up "significant" by luck alone (the family-wise error rate) approaches certainty.

A contrast asks one specific question

Four groups give six possible pairs, but most studies are designed to answer one or two specific questions. A trial of four fertilizer doses, one of which is no fertilizer at all, was designed to find out whether feeding the seedlings helps, and perhaps whether more is better. Neither of those is a pair.

A contrast is built to test the question the design had in mind. It is a weighted combination of the group means:

ψ = a₁μ₁ + a₂μ₂ + … + a_Iμ_I,   with Σa_i = 0

ψ (psi) is the quantity being estimated, and the contrast coefficients a_i define the question. The coefficients must sum to zero so that ψ measures a difference between means and not their overall level. If all I population means were identical, ψ would then be zero, whatever that common mean was.

Two shapes cover most of what a course asks for. Coefficients of [1, −1, 0, 0] subtract the second group's mean from the first and ignore the rest, which is an ordinary pairwise comparison written as a contrast. Coefficients of [1, −⅓, −⅓, −⅓] set the first group against the average of the other three, a complex contrast asking something no single pair can ask.

That second one is also written [3, −1, −1, −1], and the two are the same contrast. Multiplying every coefficient by 3 multiplies the estimate by 3 and its standard error by 3, so the t ratio and the p-value do not change. Only the units of the estimate change. The fractional version estimates the gap between the control mean and the average of the other three, in the outcome's own units. The whole-number version estimates three times that gap. It is the one to type into SPSS's dialog, because whole numbers can be entered without rounding.

Estimating it, testing it, and putting an interval on it

Swap the population means for the sample means and you have the estimate, ψ̂ = Σa_ix̄_i. Its standard error uses the error term from the ANOVA table:

SE_ψ̂ = √( MS_E · Σ a_i²/n_i )

and the test statistic is that estimate over its standard error,

t = ψ̂ / SE_ψ̂,   on df = N − I

The degrees of freedom are the error df from the same table, not the df the groups in the contrast would have by themselves. Because MS_E comes from all sixteen seedlings, a comparison between two groups of four gets 12 degrees of freedom. Its own two groups would give only 6.

The confidence interval uses the same two ingredients:

ψ̂ ± t* · SE_ψ̂

Here t* is the two-tailed critical value at your α on those degrees of freedom. A significant t and an interval that excludes zero always go together, because both come from the same inequality rearranged. Report the interval, because it shows the size of the difference and the t does not.

Worked, on the seedlings

The four fertilizer doses in Build the ANOVA Table gave mean heights of 13, 17, 19 and 23 centimeters on four seedlings each, with MS_E = 4.00 on 12 degrees of freedom. The planned question was whether any dose beats none, so the control takes a coefficient of 1 and the three fed groups share −1 between them:

ψ̂ = 13 − (17 + 19 + 23)/3 = 13 − 19.67 = −6.67 cm
Σa_i²/n_i = (1 + ⅛ + ⅛ + ⅛)/4 = ⅓
SE = √(4.00 × ⅓) = 1.15
t = −6.67 / 1.15 = −5.77 on 12 df, two-tailed p = .00009

With t*(12) = 2.179 the interval is −6.67 ± 2.179 × 1.15, or [−9.18, −4.15] centimeters. Fed seedlings outgrow unfed ones by between about four and nine centimeters on average. The fractional coefficients kept the estimate in centimeters, so the interval reads directly in those units. Run the same numbers as [3, −1, −1, −1] and you get ψ̂ = −20.00, SE = 3.46, the same t of −5.77, and the interval [−27.55, −12.45]. That is the first interval multiplied by three.

A planned contrast is also one of the few places where a one-tailed test is justified. Fertilizer was never going to shrink the seedlings, and the direction was written into the design, so you may halve the two-tailed p to get .00004. A reader cannot check this from the results, so the direction has to have been chosen before the data arrived, and the write-up has to say so.

Planned or post-hoc

The difference between the two is about timing and number. The formulas are identical either way.

The coefficients themselves do not show when they were chosen. A contrast counts as planned only if it is in a dated analysis plan, ideally a pre-registered one. Without that record, call every comparison post-hoc.

Every pair, one pooled error term

Exploratory follow-ups are contrasts too, one per pair, with 1 and −1 on the two groups involved and zero everywhere else. Put those coefficients through the formulas above and simplify, and you get the version to memorize for an exam:

t = (x̄_i − x̄_j) / ( s_p √(1/n_i + 1/n_j) ),   s_p = √MS_E,   df = N − I

This is not the two-sample t-test run again on two of the groups. The two-sample test estimates its spread from the two groups being compared and tests on n_i + n_j − 2 degrees of freedom. This one takes s_p from all I groups, so the same comparison gets a better-estimated denominator and more degrees of freedom. It relies on the assumption the ANOVA already made, that all the populations have equal spread. When that assumption fails, use Games–Howell, further down.

On the seedlings, s_p = √4.00 = 2.00 and every pair has the same standard error, 2.00 × √(¼ + ¼) = 1.41, because the groups are the same size. Six comparisons, all on 12 degrees of freedom:

The Bonferroni column shows the whole idea. Six comparisons at a family-wise .05 means each must reach .05/6 = .0083. The t that cuts off .0083 in two tails on 12 degrees of freedom is t** = 3.153, against the plain t* of 2.179. Two comparisons change their answer, the two with t = 2.83, between the two critical values. With the correction, four seedlings a dose are not enough to say that Low beats None or that High beats Medium.

The gap between the two critical values widens as the family grows and narrows as the error degrees of freedom increase. So the correction costs much less in a real study than with sixteen seedlings. Three comparisons on 73 error degrees of freedom need t** = 2.450, against an uncorrected 1.993. That critical value is about a quarter higher, where the seedlings' was almost half again as high.

Simultaneous confidence intervals

The interval version of the same correction swaps one critical value for the other:

(x̄_i − x̄_j) ± t** · s_p √(1/n_i + 1/n_j)

Built on t** rather than t*, the whole set of intervals holds at 95% together. This is called simultaneous coverage: 95% of the time, every interval in the family covers its own true difference. For None versus Low the interval is −4.00 ± 3.153 × 1.41, or [−8.46, 0.46]. It contains zero, which matches the corrected test's decision not to reject. The ordinary 95% interval on the same pair, [−7.08, −0.92], does not contain zero. Both are correct answers to different questions. A results section that prints corrected p-values beside uncorrected intervals mixes the two without telling the reader which is which.

Two ways to write the same decision

Bonferroni has two reporting conventions, and they look different enough to confuse anyone who knows only one of them.

The first divides α and leaves the p-value alone: t(73) = 6.62, p < .017, where .017 is .05/3, the threshold this p had to be under. The second leaves α at .05 and instead multiplies the p-value by the number of comparisons, capping it at 1. That way, the seedlings' None-versus-Low p of .0152 becomes 6 × .0152 = .091, and .091 against .05 gives the same decision as t** = 3.153.

SPSS prints the second. Under Bonferroni, the Sig. column of its Multiple Comparisons table already includes the multiplication, so compare it with .05 like any ordinary p-value. Dividing .05 by the number of comparisons as well would correct the same family twice. Whichever convention you use, name it. The family size cannot be recovered from the numbers, and without it the two conventions cannot be converted into each other.

How corrections fix it

The idea is to make each individual test more demanding so the overall error stays at 5%:

• Bonferroni is the simplest: divide your α by the number of comparisons, so with ten comparisons each must reach p < 0.005. It is simple and always valid, but conservative (it can miss real differences).
• Tukey's HSD is designed for comparing all pairs of means after ANOVA, and it is more powerful than Bonferroni when you make every pairwise comparison.
• Holm is a step-down version of Bonferroni that is uniformly more powerful and still controls the family-wise error.

Two cases Tukey does not cover

Tukey's HSD makes the same assumption as the ANOVA before it: all the groups share one spread, so every comparison can use one pooled error term. When the spreads differ, Games–Howell keeps the studentized range but gives each pair its own standard error and its own Welch–Satterthwaite degrees of freedom, the same fractional df a Welch t-test reports. On the practice file study-methods.csv, where the three group SDs are 7.97, 8.86 and 11.97, the two procedures disagree in both directions:

Games–Howell is often described as the conservative choice, but the middle row shows that this is too simple. Flashcards and rereading are the two groups with the smaller spreads, so a comparison between them no longer includes the third group's larger variance, and its p-value is about five times smaller. The comparisons that involve the noisiest group get larger p-values instead. Tukey's pooled error term averages the three variances, and Games–Howell does not. The assumption-checks sheet pairs the unequal-variance check with Welch's ANOVA and Games–Howell on one printable page.

The second case is about the design, not an assumption. Some studies compare several treatments against one control and are not interested in whether the treatments differ from each other. Testing all pairs then corrects for comparisons the study never asked about. Dunnett's test corrects only for the ones you made, and it uses the fact that every comparison shares the control group. With four groups and 20 error degrees of freedom, the two-sided .05 cutoff for a t-like statistic is 2.54 under Dunnett, against 2.61 under Bonferroni and 2.80 under Tukey. The gap widens as groups are added. Whichever you use, say so in the write-up, because a reader cannot recover the correction from the p-values.

When 5% of your findings may be wrong

Family-wise correction controls the chance of even one false alarm anywhere in the family. For a handful of group comparisons, that is the right thing to control. For a genomics screen across 20,000 genes, or a brain-imaging analysis across 100,000 voxels, it is so strict that almost nothing stays significant. So those fields control a different error rate.

The false discovery rate asks a different question: among the results I call significant, what share should I expect to be wrong? The Benjamini–Hochberg procedure controls it. Sort your m p-values from smallest to largest and compare the ith one against i/m × q, where q is the false-discovery rate you are willing to accept (usually .05). Find the largest i whose p-value is at or below its threshold, and call that result and every smaller one significant.

Ten p-values show the difference. Bonferroni requires all ten to be at or below .005. Benjamini–Hochberg requires the smallest to be at or below .005, the second .010, the third .015, and so on. Given p = .001, .008 and .012, with the other seven all above .05, Bonferroni finds one result and Benjamini–Hochberg finds three. The cost is stated in advance: about 5% of those three are expected to be false.

Problem 35 of the practice problems takes the significant ANOVA from Problem 34 and finds which groups differ, by Bonferroni and by Tukey’s HSD, on the same three group means. Problem 46 works a control-versus-treatments contrast from four groups’ standard deviations through to its confidence interval, and Problem 47 runs three pairwise comparisons on one pooled error term and writes the corrected result up.