Repeated Measures ANOVA

When you measure the same people under several conditions (the same patients before, during, and after treatment), you have a repeated-measures design. Like the paired t-test, repeated-measures ANOVA can remove the stable differences between people from the comparison. That often makes it far more sensitive than a between-subjects design.

The problem with big individual differences

Suppose you test three conditions and people differ a lot to begin with: some always score high, some always low. In a between-subjects design, that person-to-person variability becomes noise that can hide the condition effect. You might have a real effect and still fail to find it.

The repeated-measures trick

Because every person is measured in every condition, each person serves as their own control. The analysis can separate out a "subjects" part of the variation and set it aside, leaving a much cleaner comparison of the conditions. In the subject-centered view above, each person's own average is subtracted out, and the tangle of lines becomes a clear shared pattern. Repeated-measures ANOVA does the same thing in its arithmetic. For the same reason, a within-subjects result is best drawn as one line per person, not as a row of bars. The chart chooser calls that a slopegraph, and Tables & Figures That Don't Lie shows what goes wrong when you draw the bars instead.

Two conditions first: the difference score

With only two conditions there is nothing new to learn, because repeated-measures ANOVA is then the paired t-test in different notation. Write D = X₂ − X₁ for each person and the whole design reduces to one column of numbers. The null hypothesis becomes μ_D = 0, and a one-sample t on that column is the test. Two conditions also show most clearly why pairing works, so here is a small example before the general table.

Five people did a reaction-time task twice, once on its own and once with a distracting second task running. Times are in milliseconds, so smaller is faster.

The two columns are very spread out. People differ from each other by 60 milliseconds or more, and the effect is seven. Treat the ten numbers as two independent samples and the test finds nothing: s_p = 25.35, t(8) = 0.44, p = .67. Use the difference column instead and the same seven milliseconds give t(4) = 7.0/(2.00/√5) = 7.83, p = .001, well past the t* of 2.776. Every person got slower under the distractor, and four of the five by between six and nine milliseconds.

Why the difference column varies so little

The reason is one identity from probability, and it is the whole justification for the design:

Covariance is the unstandardized version of the correlation: the average product of the two variables' deviations from their own means, in whatever units they came in. Divide it by both standard deviations and you get r back. In the table above Var(single) = 616.0, Var(dual) = 669.0 and r = .998, so Cov = .998 × 24.82 × 25.87 = 640.5, and the identity gives

616.0 + 669.0 − 2(640.5) = 4.0.

That is the 2.00² the difference column shows. Subtracting twice the covariance is the whole mechanism. When people keep their rank order across conditions, the covariance is nearly as large as the two variances. Almost all of the between-person variation then cancels, and what is left is the part that differs within a person. When the correlation is weak, the covariance is small and little cancels, so the paired design gains you nothing and still costs degrees of freedom. Around r = .5 the two designs break even, and between vs within designs works through why.

The repeated-measures table

With three or more conditions there is no single difference to take, so the difference score no longer works. In its place is a partition of the sums of squares that does the same job more generally. Start from the one-way table, where the total splits into a conditions part and everything else, then split that "everything else" once more:

A between-subjects design cannot make the split in the second line. When each person is measured once, there is no way to tell a stable personal level apart from random variation, so both stay in one error term. Measuring each person k times separates them, and the subjects part no longer counts as error.

Each piece is a sum of squared deviations from a mean. The pieces differ only in which mean they use. Writing n for the number of people, k for the number of conditions and N = nk for the number of observations:

Two things about the error row matter. Its formula is a residual: what is left in a score once its own person's level and its own condition's level have both been accounted for. And its degrees of freedom, (n − 1)(k − 1), are the same as an interaction's in a two-way table. That is because the repeated-measures error is the person-by-condition interaction: the extent to which the conditions affect different people differently. Mean squares then follow the usual rule, MS = SS/df, and the test is

F = MS_conditions / MS_error, on F(k − 1, (n − 1)(k − 1)).

The subjects row gets no F of its own in most reports. Its F would test whether people differ from each other, which is rarely in doubt and is not the study's question.

The table on eight people

The widget above starts with a fixed set of eight people measured in three conditions, chosen so that the sums of squares come out as whole numbers. Reload the page or press "Back to the built-in eight" to get them back after exploring.

The three condition means are 41, 50 and 59, and the grand mean is 50. The eight person means run from 17 to 83. Both go straight into the arithmetic:

SS_C = 8[(41 − 50)² + (50 − 50)² + (59 − 50)²] = 8(162) = 1296
SS_S = 3[(17 − 50)² + (28 − 50)² + … + (83 − 50)²] = 3(3388) = 10164
SS_T = 11516, so SS_E = 11516 − 1296 − 10164 = 56

The last step uses the identity, not the definition. Computing SS_E from its own residual formula gives the same 56, and an exam question that gives you three of the four sums of squares expects you to find the fourth by subtraction. The completed table:

F(2, 14) = 162.00 against a critical value of 3.74 at α = .05, so p < .001, and partial η² = 1296/(1296 + 56) = .96. Now run the same 24 numbers as a one-way ANOVA that ignores who produced them. SS_C is unchanged at 1296, but there is no subjects row, so its 10164 goes back into the error: 10220 on 21 degrees of freedom, MS = 486.67, and F(2, 21) = 1.33 with p = .29. The data and the condition means are the same, but one analysis finds a very large effect and the other finds nothing. The two bars in the widget show this. Pushing the "how different people are" slider up widens the gap. The repeated-measures F does not move, because the slider adds variation only to the subjects row. The one-way F falls, because all of that variation goes into its error.

Drag the slider the other way, down to zero, and the result reverses. The one-way F climbs to 243.00 and overtakes the repeated-measures F, which is still 162.00. With no individual differences left to remove, the subjects row removes nothing. The seven degrees of freedom it took from the error term are wasted: the one-way error is 56 over 21 df, while the repeated-measures error is the same 56 over 14. A within-subjects design helps only when people differ from each other, and the more they differ, the more it helps. A second factor in a factorial design involves the same trade. So think about the correlation between conditions before you choose the design, not after.

Reading the design back out of the df

A paper that reports F(3, 84) tells you more than it seems to. The first number is k − 1, so there were four conditions. The second is (n − 1)(k − 1) = (n − 1) × 3 = 84, so n − 1 = 28 and the study ran 29 participants, who between them produced 116 observations. Exams ask this in both directions, and it is also a quick check on a paper. A repeated-measures error df that is not a multiple of k − 1 means something was reported wrong, and a df that implies a surprisingly small n deserves a second look.

Mixed designs: one factor between, one within

Most real studies cannot make every factor within-subjects. You can measure the same person before and after training, but you cannot put them in both the training group and the control group. A mixed design repeats the factors that can be repeated and assigns people to levels of the ones that cannot. One between-subjects factor crossed with one within-subjects factor is by far the most common shape in psychology. A within-subjects factor is one whose levels every participant experiences, and a between-subjects factor is one whose levels different participants experience.

Say forty people are randomly assigned to a meditation course or a waiting list, twenty in each, and everyone is tested for attention at baseline, after four weeks and after twelve. Group is between, time is within, and the table splits in two, because the two factors are tested against different error terms. Group differences are differences between people, so they are judged against how much people differ. Time differences happen inside a person, so they are judged against the much smaller person-by-time residual.

Here g is the number of groups, k the number of measurement occasions and N the number of participants rather than observations. The five rows add to 119, which is Nk − 1 for 120 observations. SPSS prints them as two separate tables, Tests of Between-Subjects Effects and Tests of Within-Subjects Effects, and the second holds both the time effect and the interaction. Reading the error df backwards works as before, with one extra step. 76 = 38 × 2 gives 38 = N − g, and a between-subjects error df of 38 with two groups means 40 people in total, not 40 per group.

Read the table in the same order as a two-way table. Look at Group × Time first, because it is usually the question the study was designed to answer: did the two groups change differently? If it is significant, the two main effects are averages that may not describe either group, and each is reported as qualified by the interaction. A main effect of time then means "averaging over both groups, scores moved". A main effect of group means "averaging over all three occasions, one group scored higher". In a pre-post study, that is often due to a baseline difference, not to the treatment.

What the design costs

Measuring the same person repeatedly also creates problems a between-subjects design never has. Most of them are order effects: changes in the response that come from a condition's position in the sequence, not from the condition itself. Two of them push in opposite directions. Practice improves performance as people get used to the task, so whichever condition runs last looks best. Fatigue and boredom do the reverse, so whichever condition runs last looks worst. A carryover effect is the more specific problem of one condition still acting during the next: a drug not yet cleared, a strategy just learned, a mood left over from the last block.

The standard defense is counterbalancing, running the conditions in different orders across participants so that a pure order effect falls equally on every condition and cancels in the means. With two conditions, half the sample gets each order. With more, a Latin square puts each condition in every position without needing all k! sequences. But counterbalancing does not remove the variability. A symmetric order effect no longer biases the condition means, but it stays in the error term. An asymmetric one, where A changes B more than B changes A, is not balanced away at all. When the carryover is severe enough, use a between-subjects design.

Three further costs decide the choice of design as often as power does. A participant who misses one condition is lost from every condition, since the table needs a complete row. The conditions have to be repeatable in the first place, which rules out anything that teaches, reveals or permanently changes something. And the analysis has an extra assumption, sphericity, which the rest of this page is about.

The catch: sphericity

Repeated measures come with their own assumption, sphericity: roughly, that the differences between every pair of conditions have similar variance. When it's violated (common with time-based data, where adjacent timepoints correlate more than distant ones), the test becomes too liberal. Standard corrections (Greenhouse-Geisser, Huynh-Feldt) adjust the degrees of freedom to compensate.

What ε is, and what it costs

Both corrections work through a single number, epsilon, written ε. It measures how far the pattern of condition-to-condition differences departs from sphericity, and it runs from 1 down to a lower limit of 1/(k − 1). Perfect sphericity gives ε = 1. With three conditions ε can be no lower than .5, and with four no lower than .33. Multiply both degrees of freedom by ε and read the F table again. The F ratio itself never changes, only the distribution it is compared against.

Problem 37 of the practice problems gives a case with 12 typists and 3 keyboard layouts, F = 5.84, and a Greenhouse-Geisser ε of .71. Here are four ways of reading the same ratio:

The correction more than doubles the p-value here without changing a single data point. Even the lower-bound correction leaves this result significant, but a study with an uncorrected p of .04 would not stay significant after correction.

Greenhouse-Geisser tends to underestimate ε when the violation is mild. Huynh-Feldt adjusts it upward (that .785 is computed from the .71, not measured separately), and it can reach 1, where software caps it. Most textbooks say to use Greenhouse-Geisser when its ε is below .75 and Huynh-Feldt otherwise. Report which one you used and give the ε beside it. Fractional degrees of freedom tell a reader that something was corrected, but not by how much. The assumption-checks sheet lists this check alongside every other check for a comparison of means.

A more modern alternative is a mixed (multilevel) model. It treats each participant as a random effect and estimates the correlation structure directly. So it needs no sphericity correction, handles unequal spacing between timepoints, and keeps a participant who missed one session. The word mixed means something different here than in "mixed design" above. A mixed design is about which factors repeat, and a mixed model is about fixed and random effects. A mixed design is usually what a mixed model is fitted to. Repeated-measures ANOVA is still the clearer way to understand what a within-subjects design gains, so it is taught first. When the repeated outcome is a category rather than a number, the same reasoning leads to McNemar's test: pair each person with themselves and count only the ones who changed.

Problem 37 of the practice problems works the sphericity case above from the sums of squares up, so you build the F yourself before correcting it. Problem 53 completes a repeated-measures table from three sums of squares and the design, and Problem 54 reads a mixed-design output and recovers the study from its df column. Problem 67 asks the prior question: whether a study is between, within, or neither.