ANOVA & t-Test Assumptions
Every test you've met rests on assumptions: conditions that must roughly hold for its p-values to be correct. If they fail, the test still gives you a p-value, but it is the wrong one. There are only a few main assumptions, and most of them are easy to check by eye.
The big three
- Normality. The data (or, more precisely, the sampling distribution of the mean) is roughly bell-shaped. Thanks to the CLT, this matters less as samples grow.
- Homogeneity of variance. The groups you're comparing have similar spread. t-tests and ANOVA assume it (Welch's versions relax it).
- Independence. Observations don't influence each other. This is the one that cannot be repaired: no correction fixes a test when independence is violated by, say, repeated measurements treated as separate people.
Those three cover tests that compare group means. The moment you fit a line, a fourth joins them: linearity, the assumption that the relationship you are modeling is straight. No diagnostic on this page can catch it. Curved data can be perfectly normal and perfectly equal-variance, and a straight line through it will still give you a slope, but the slope misdescribes the relationship. A residual plot shows the curve, and regression diagnostics covers those.
Checking normality with a Q-Q plot
A histogram gives a rough idea of shape, but the Q-Q plot shows departures from normality more clearly. It plots your sorted data against the values you'd expect if the data were perfectly normal. If the points fall along a straight line, the data is behaving like a normal sample. A systematic curve away from the line shows how the data departs from normal. Switch the underlying distribution below and learn to read the signatures.
🎮 Q-Q Plot Detective
Top: a histogram of the sample. Bottom: its Q-Q plot. Normal data stays close to the diagonal, and the shape of a departure shows how the data is non-normal.
Histogram
Q-Q plot: sample quantiles vs. normal quantiles
The Q-Q signatures: a straight line = normal. An upward-curving arc = right-skewed (a long high tail). An S-shape = heavy tails / outliers (extremes more extreme than normal). A flat step in the middle = bimodal. Once you've seen each a few times, you can diagnose a distribution in a glance.
How crooked does normal data look?
The playground shows four clean signatures, but real samples are messier. Before you call a Q-Q plot bent, you need to know how far the plot of a sample from a normal distribution can bend. Near the ends it can bend a lot.
Draw 20 values from a perfect normal distribution and plot them. The highest point is plotted at 1.87 standard deviations, the twentieth of twenty expected quantiles. Its actual value varies a lot: across 200,000 simulated normal samples it fell between 1.29 and 2.71 in 95% of them. The point in the middle of the same plot stayed inside [−0.38, 0.26]. So the ends of a normal Q-Q plot vary about 2.2 times as much as the middle, and a faint S-curve is usually just that extra variation at the ends.
Now draw a band around the line that holds 95% of normal samples at each position, one position at a time. Then count the normal samples with at least one point outside the band: 49% at n = 20, and 66% at n = 50. About half of all perfectly normal samples of 20 have a point outside the band, simply because there are twenty points that could fall outside it.
Give the middle of the plot more weight than the ends, and look for a systematic bend, not for single stray points. When the decision matters, stop judging by eye and draw the band: R's car::qqPlot(x) shades it for you, and by hand it's a short loop. Simulate a few hundred normal samples of your own n and keep the 2.5th and 97.5th percentile at each position. Then check whether your data leaves that envelope in a run of neighboring points or only at one end.
How much do violations matter?
t-tests and ANOVA are surprisingly robust to mild non-normality, especially with larger samples, thanks to the CLT. They're more sensitive to very unequal variances (use Welch's version of the test) and badly thrown off by outliers in small samples. The descriptives calculator will flag those outliers and print the skewness for any data you paste into it. A violation of independence cannot be fixed in the analysis at all. You have to prevent it in the study design.
When an assumption really is broken
- Transform the data (a log transform often reduces right skew).
- Use a robust variant (Welch's t, trimmed means).
- Switch to a non-parametric test that makes no normality assumption at all.
A harder decision comes first: whether a flagged point is an error or a real but unusual person. The complete worked project works through four such points in one dataset and keeps all four.
Why it matters: when an assumption fails, the output looks the same as always, but the p-value can be wrong. A 10-second look at a Q-Q plot and a residual plot can save you from reporting a wrong conclusion. When a check does fail, the test chooser and the printable assumptions cheat sheet both name the alternative.
The printable Assumption Checks cheat sheet also lists, for each family of tests, what to check, how to check it, and the fix when it's violated. For designs that break more than one assumption at once, the guide Choosing Statistics for Your Dissertation has a section on what to do when your study does not match any textbook design.
Problem 38 of the practice problems describes two clinics whose SDs are 3.1 and 6.9 and asks you to decide whether to pool before you look at either answer. On the same numbers, Welch and Student disagree by more than you might expect.
Common questions
How do I check if my data is normally distributed?
Look at a plot before you run a test. A Q-Q plot (points close to the diagonal mean normal) and a histogram tell you more than any p-value. Formal tests such as Shapiro–Wilk have a problem. In large samples they flag small deviations that do no harm, and in small samples they miss serious ones. Learn the Q-Q signatures in the playground above, then compare them with the section on how crooked normal data looks. Roughly half of normal samples put at least one point outside a 95% band, and that point is almost always at one end. Judge the middle of the plot, and expect the ends to stray.
Can I let the assumption test decide which test I run?
You can, but it changes your actual false-positive rate. Running Levene's test and then choosing Student's or Welch's t from its result is a two-stage procedure. The false-positive rate that matters is the rate of the whole procedure, not of the test you end up running. In 200,000 simulated samples with unequal groups (n = 45 and 15), where the smaller group had the larger variance, always pooling gave a 15.2% false-positive rate at a nominal 5%. Welch alone gave 4.9%, and letting Levene choose gave 5.7%. That is better than always pooling, but still above 5%. A better habit is to decide from the design before the data arrives and write the choice down. Then you are not choosing a test after you have seen the result.
Which statistical assumption matters most?
Independence matters most. The CLT usually makes mild non-normality harmless, and unequal variances have Welch-style fixes. But if you treat correlated observations (repeated measures, students in the same classroom) as independent, your standard errors come out too small and you get significant results that are not real. No correction after the fact fixes this. You deal with it in the design, or with models built for grouped data, such as multilevel models.