Assumptions & When They Break
Every test you've met comes with fine print — conditions that must roughly hold for its p-values to mean what they claim. Ignore them and the test still spits out a number; it's just the wrong number. The good news: the main assumptions are few, and most are easy to check by eye.
The big three
- Normality — the data (or, more precisely, the sampling distribution of the mean) is roughly bell-shaped. Thanks to the CLT, this matters less as samples grow.
- Homogeneity of variance — the groups you're comparing have similar spread. t-tests and ANOVA assume it (Welch's versions relax it).
- Independence — observations don't influence each other. This is the unforgiving one: no clever correction saves a test when independence is violated by, say, repeated measurements treated as separate people.
Those three cover tests that compare group means. The moment you fit a line, a fourth joins them: linearity, the assumption that the relationship you are modeling really is straight. It is the one no diagnostic on this page can catch. Curved data can be perfectly normal and perfectly equal-variance, and a straight line through it will still give you a slope that describes nothing. Seeing it takes a residual plot, which is where regression diagnostics picks the story up.
Checking normality with a Q-Q plot
A histogram hints at shape, but the sharpest tool is the Q-Q plot. It plots your sorted data against the values you'd expect if the data were perfectly normal. If the points fall on a straight line, the data is normal. Systematic curves away from the line reveal exactly how it deviates. Switch the underlying distribution below and learn to read the signatures.
🎮 Q-Q Plot Detective
Top: a histogram of the sample. Bottom: its Q-Q plot. Normal data hugs the diagonal; departures tell you what's wrong.
Histogram
Q-Q plot — sample quantiles vs. normal quantiles
The Q-Q signatures: a straight line = normal. An upward-curving arc = right-skewed (a long high tail). An S-shape = heavy tails / outliers (extremes more extreme than normal). A flat step in the middle = bimodal. Once you've seen each a few times, you can diagnose a distribution in a glance.
How much do violations matter?
Not all violations are equal. t-tests and ANOVA are surprisingly robust to mild non-normality, especially with larger samples — the CLT does the heavy lifting. They're more sensitive to very unequal variances (use Welch's correction) and badly thrown off by outliers in small samples. The descriptives calculator will flag those outliers and print the skewness for any data you paste into it. And nothing rescues violated independence — that one you fix in the study design, not the analysis.
When an assumption really is broken
- Transform the data (a log transform often tames right-skew).
- Use a robust variant (Welch's t, trimmed means).
- Switch to a non-parametric test that makes no normality assumption at all.
The harder judgment is the one before that: deciding whether a flagged point is a violation or just a person. The complete worked project works through four such points on a real dataset and keeps all four.
Why it matters: checking assumptions is the difference between a result you can trust and a number that merely looks official. A 10-second glance at a Q-Q plot and a residual plot prevents a lot of embarrassing conclusions. When a check does fail, the test chooser and the printable assumptions cheat sheet both name the alternative rather than leaving you to guess.
Keep the whole checklist within reach: the printable Assumption Checks cheat sheet lists, per test family, what to check, how, and the fix when it's violated. For the designs that break more than one rule at once, the guide Choosing Statistics for Your Dissertation has a section on what to do when reality refuses to match a textbook design.
Problem 17 of the practice problems hands you two clinics whose SDs are 3.1 and 6.9 and asks you to decide, before looking at either answer, whether to pool. Welch and Student disagree by more than you might expect on the same numbers.
Common questions
How do I check if my data is normally distributed?
Look, don't just test: a Q-Q plot (points hugging the diagonal = normal) plus a histogram tells you more than any p-value. Formal tests like Shapiro–Wilk have a trap: in large samples they flag trivial, harmless deviations, and in small samples they miss serious ones — exactly backwards from what you need. Learn the Q-Q signatures in the playground above and trust your eyes.
Can I let the assumption test decide which test I run?
It is the obvious move, and it changes what your α means without telling you. Running Levene's test and then picking Student's or Welch's t on the result is a two-stage procedure, and the false-positive rate belongs to the procedure as a whole, not to whichever test you land on. Simulated at 200,000 samples with unequal groups (n = 45 and 15) whose smaller group has the larger variance, always pooling gives a 15.2% false-positive rate at a nominal 5%; Welch alone gives 4.9%; letting Levene choose gives 5.7%. Better than pooling blindly, still not the 5% it advertises. The cleaner habit is to decide from the design before the data arrives and write the choice down, which also spares you a decision you would otherwise be making with the answer already in view.
Which statistical assumption matters most?
Independence, without question. Mild non-normality is usually forgiven by the CLT, and unequal variances have Welch-style fixes — but treating correlated observations (repeated measures, students in the same classroom) as independent silently shrinks your standard errors and manufactures significance. No correction rescues it afterward; it's fixed by design or by models built for structure, like multilevel models.