Regression Diagnostics
You can fit a regression line to literally any cloud of points and get a slope, an intercept, and an R². None of that tells you whether a straight line was the right model. The verdict on that comes not from the line itself but from what it leaves behind: the residuals.
The residual plot
A residual is the gap between an actual point and the line's prediction. Plot the residuals against the fitted values and you get a diagnostic X-ray of your model. The thing you want to see is nothing — a formless, random cloud centered on zero. Any pattern in the residuals is the model telling you it missed something.
🎮 Residual Plot Reader
Top: data with the least-squares line. Bottom: residuals vs. fitted values. Switch the scenario and learn each tell-tale shape.
Data & fitted line
Residuals vs. fitted (you want a shapeless cloud around zero)
The residual signatures: a shapeless cloud = healthy. A curve / U-shape = the true relationship is nonlinear (a straight line is wrong — add a curve term). A fan or funnel = non-constant variance (heteroscedasticity), which breaks the standard errors. A lone point far from the rest = an influential outlier that may be single-handedly tilting the line.
The five things to check
- Linearity. The residual cloud has no curve. If it bends, the relationship isn't a straight line.
- Constant variance. The vertical spread of residuals is the same all the way across. A funnel means it isn't.
- Normal residuals. Best checked with a Q-Q plot of the residuals; matters most for small samples.
- Independent residuals. One row's error should tell you nothing about the next row's. This is the assumption the plot above cannot show you, and the one that does the most damage when it fails.
- Influential points. No single observation should dominate the fit. Leverage and influence are not the same thing, and the regression assumptions lesson separates them: an unusual predictor value gives a point leverage, but only leverage together with a large residual actually swings the line. Cook's distance measures that combination.
Independence: the check the plot can't make
Residuals against fitted values will look perfectly healthy on data that breaks independence, because the violation lives in the order or the grouping of the rows rather than in their spread. Two versions turn up constantly. Measurements taken in sequence (daily sales, weekly symptom scores) tend to be followed by similar values, which is autocorrelation. Plotting residuals against time or row number, instead of against fitted values, is what exposes it, and the Durbin-Watson statistic your software prints is the formal version: values near 2 mean no autocorrelation, values near 0 mean each residual is positively correlated with the one before it.
The second version is clustering: several rows from the same participant, class, clinic, or family. Ordinary regression counts every row as a fresh piece of evidence, so ten measurements from one person are treated as ten people. Standard errors come out too small and p-values too optimistic, and unlike the other four problems on this list, nothing you do to the residuals afterwards repairs it. The fix is a model that knows about the grouping, which is what mixed and multilevel models are for.
What to do about it
A curved pattern? Transform a variable or add a polynomial/nonlinear term. A funnel? Transform the outcome (logs often help) or use robust/weighted standard errors. An influential outlier? Investigate it — is it a data-entry error, or a real-but-rare case? — and report the fit with and without it.
Why it matters: a high R² on a mis-specified model is a confident wrong answer. Residual plots are the cheapest insurance in all of statistics — a few seconds of looking saves you from trusting a model that doesn't fit. This habit carries straight into multiple regression.
Problem 22 of the practice problems hands you a fan-shaped residual plot, one case with leverage 0.31 and Cook’s D 0.68, and two predictors correlated at .92, and asks what to fix before reading a single coefficient.
Common questions
How do I know whether a pattern in my residual plot is real or just noise?
Ask what randomness actually looks like at your sample size, because it looks patterned far more often than people expect. With thirty points, clumps and gentle curves show up constantly in data generated from a perfectly linear model, and a reader hunting for structure will find some. The cheap calibration trick is a lineup: simulate several residual plots from a model where the assumptions hold by construction, shuffle your real plot in among them, and see whether you can pick yours out. If you can't, whatever worried you sits inside the range of ordinary noise. That is the lineup protocol of Buja and colleagues (2009), and running it informally two or three times will retrain your eye better than any threshold.
What is heteroscedasticity in simple terms?
Residual spread that changes across the range of predictions — typically a funnel: tight errors for small fitted values, wide ones for large (income data does this constantly). The slope estimate stays unbiased, but its standard errors and p-values become unreliable. Fixes: transform y (log is the usual medicine), or use robust (heteroscedasticity-consistent) standard errors.
Should I delete outliers from my regression?
Not as a reflex. First investigate: a data-entry error (age = 250) gets fixed or removed; a genuine-but-extreme case is information about the world. For real outliers, check influence (Cook's distance), then report the model with and without the point — if conclusions flip on one observation, that fragility is the finding.