Assumptions of Regression

A regression coefficient and its p-value are only trustworthy if the model's assumptions hold. For ordinary least squares there are four, conveniently spelling LINE. Beyond them lurks a separate, sneaky threat: a single data point with enough leverage to bend the whole line to its will.

The LINE assumptions

• L · Linearity: the relationship between predictors and the mean of y is actually a straight line (or plane). Check with residual plots.
• I · Independence: the errors are independent of each other. Violated by time series and clustered/repeated data.
• N · Normality of residuals: the errors are roughly normal. Matters most for small samples; check with a Q-Q plot of residuals.
• E · Equal variance (homoscedasticity): the residuals have constant spread across all fitted values. A funnel shape means trouble.

Leverage vs. influence vs. outlier

These three get muddled, but they're distinct. An outlier has an unusual y (far from the line). A high-leverage point has an unusual x (far from the other predictors). A point is influential only when it's both: extreme in x and off the trend, so that removing it noticeably moves the line. Drag the point below and feel the difference.

Park the point in the middle (average x) and even a big vertical move barely budges the line: low leverage. Drag it far left or right and the line snaps toward it: high leverage. A point is dangerous only when high leverage meets a large residual.

How high is "high"?

The readout above gives the dragged point's leverage as a number and as a multiple of average leverage, and that second figure is the one worth understanding, because leverage comes with a fixed budget. Across all n points the leverages always sum to p, the number of things the model estimates (an intercept plus one slope here, so p = 2). Average leverage is therefore p/n, no matter what the data look like. This chart has 13 fixed points plus the one you drag, so its average is 2/14 = 0.14, and a point can only rise above that by taking leverage away from everybody else.

The conventional flags are 2p/n, and sometimes a stricter 3p/n, which work out to 0.29 and 0.43 for this chart. Park the point at the center and its leverage falls to 0.07, half the average. Drag it hard against either wall and it reaches about 0.27, which is 1.9 times average and still under the flag. Thirteen well-spread neighbors are hard to outvote, and that is the whole reason leverage is relative: the same x value is unremarkable in a wide sample and extreme in a narrow one.

Residuals need the same treatment, and for a reason that follows from leverage. A high-leverage point pulls the line toward itself, so its raw residual is too small to be compared with anybody else's. The fix is a studentized residual, which divides each residual by its own standard error, s√(1 − h). SPSS's Casewise diagnostics box flags on the plain standardized residual, so the studentized version is a separate tick in the same dialog's Save… panel, and it is the one worth having: a point can look tame by the first measure and alarming by the second precisely because of its leverage.

Put the two together and you get Cook's distance in closed form: D = (r²/p) × h/(1 − h), where r is the studentized residual. The first factor is the residual, the second is the leverage, and the product is the sentence "influence requires both" written as arithmetic. Either factor near zero drives D to zero. The usual cutoffs are D > 1 and the stricter 4/n, which for these 14 points is also 0.29.

Numbers like these are triage, not verdicts. They tell you which rows to open and read; whether the row is an error or a genuinely interesting case is a question about the study, not about the arithmetic.

When assumptions fail

• Nonlinearity → add polynomial or transformed terms.
• Non-independence → use models built for it (time-series, mixed/multilevel models).
• Non-normal or heteroscedastic errors → transform y, or use robust and bootstrapped standard errors.
• Influential points → report the fit with and without them, and explain the difference.

"Robust" in that third bullet has a specific meaning worth knowing, since your software will offer it by an abbreviation. A heteroscedasticity-consistent standard error (the HC family, usually HC3 by default) rebuilds each coefficient's standard error from the individual squared residuals rather than from one pooled variance. Your slope estimates do not move at all; only the uncertainty around them does. That is the right shape of repair, because unequal variance distorts the standard errors and leaves the coefficients themselves alone.

Problem 22 of the practice problems puts one case on trial using both measures at once: on a 40-property regression, case 31 has leverage h = 0.31 and Cook’s D = 0.68, and you have to say whether either number is unusual enough to act on.