Assumptions of Regression

A regression coefficient and its p-value can be trusted only if the model's assumptions hold. Ordinary least squares has four, and their initials spell LINE. There is also a separate problem: a single data point with enough leverage to move the whole line.

The LINE assumptions

• L · Linearity: the relationship between predictors and the mean of y is actually a straight line (or plane). Check with residual plots.
• I · Independence: the errors are independent of each other. Violated by time series and clustered/repeated data.
• N · Normality of residuals: the errors are roughly normal. Matters most for small samples; check with a Q-Q plot of residuals.
• E · Equal variance (homoscedasticity): the residuals have constant spread across all fitted values. A funnel shape means this assumption is violated.

Leverage vs. influence vs. outlier

These three are often confused, but they are different. An outlier has an unusual y (far from the line). A high-leverage point has an unusual x (far from the other predictors). A point is influential only when it's both: extreme in x and off the trend, so that removing it noticeably moves the line. Drag the point below to see the difference.

Put the point in the middle (average x) and even a big vertical move barely changes the line: low leverage. Drag it far left or right and the line follows it: high leverage. A point changes the fit a lot only when it has both high leverage and a large residual.

How high is "high"?

The readout above gives the dragged point's leverage as a number and as a multiple of the average leverage. The second figure matters because the total leverage is fixed. Across all n points the leverages always sum to p, the number of parameters the model estimates (an intercept plus one slope here, so p = 2). So the average leverage is p/n, whatever the data look like. This chart has 13 fixed points plus the one you drag, so its average is 2/14 = 0.14. A point can rise above that only by taking leverage from the other points.

The conventional cutoffs are 2p/n, and sometimes a stricter 3p/n, which come to 0.29 and 0.43 for this chart. Put the point at the center and its leverage falls to 0.07, half the average. Drag it to either edge and it reaches about 0.27, 1.9 times the average and still under the cutoff. With thirteen well-spread points, one point cannot gain much leverage. Leverage is relative: the same x value is unremarkable in a wide sample and extreme in a narrow one.

Residuals need adjusting too, because of leverage. A high-leverage point pulls the line toward itself, so its raw residual is too small to compare with the others. The fix is a studentized residual, which divides each residual by its own standard error, s√(1 − h). SPSS's Casewise diagnostics box flags points using the plain standardized residual. The studentized version is a separate tick in the same dialog's Save… panel, and it is the better one to use. Because of its leverage, a point can look unremarkable by the first measure and extreme by the second.

Put the two together and you get Cook's distance in closed form: D = (r²/p) × h/(1 − h), where r is the studentized residual. The first factor comes from the residual and the second from the leverage. Because D is a product, it is near zero whenever either factor is, so influence requires both. The usual cutoffs are D > 1 and the stricter 4/n, which for these 14 points is also 0.29.

These numbers tell you which rows to look at, not what to do with them. Whether a row is an error or a real, interesting case is a question about the study, not about the arithmetic.

When assumptions fail

• Nonlinearity → add polynomial or transformed terms.
• Non-independence → use models built for it (time-series, mixed/multilevel models).
• Non-normal or heteroscedastic errors → transform y, or use robust and bootstrapped standard errors.
• Influential points → report the fit with and without them, and explain the difference.

"Robust" in the third bullet has a specific meaning, and your software will list it by an abbreviation. A heteroscedasticity-consistent standard error (the HC family, usually HC3 by default) computes each coefficient's standard error from the individual squared residuals, not from one pooled variance. The slope estimates do not change at all. Only their standard errors do. That is the right fix, because unequal variance distorts the standard errors but leaves the coefficients unbiased.

Problem 43 of the practice problems applies both measures to one case. In a 40-property regression, case 31 has leverage h = 0.31 and Cook’s D = 0.68, and you have to say whether either number is unusual enough to act on.