Multicollinearity & Variable Selection
Multiple regression gives each predictor its own "holding the others constant" coefficient. But that only works if the predictors are actually distinguishable. When two predictors are highly correlated with each other (multicollinearity), the model genuinely can't tell whose effect is whose, and the coefficients become wildly unreliable.
The problem: overlapping predictors
Imagine predicting salary from "years of experience" and "age." They move together almost in lockstep. If you ask the model "what's the effect of experience holding age fixed?" you're asking about a situation that barely exists in your data: people with lots of experience but low age are rare. With almost no independent variation to learn from, the estimate becomes a wild guess.
🎮 Watch the Coefficients Go Haywire
The true effect of x₁ is exactly +1.00 (dashed line). Each dot is x₁'s estimated coefficient from one fresh sample. Slide the predictors' correlation up and watch the estimates scatter wildly.
The tell-tale signs
- Huge standard errors: coefficients with enormous uncertainty, so individual predictors look "not significant" even though the model as a whole fits well.
- Unstable, surprising coefficients: estimates that flip sign or balloon in size, and lurch around when you add or drop a variable.
- A high VIF. The Variance Inflation Factor measures exactly this. VIF = 1/(1 − R²) where R² is how well the other predictors explain this one. A common alarm threshold is VIF > 5 (some say 10).
- A low tolerance. Software usually prints a second column beside the VIF called tolerance, which is the same information upside down: tolerance = 1 − R², the share of a predictor's variance that is its own rather than borrowed from the others. Tolerance and VIF are reciprocals, so VIF > 5 is tolerance < .20 and VIF > 10 is tolerance < .10. A predictor with tolerance .08 has 8% of itself left over once the rest of the model has had its say.
Key reassurance: multicollinearity inflates the uncertainty of individual coefficients, but it does not hurt the model's overall predictions or its R². If you only care about predicting y, you can often ignore it. It only bites when you need to interpret individual coefficients.
What to do about it
- Drop or combine redundant predictors (keep one of experience/age, or average several items into a single index).
- Use domain knowledge to choose which variable to keep, not a blind algorithm.
- Regularize: ridge regression deliberately shrinks coefficients to tame the instability, and its cousin the lasso will drop the redundant predictor for you.
A warning about stepwise selection
It's tempting to let the computer pick variables automatically (stepwise: add/drop predictors by p-value). Resist over-relying on it. Automated selection capitalizes on chance, produces over-optimistic p-values and R², and isn't reproducible. Thoughtful, theory-driven choices, backed by honest validation on held-out data, beat algorithmic dredging.
Why it matters: with real datasets full of related measurements, multicollinearity is the rule, not the exception. Recognizing it stops you from over-interpreting a coefficient that's really just statistical noise.
Problem 29 of the practice problems hands you the textbook symptom in one output block: R² = .31 with F(3, 146) = 21.87, p < .001, not a single significant predictor, and a negative sign on years of experience. You work the tolerance and VIF out from each predictor’s R² on the other two, then price three possible fixes.
Common questions
What VIF value indicates a problem?
Common alarm thresholds are VIF > 5 (cautious) or VIF > 10 (lenient) — VIF = 5 means that predictor's coefficient variance is inflated 5-fold because other predictors largely explain it. Treat these as smoke detectors, not verdicts: a high VIF between a predictor and its own squared term is normal, while VIF 4 between two conceptually distinct predictors might still deserve thought.
Every pairwise correlation is below .6. Am I safe from multicollinearity?
No. Scanning the correlation matrix catches only two-variable overlap, and collinearity is a many-variable problem: a predictor can be almost perfectly predicted by a combination of the others while correlating modestly with each one. Take four predictors where the fourth is roughly the sum of the first three. Every pairwise correlation sits at about .55, which looks harmless, yet the fourth predictor's VIF is 13 and its tolerance is .08. This is the reason VIF exists as a separate diagnostic: it regresses each predictor on all the others at once, which is the question the correlation matrix never asks.
Why is stepwise regression frowned upon?
Because it runs many hidden tests and keeps whatever chanced below .05: the surviving p-values are biased low, R² is inflated, and small data perturbations produce entirely different "final" models. It also can't use what it doesn't know — theory. Choose predictors from domain knowledge, and if you need automatic selection for prediction, use penalized methods like the lasso with cross-validation.