Multiple Regression
Income depends on education and experience and region; health depends on diet and exercise and age. Multiple regression extends the simple line to several predictors at once, and in doing so unlocks its single most important idea: holding everything else constant.
What a coefficient means now
The model is just a line with more terms:
ŷ = b₀ + b₁x₁ + b₂x₂ + … + bₖxₖ
The magic is in how you read each coefficient. b₁ is the effect of x₁ on y while holding every other predictor fixed. It's the "all else equal" effect: the bit of x₁'s influence that isn't already explained by the other variables. This is what lets regression control for confounders, and it can completely change the story.
🎮 The Power of "Controlling For"
Outcome vs. x₁, with a lurking second variable shown by color (two groups). The gray line ignores it; the colored lines hold it constant. Increase the confounding and watch them disagree.
When the confounder is strong, the naive slope (gray) can point the opposite way from the true within-group effect (colored). The gray line says "higher x₁ goes with higher y"; control for the group and the real relationship is negative. This is Simpson's paradox, and it's exactly the kind of mistake multiple regression exists to prevent.
The headline idea: a multiple-regression coefficient answers "what's the effect of this variable, among people who are otherwise alike?" Leaving out an important predictor (an omitted variable) can bias the others — sometimes enough to reverse their sign.
Reading the whole model
- Each coefficient is a partial effect, holding the other predictors constant.
- R² now reports how much of the outcome's variation all the predictors explain together.
- Each coefficient gets its own p-value, testing whether that predictor adds anything beyond the others.
🎮 More Predictors Always Look Better
The first two predictors are real; the rest are pure noise. Add them and watch the scoreboard: plain R² only ever climbs, but adjusted R² rises only for genuine signal, then flattens out. The widening gap between the two curves is R² fooling you.
Plain R² is a ratchet: it can only go up, because adding a column never hurts the best in-sample fit, even if that column is random noise. Adjusted R² charges a toll for every predictor, so it only rises when a variable earns its keep. That's why it's the fairer scoreboard for comparing models of different sizes, and why chasing a high R² by piling on predictors is a trap. When the comparison really matters, model comparison has sharper tools than either number.
Putting the predictors on one scale
Each b is measured in the units of its own predictor, so their sizes say nothing about which predictor matters more. Weekly study hours and nightly sleep hours are both "hours" and still aren't comparable: study time varies enormously between students, sleep barely at all. The standard repair is the standardized coefficient, written β:
β = b × (SD of the predictor) ÷ (SD of the outcome)
You get the same numbers by z-scoring every variable before fitting, which is why β reads as "a one-standard-deviation rise in this predictor moves the outcome by β standard deviations". With a single predictor it is exactly the correlation r. Software usually prints it next to b (SPSS labels the column Beta), and APA style writes it with no leading zero: β = .42.
Two limits are worth stating out loud. β depends on how much your particular sample happened to vary, so the same variable earns a larger β in a diverse sample than in a narrow one, and βs from different studies aren't interchangeable. It also means little for a dummy-coded group, whose "standard deviation" is an accident of how many people landed in each group. Rank predictors within one model with β; quote the raw b when you want a sentence a reader can picture. If two predictors are strongly correlated with each other, neither coefficient is stable enough to rank at all, which is multicollinearity.
A caution
"Holding constant" is a statistical operation, not a real experiment. Regression can only adjust for variables you actually measured and put in the model; it can't control for confounders you forgot. And it never, by itself, proves causation: which variables belong in the model is a question about causal structure, settled before you fit anything. It's a powerful lens, not a magic wand.
Why it matters: multiple regression is the workhorse of the social and health sciences, and the foundation for almost everything left in this course: interactions, categorical predictors, logistic regression, and more. The fourth panel of the printable test-chooser poster maps those variants against the outcome each one expects. The assumptions are what keep all of it honest. If you're planning a model of your own rather than reading someone else's, Plan My Analysis turns the design into a test, a sample size, and a write-up skeleton.
Problem 23 of the practice problems takes a two-predictor model whose raw coefficients point one way and whose standardized coefficients point the other, and asks which one a journalist should have quoted.
Common questions
What does 'controlling for' a variable actually mean?
It means comparing like with like, statistically: the coefficient for x₁ describes how y differs between cases that differ in x₁ but are identical on every other predictor in the model. It's an adjustment computed from the data, not a real experiment — and it only works for confounders you actually measured and included.
How many predictors can I put in a regression?
A classic rule of thumb: at least 10–15 observations per predictor, or the model starts fitting noise — with n = 50, stay around 3–5 predictors. It's a guideline rather than a law (what really matters is effect sizes and collinearity), but models that flout it produce coefficients and R² values that collapse on new data. Honest checks: adjusted R² and cross-validation.
My R² is low but my predictors are significant. Is the model useless?
Those two numbers answer different questions. Significance asks whether a coefficient is distinguishable from zero; R² asks how tightly the cloud hugs the fitted surface. Given enough people the first can be emphatic while the second stays tiny: at n = 1,000, an R² of only .01 still gives F(1, 998) = 10.08, p = .002. Human behavior works like that, and R² between .10 and .30 is ordinary in psychology or epidemiology where an engineering calibration would sit at .99. Read it as two facts at once: your predictor moves the outcome dependably, and the outcome also moves for many reasons you never measured. A model like that can teach you something real and still be poor at predicting individuals.