Simple Linear Regression

Correlation tells you two variables move together. Regression goes further: it draws the single best straight line through the cloud, so you can actually predict one variable from the other. The whole game is choosing that line, and there's a precise, beautiful rule for what "best" means.

The line and its errors

A regression line is just ŷ = b₀ + b₁x: an intercept and a slope. For any line you propose, each data point sits some vertical distance above or below it. That gap is the residual: the prediction error for that point.

A good line makes those residuals small overall. But small how? Add them up and positives cancel negatives. The fix: square each residual (killing the sign and punishing big misses extra hard), then add them up. That total is the sum of squared residuals (SSR), and the least-squares line is the one, out of all possible lines, that makes it as small as possible.

What the slope and intercept mean

• Slope b₁: how much ŷ changes for each one-unit increase in x. A slope of 0.6 means "every extra point of x buys 0.6 more y, on average."
• Intercept b₀: the predicted y when x = 0. Sometimes meaningful, sometimes just where the line crosses the axis.

Finding the line by hand

Sliding two sliders until the error stops falling is the idea. You would not want to do it on paper. There is a formula, and it needs nothing you have not already computed in §1.17: the two means, the two standard deviations, and the correlation.

b₁ = r · s_y / s_x     b₀ = ȳ − b₁ x̄

Read the slope formula as a translation. The correlation says how many standard deviations of y you get per standard deviation of x, and multiplying by s_y / s_x converts that exchange rate back into the units the variables were measured in. The intercept formula is then a rearrangement of one fact worth remembering on its own: the least-squares line always passes through (x̄, ȳ), the point of means. Whatever the slope turns out to be, the line is pinned to that point, which is also why a line fitted to data centered on zero in both variables has an intercept of zero.

Take the five music students from §1.17: x̄ = 3.0 practice hours, ȳ = 66.0 marks, s_x = 1.58, s_y = 9.62 and r = 0.740. Then b₁ = 0.740 × 9.62 / 1.58 = 4.5 marks per hour (the rounded inputs give 4.506; the unrounded ones give exactly 4.5), and b₀ = 66.0 − 4.5 × 3.0 = 52.5, so the line is ŷ = 52.5 + 4.5x. Substituting a value of x gives the predicted value ŷ: a student practicing 4 hours is predicted 52.5 + 18 = 70.5, and since the fourth student actually scored 65, their residual is 65 − 70.5 = −5.5, five and a half marks below what the line expected of them.

R²: how much the line explains

Once you've got the best line, R² (which is just the correlation r squared) says what fraction of the variation in y the line accounts for. R² = 0.7 means the line explains 70% of the ups and downs in y; the rest is scatter the line can't capture. Generate new data a few times and watch SSR and R² move together: tighter clouds give smaller SSR and bigger R².

The textbook phrasing is worth memorizing because it names both halves: R² is the fraction of the variation in the values of y that is explained by the least-squares regression of y on x. The variation being divided up is the spread of y around its own mean, and the comparison is against the flat line at ȳ, which is the best you could do knowing nothing about x. The five music students give R² = 0.740² = 0.547, so knowing how long somebody practiced accounts for about 55% of the differences between their marks and leaves 45% to everything else.

Reading a residual plot

R² is one number for the whole fit, and one number cannot say where a line goes wrong. That job belongs to the residual plot: the same points again, with each one's residual on the vertical axis instead of its y value, and a line at zero for reference. Tick "Show the residual plot" in the playground above and it appears under the scatter, drawn against whatever line the sliders are currently holding.

Subtracting the line is what makes the plot useful. A scatter with a steep trend in it is dominated by the trend, and small departures from it are hard to see against that slope. Take the line away and those departures are all that is left, magnified across the full height of the panel. Three shapes are worth knowing.

1. A shapeless band of points scattered evenly above and below zero, with no drift and no fanning. That is the picture a straight line is entitled to, and it means the line has taken everything systematic out of the data.
2. An arch or a valley, all the residuals positive in the middle and negative at both ends or the other way round. The relationship bends and the line does not; the fix is a curve or a transformed variable rather than a better straight line, which is what Non-Linear Relationships & Transformations is for.
3. A wedge, narrow at one end and wide at the other. The scatter around the line grows with x, so predictions are much more reliable at one end of the range than the other, and the standard errors that Stats 2 computes from a single shared spread will be wrong.

Move the sliders off the best fit and a fourth shape appears that is not about the data at all: the residuals tilt, sloping up or down across the plot. A tilt means the line still has a trend left in it to absorb, which is exactly what "least squares" removes. Snap back to the best fit and the tilt disappears every time, because the residuals from the least-squares line always sum to zero and always have zero correlation with x. Regression Diagnostics in Stats 2 turns these readings into a full procedure, standardized residuals and influence measures included.

Where the name comes from

Francis Galton measured the heights of parents and their grown children in 1886 and found something he called regression toward mediocrity: unusually tall parents had tall children, but on average less tall than themselves. The word stuck to the method, and the effect it names is still one of the easiest ways to fool yourself.

It falls straight out of the slope. Put both variables in standard-deviation units and the least-squares slope becomes exactly r, so a predicted score is always closer to its mean, in SD units, than the predictor was. Take a test whose retest correlation is .70. Of the people who scored 2 SD above the mean on Monday, the group averages 1.4 SD on Friday, and 80% of them score lower the second time. No fatigue, no complacency, no lost form. The same arithmetic runs in reverse for the people who bombed on Monday: they will look improved on Friday, on average, having done nothing.

When a third variable flips the line

A fitted line describes the points you gave it, and it has no way of knowing which of them belong together. §1.17 introduces the lurking variable, a third quantity nobody measured that is driving both of the two that were. When that third quantity is categorical, a group label rather than a number, it can do something more dramatic than weaken a relationship. It can reverse it.

Two clinics treat the same condition with the same drug and record, for each patient, the dose given and how well they recovered. Inside each clinic the relationship is clear and positive: more drug, better recovery. The second clinic is a specialist referral center, so its patients arrive sicker, recover less well whatever is done for them, and are given larger doses because they are sicker. Pool the two clinics into one scatter and the picture inverts.

Fitted separately, clinic A gives a slope of +4.0 and clinic B a slope of +3.9, both with correlations above .98. Fitted to all ten patients at once, the slope is −3.2 and the correlation is −.60. Nothing in the pooled numbers hints at the problem, and no amount of care with the arithmetic would find it. What finds it is plotting the points with the group marked, which costs one extra column in the data file. This reversal has a name, Simpson's paradox, and the general repair is to put the group into the model as a predictor of its own, which is where multiple regression begins.

Problem 21 in the practice problems fits a line to that same six-person dataset, predicts a value, and checks the residual against it.

Problem 24 builds a line from five summary numbers alone, the way an exam gives them, and problem 25 reads a residual plot whose R² of .500 hides a curve.

Everything on this page is description: the line that fits these points best, and how well it fits. Whether the slope would survive another sample is a separate question with its own machinery, and Inference for Regression in Stats 2 supplies it: the standard error of the slope, its t test and confidence interval, the regression ANOVA table, and the two bands around the fitted line.