Simple Linear Regression

Correlation tells you two variables move together. Regression goes further: it draws the single best straight line through the cloud, so you can predict one variable from the other. The work is in choosing that line, and there is a precise rule for what "best" means.

The line and its errors

A regression line is just ŷ = b₀ + b₁x: an intercept and a slope. For any line you propose, each data point is some vertical distance above or below it. That gap is the residual: the prediction error for that point.

A good line makes those residuals small overall. But small how? Add them up and positives cancel negatives. The fix is to square each residual (which removes the sign and penalizes big misses heavily), then add them up. That total is the sum of squared residuals (SSR), and the least-squares line is the one, out of all possible lines, that makes it as small as possible.

What the slope and intercept mean

• Slope b₁: how much ŷ changes for each one-unit increase in x. A slope of 0.6 means "every extra point of x goes with 0.6 more y, on average."
• Intercept b₀: the predicted y when x = 0. Sometimes meaningful, sometimes just where the line crosses the axis.

Finding the line by hand

Moving the sliders until the error stops falling shows the idea, but you would not want to do it on paper. There is a formula, and it needs only quantities you already computed in §1.17: the two means, the two standard deviations, and the correlation.

b₁ = r · s_y / s_x     b₀ = ȳ − b₁ x̄

The slope formula converts units. The correlation gives the number of standard deviations of y per standard deviation of x, and multiplying by s_y / s_x converts that rate back into the units the variables were measured in. The intercept formula is a rearrangement of one fact to remember: the least-squares line always passes through (x̄, ȳ), the point of means. Whatever the slope, the line goes through that point. So a line fitted to data centered on zero in both variables has an intercept of zero.

Take the five music students from §1.17: x̄ = 3.0 practice hours, ȳ = 66.0 marks, s_x = 1.58, s_y = 9.62 and r = 0.740. Then b₁ = 0.740 × 9.62 / 1.58 = 4.5 marks per hour (the rounded inputs give 4.506; the unrounded ones give exactly 4.5), and b₀ = 66.0 − 4.5 × 3.0 = 52.5, so the line is ŷ = 52.5 + 4.5x. Substituting a value of x gives the predicted value ŷ. A student practicing 4 hours is predicted to score 52.5 + 18 = 70.5. The fourth student practiced 4 hours and scored 65, so their residual is 65 − 70.5 = −5.5, five and a half marks below the prediction.

R²: how much the line explains

Once you've got the best line, R² (which is just the correlation r squared) says what fraction of the variation in y the line accounts for. R² = 0.7 means the line explains 70% of the ups and downs in y; the rest is scatter the line can't capture. Generate new data a few times and watch SSR and R² move together: tighter clouds give smaller SSR and bigger R².

Memorize the textbook phrasing, because it names both parts: R² is the fraction of the variation in the values of y that is explained by the least-squares regression of y on x. The variation is the spread of y around its own mean. The comparison is with the flat line at ȳ, the best prediction you could make knowing nothing about x. The five music students give R² = 0.740² = 0.547, so practice time accounts for about 55% of the differences between their marks and leaves 45% to everything else.

Reading a residual plot

R² is one number for the whole fit, and it cannot tell you where a line goes wrong. For that you need the residual plot: the same points again, but with each one's residual on the vertical axis, and a line at zero for reference. Tick "Show the residual plot" in the playground above and it appears under the scatter, drawn for whatever line the sliders currently set.

The plot is useful because the line has been subtracted. In a scatter with a steep trend, small departures from the line are hard to see. Take the line away and only those departures are left, spread across the full height of the panel. There are three shapes to know.

1. A shapeless band of points scattered evenly above and below zero, with no drift and no fanning. This is what a good straight-line fit looks like: the line has removed everything systematic from the data.
2. An arch or a valley, all the residuals positive in the middle and negative at both ends or the other way round. The relationship bends and the line does not. Fit a curve or transform a variable, and Non-Linear Relationships & Transformations covers both.
3. A wedge, narrow at one end and wide at the other. The scatter around the line grows with x, so predictions are much more reliable at one end of the range than the other. The standard errors that Stats 2 computes from a single shared spread will also be wrong.

Move the sliders off the best fit and a fourth shape appears, caused by the line and not the data: the residuals tilt up or down across the plot. A tilt means the line has missed part of the trend, and least squares removes it. Snap back to the best fit and the tilt disappears every time, because the residuals from the least-squares line always sum to zero and always have zero correlation with x. Regression Diagnostics in Stats 2 turns these readings into a full procedure, including standardized residuals and influence measures.

Where the name comes from

Francis Galton measured the heights of parents and their grown children in 1886 and found something he called regression toward mediocrity: unusually tall parents had tall children, but on average less tall than themselves. The word stuck to the method, and the effect it names is still easy to mistake for something real.

It follows from the slope. Put both variables in standard-deviation units and the least-squares slope is exactly r, so a predicted score is always closer to its mean, in SD units, than the predictor was. Take a test whose retest correlation is .70. People who scored 2 SD above the mean on Monday average 1.4 SD on Friday, and 80% of them score lower the second time. None of this needs fatigue, complacency or lost form. The same arithmetic works in reverse for people who did badly on Monday: on average they will look improved on Friday without having done anything.

When a third variable flips the line

A fitted line describes the points you give it and cannot tell which of them belong together. §1.17 introduces the lurking variable, an unmeasured third quantity that drives both of the measured ones. When that third quantity is categorical, a group label and not a number, it can do more than weaken a relationship: it can reverse it.

Two clinics treat the same condition with the same drug and record, for each patient, the dose given and how well they recovered. Inside each clinic the relationship is clear and positive: more drug, better recovery. The second clinic is a specialist referral center, so its patients arrive sicker, recover less well whatever is done for them, and are given larger doses because they are sicker. Pool the two clinics into one scatter and the relationship reverses.

Fitted separately, clinic A gives a slope of +4.0 and clinic B a slope of +3.9, both with correlations above .98. Fitted to all ten patients at once, the slope is −3.2 and the correlation is −.60. Nothing in the pooled numbers hints at the problem, and careful arithmetic would not find it. Plotting the points with the group marked does, and that needs only one extra column in the data file. This reversal is called Simpson's paradox. The general fix is to add the group to the model as a predictor of its own, and that is the starting point of multiple regression.

Problem 42 in the practice problems fits a line to that same six-person dataset, predicts a value, and checks the residual against it.

Problem 24 builds a line from five summary numbers alone, the way an exam gives them, and problem 25 reads a residual plot whose R² of .500 hides a curve.

Everything on this page is description: the line that fits these points best, and how well it fits. Whether the slope would hold up in another sample is a separate question. Inference for Regression in Stats 2 answers it with the standard error of the slope, its t test and confidence interval, the regression ANOVA table, and the two bands around the fitted line.