Inference for Regression

Fitting a line to a scatterplot is description. Section 1.18 did that: least squares, the slope, the intercept, the residuals. This lesson asks the next question, which is whether the slope you got would survive a second sample, and it answers it with the same t machinery you already know.

The population regression model

Inference needs a model of where the data came from. The standard one says that for each value of x there is a whole population of y values, that their means fall exactly on a straight line, and that they scatter around it by the same amount everywhere:

y = β₀ + β₁x + ε ε ~ N(0, σ)

β₀ (beta-nought) and β₁ (beta-one) are the parameters, fixed and unknown, and σ (sigma) is the size of the scatter. The line you fit gives b₀ and b₁, which estimate them. Four conditions come packaged with that equation, and they are the ones Section 2.8 teaches you to check: the relationship really is linear, the observations are independent, the scatter is the same at every x, and the deviations are roughly normal.

σ is estimated by the residual standard error, sometimes just called s, which is the typical size of a residual:

s = √(Σ(y − ŷ)² / (n − 2))

The divisor is n − 2 because fitting the line used up two degrees of freedom, one for the slope and one for the intercept. Every test below runs on those n − 2.

The standard error of the slope

How precisely you know the slope depends on two things, and the formula puts them side by side:

SE(b₁) = s / √(Σ(x − x̄)²)

Tight scatter around the line makes s small and the slope precise. Widely spread x values make the denominator large and do the same. That second half is a design lesson rather than an analysis one: if you get to choose where to measure, spreading your x values out buys precision for free, which is why a study that samples only a narrow slice of the predictor struggles to detect anything. Slide the scatter up in the widget and watch the bands swell while the points stay in the same left-to-right pattern.

Testing and estimating the slope

The null hypothesis worth testing is that the predictor tells you nothing, H₀: β₁ = 0. As always, the statistic is estimate over standard error, and it is a t on n − 2 degrees of freedom:

t = b₁ / SE(b₁) CI: b₁ ± t* SE(b₁)

Take the shipped screen-time.csv, 60 people with daily screen hours and a sleep-quality score. The fitted line is ŷ = 6.845 − 0.235x, with s = 1.922 and SE(b₁) = 0.147. That gives t(58) = −1.598, p = .115, and a 95% interval for β₁ of [−0.529, 0.059].

The interval is the more informative half of that output. It contains zero, which is the same verdict the test just returned, and it also says what the data do and do not rule out: a drop of half a point of sleep quality per screen hour is entirely compatible with these 60 people, and so is a small increase. That is a study too small to settle the question, which is a different statement from "there is no effect", and Section 3.5 is about writing the difference carefully.

The ANOVA table for a regression

Software prints an analysis-of-variance table above the coefficients, and it is the same decomposition one-way ANOVA uses. Total variation in y splits into the part the line explains and the part it leaves behind:

SST = SSR + SSE

Two things in that table are worth pinning down. R² is SSR/SST, here 9.43/223.59 = .042, so the line accounts for about 4% of the variation in sleep quality. And the F statistic is exactly the square of the t from the slope test: 2.554 = (−1.598)², and both report p = .115. In simple regression the F test adds nothing, because there is only one predictor to test. It starts earning its keep in multiple regression, where it asks whether the predictors as a set do anything.

Confidence band or prediction band?

Both bands are centered on the same fitted line and they answer different questions, which is the distinction the widget exists to make visible.

A confidence interval for the mean response at a given x asks where the average y for everyone at that x lies. A prediction interval asks where one new individual at that x will land. The second has to absorb an extra source of variation, that individual's own deviation from the line, and its standard error carries an extra 1:

mean: s√(1/n + (x₀ − x̄)²/Σ(x − x̄)²) new case: s√(1 + 1/n + …)

At 6 screen hours the line predicts 5.44. The confidence interval for the mean runs [4.60, 6.28], a width of 1.68; the prediction interval for one person runs [1.50, 9.37], a width of 7.87. Nearly five times wider, and it barely narrows as n grows, because that leading 1 does not shrink with sample size. Collecting more data pins down the line; it never makes an individual predictable.

Both bands are narrowest at x̄ and flare toward the edges, since a small error in the slope is magnified by distance from the center. That flare is also the visual argument against extrapolating: the band is telling you how little the data constrain the line out there, before you even reach the range where the model may simply stop being true.