Regularization: Ridge & Lasso

Ordinary regression finds the coefficients that fit your sample best. When your goal is prediction, that is a problem, because the best fit to one sample gives large coefficients to noise. Regularization fixes this by shrinking the coefficients toward zero on purpose. The shrunk coefficients are slightly biased but much less variable, so the model predicts new cases better.

Why shrinking helps

Least squares minimizes the training error alone, so with many predictors (especially correlated ones) it produces large, unstable coefficients that change a lot from sample to sample. That instability is variance, and variance is part of prediction error. Regularization adds a penalty on coefficient size to the thing being minimized:

minimize   (training error)  +  λ × (size of the coefficients)

The penalty pulls every coefficient toward zero. A coefficient stays large only if it reduces the training error by more than the penalty adds. Noise predictors reduce the error very little, so their coefficients shrink to near zero, while real predictors keep more of theirs. λ sets the bias–variance trade-off: a little more bias, much less variance, and lower error on new data.

Ridge shrinks, lasso selects

The two main methods differ only in how they measure "size of the coefficients":

• Ridge penalizes the sum of squared coefficients (the L2 penalty). It shrinks every coefficient smoothly toward zero but never exactly to zero, so every predictor stays in the model with a smaller coefficient. Ridge works well when predictors are correlated: it spreads the weight across them, so the estimates do not become huge and unstable.
• Lasso penalizes the sum of absolute coefficients (the L1 penalty). The shape of this penalty drives weak coefficients to exactly zero, so lasso does automatic variable selection: the final model leaves some predictors out entirely.

The penalty strength λ controls how flexible the model is. At λ = 0 you have ordinary least squares, and as λ grows, the coefficients shrink toward a flat model with every coefficient at zero. You don't guess λ. You choose it by cross-validation, picking the value with the lowest held-out error. One more requirement: standardize your predictors first, so the penalty treats a variable measured in grams the same as one measured in kilograms.

As you raise λ from zero, the five noise coefficients (which only ever fit random variation) shrink toward zero first, and the three real coefficients stay large much longer. With lasso the noise coefficients reach exactly zero one by one, removing those predictors from the model. With ridge they shrink smoothly but never reach zero. The cross-validated error is lowest at a λ in the middle, where the noise coefficients are small and the real ones are still large.

Which one should you use?

• Lasso when you want a sparse, interpretable model or suspect many predictors are irrelevant, since it picks a subset for you. Caveat: among a group of correlated predictors it tends to keep one somewhat arbitrarily and zero the rest.
• Ridge when you believe most predictors matter a little, or when they're highly correlated and you'd rather share the coefficient than have lasso choose one at random.
• Elastic net when you can't decide: it combines the two penalties, so it selects variables like lasso but tends to keep or drop correlated predictors together. Many people use it as the default in predictive modeling.

All three are still linear models: each coefficient is still the familiar "change in y per unit of x," only shrunk. So regularization is the easiest step from the regression you know to the predictive models in the rest of this course.

What you may not do with the survivors

Lasso returns a short list of predictors, and it is tempting to refit those few with lm() and report their coefficients, standard errors and p-values. Those p-values are not valid. The same data chose the predictors and then tested them. Every predictor on the list is there because it looked strong in this sample, and the test does not take that selection into account.

A simulation shows how large the distortion is. Give lasso 200 rows and 40 predictors that are pure noise, with an outcome unrelated to every one of them, and set the penalty so it keeps about six. It does keep about six, so even on data with no signal, lasso still returns a model. Refit those six by least squares and 32.8% of their coefficients come back with p < .05, where a valid test would give 5%. Run the same fit on six predictors picked at random before anyone looked at the outcome and the rate is 4.7%, median p = .49. So the excess comes from the selection step and not from the least-squares fit.

There are two valid ways forward. If the model is for prediction, judge it by held-out error and report no p-values from it at all. If you want inference about particular predictors, split the sample: choose the model on one half, then estimate and test on the other. You lose power, but the p-values are valid. There is also a formal method, post-selection inference, whose intervals are wider to account for the search (R's selectiveInference). Reviewers sometimes ask for it by that name.