Section 5.3

Regularization: Ridge & Lasso

Q: Does regularization work for logistic regression too?

Yes, and in scikit-learn it is switched on before you ask. The penalty attaches to whatever the model is minimizing, so for logistic regression it rides on the log-likelihood instead of the sum of squares, and everything else carries straight over: ridge shrinks all the log-odds coefficients, lasso zeroes the weak ones, cross-validation picks λ, and you still standardize the predictors first. In R that is glmnet(..., family = "binomial"). In Python LogisticRegression applies an L2 penalty by default at C = 1, where C is the inverse of λ, so an unpenalized fit is the thing you have to request (penalty=None). A penalty also rescues the case where ordinary logistic regression fails outright: when a predictor separates the outcome perfectly the unpenalized coefficients run off toward infinity, and shrinking them (Firth's method is the classic version) returns a finite, reportable estimate.

Q: What is lambda, and how do I choose it?

Lambda (λ) is the penalty strength, the dial controlling how hard the coefficients are shrunk. At λ = 0 you have ordinary least squares with no shrinkage; as λ grows, coefficients shrink toward zero and the model becomes simpler, more biased, and less variable. You don't set λ by hand: fit the model across a grid of λ values and choose the one with the lowest cross-validated error (cv.glmnet in R, LassoCV/RidgeCV in Python). A common, slightly conservative choice is the largest λ within one standard error of the minimum ('lambda.1se'), which buys a simpler model for almost the same error.

Q: Do I need to standardize my predictors before ridge or lasso?

Almost always, yes. The penalty acts on the size of the coefficients, and a coefficient's size depends on its predictor's units — the same variable measured in grams gets a coefficient a thousand times larger than in kilograms, and would then be penalized a thousand times more heavily. Standardizing every predictor to mean 0 and standard deviation 1 first puts them on equal footing so the penalty is fair. Many packages (such as glmnet) standardize internally by default and report the coefficients back on the original scale, but if yours doesn't, do it yourself. The outcome variable usually doesn't need standardizing — only the predictors.

Ordinary regression finds the coefficients that fit your sample best. But "best fit to this sample" is a trap when your goal is prediction — it happily assigns big coefficients to noise. Regularization fixes this with a deceptively simple move: deliberately shrink the coefficients toward zero. Accepting a little bias buys a large drop in variance, and the model that generalizes better is the one that didn't try quite so hard to fit.

Why shrinking helps

Least squares minimizes the training error alone, so with many predictors (especially correlated ones) it produces large, unstable coefficients that swing wildly from sample to sample. That instability is variance, and variance is prediction error. Regularization adds a penalty on coefficient size to the thing being minimized:

minimize (training error) + λ × (size of the coefficients)

The penalty pulls every coefficient toward zero. Big, confident coefficients now have to earn their size by reducing error enough to outweigh the penalty. Noise predictors can't, so they get crushed; real predictors survive. This is the bias–variance trade-off turned into a dial: a bit more bias, a lot less variance, lower error on new data.

Ridge shrinks, lasso selects

The two workhorses differ only in how they measure "size of the coefficients":

Ridge penalizes the sum of squared coefficients (the L2 penalty). It shrinks everything smoothly toward zero but never exactly to zero — every predictor stays in the model, just quieter. Ridge shines when predictors are correlated: it spreads the credit across them instead of letting the estimates explode.
Lasso penalizes the sum of absolute coefficients (the L1 penalty). Its geometry drives weak coefficients to exactly zero, so lasso does automatic variable selection — it hands you a shorter model with some predictors switched off entirely.

The penalty strength λ is the flexibility dial: λ = 0 is ordinary least squares; as λ grows, coefficients shrink toward a flat, all-zero model. You don't guess λ; you choose it by cross-validation, picking the value with the lowest held-out error. One prerequisite: standardize your predictors first, so the penalty treats a variable measured in grams the same as one measured in kilograms.

📉 The coefficient-path explorer

Eight standardized predictors feed the outcome: 3 are real and 5 are pure noise. The top panel traces each coefficient as the penalty λ grows (left = none, right = heavy). Below, cross-validated error picks the best λ. Switch between ridge and lasso, and watch the noise coefficients die before the real ones.

Penalty strength λ = 0

Nonzero coefficients—

Noise still in the model—

Cross-validated MSE—

Drag λ up from zero and the story is always the same: the five noise coefficients (which only ever fit random wiggles) collapse toward zero first, while the three real coefficients hold on much longer. With lasso the noise coefficients snap to exactly zero one by one, literally deleting the junk predictors; with ridge they shrink smoothly but linger. And the cross-validated error dips to a minimum in the middle — that best-λ model has quieted the noise without yet crippling the signal. That's regularization doing your variable selection and your overfitting control in one move.

Which one should you use?

Lasso when you want a sparse, interpretable model or suspect many predictors are irrelevant — it picks a subset for you. Caveat: among a group of correlated predictors it tends to keep one somewhat arbitrarily and zero the rest.
Ridge when you believe most predictors matter a little, or when they're highly correlated and you'd rather share the coefficient than have lasso choose one at random.
Elastic net when you can't decide — it blends the two penalties, getting lasso's selection while handling correlated groups gracefully. It's the pragmatic default in modern predictive modeling.

All three are still linear models: the coefficients remain the familiar "change in y per unit of x," just shrunk. That makes regularization the friendliest possible on-ramp from the regression you know to the predictive modeling of the rest of this course.

What you may not do with the survivors

Lasso hands back a short list of predictors, and the next move is almost irresistible: refit those few with lm() and report their coefficients, standard errors and p-values. Those p-values are not valid. The same data chose the predictors and then tested them, so every survivor is on the list precisely because it looked strong in this sample, and the test has no way of knowing it was auditioned.

The size of the distortion is easy to measure. Give lasso 200 rows and 40 predictors that are pure noise, with an outcome unrelated to every one of them, and set the penalty so it keeps about six. It does keep about six: on data with nothing in it, lasso still returns a model. Refit those six by least squares and 32.8% of their coefficients come back with p < .05, where an honest test would give 5%. Run the same fit on six predictors picked at random before anyone looked at the outcome and the rate is 4.7%, median p = .49. Selection did that, not the arithmetic.

Two honest ways forward. If the model is for prediction, judge it by held-out error and report no p-values from it at all. If you want inference about particular predictors, split the sample: choose the model on one half, then estimate and test on the other. That costs power and buys p-values that mean what they say. There is also a formal machinery, post-selection inference, whose intervals widen to pay for the search (R's selectiveInference), and the phrase is worth knowing so you recognize a reviewer asking for it.

Why it matters: unpenalized regression overfits whenever predictors are many or correlated, inventing large coefficients for noise. Ridge and lasso trade a little bias for much less variance (shrinking, and in lasso's case deleting, the coefficients your data can't actually support), and cross-validation tells you exactly how hard to shrink.

Common questions

Does regularization work for logistic regression too?

Yes, and in scikit-learn it is switched on before you ask. The penalty attaches to whatever the model is minimizing, so for logistic regression it rides on the log-likelihood instead of the sum of squares, and everything else carries straight over: ridge shrinks all the log-odds coefficients, lasso zeroes the weak ones, cross-validation picks λ, and you still standardize the predictors first. In R that is glmnet(..., family = "binomial"). In Python LogisticRegression applies an L2 penalty by default at C = 1, where C is the inverse of λ, so an unpenalized fit is the thing you have to request (penalty=None). A penalty also rescues the case where ordinary logistic regression fails outright: when a predictor separates the outcome perfectly the unpenalized coefficients run off toward infinity, and shrinking them (Firth's method is the classic version) returns a finite, reportable estimate.

What is lambda, and how do I choose it?

Lambda (λ) is the penalty strength, the dial controlling how hard the coefficients are shrunk. At λ = 0 you have ordinary least squares with no shrinkage; as λ grows, coefficients shrink toward zero and the model becomes simpler, more biased, and less variable. You don't set λ by hand: fit the model across a grid of λ values and choose the one with the lowest cross-validated error (cv.glmnet in R, LassoCV/RidgeCV in Python). A common, slightly conservative choice is the largest λ within one standard error of the minimum ('lambda.1se'), which buys a simpler model for almost the same error.

Do I need to standardize my predictors before ridge or lasso?

Almost always, yes. The penalty acts on the size of the coefficients, and a coefficient's size depends on its predictor's units — the same variable measured in grams gets a coefficient a thousand times larger than in kilograms, and would then be penalized a thousand times more heavily. Standardizing every predictor to mean 0 and standard deviation 1 first puts them on equal footing so the penalty is fair. Many packages (such as glmnet) standardize internally by default and report the coefficients back on the original scale, but if yours doesn't, do it yourself. The outcome variable usually doesn't need standardizing — only the predictors.