Model Comparison
You can almost always make a model fit your data better: just add more predictors or more flexibility. But "fits the data I have" and "captures the real pattern" are different things, and chasing the first leads straight to overfitting: a model that has memorized the noise and falls apart on anything new.
Why R² can't pick the winner
Adding any predictor (even pure random noise) never decreases R². So if you choose models by R² alone, you'll always crown the most complicated one. You need a criterion that rewards good fit but penalizes needless complexity. The clearest way to see why is to watch a model overfit in real time.
🎮 Overfitting in Action
Fit a polynomial of rising degree to the blue training points. Watch it bend to hit every one, while its error on fresh orange test points (data it never saw) gets worse.
Notice the pattern: training R² climbs steadily toward a perfect 1.0 as the curve wiggles through every training point. But test error falls, bottoms out near the true complexity, and then shoots back up as the model starts fitting noise. That U-shape is the signature of the bias–variance trade-off: too simple underfits, too complex overfits, and the best model lives in between.
Tools for honest comparison
- Adjusted R² — like R², but docks you for each predictor added; it can go down if a variable doesn't earn its keep.
- AIC & BIC — information criteria that score fit minus a complexity penalty (BIC penalizes harder). Lower is better, and they let you compare non-nested models.
- Nested F-test — for models where one is a special case of the other, tests whether the extra terms significantly improve the fit.
- Cross-validation — the gold standard: actually measure error on held-out data, exactly like the orange points here.
Parsimony wins. When two models explain the data about equally well, prefer the simpler one. It's easier to interpret, less likely to be overfit, and more likely to hold up on new data. Occam's razor is a measurable advantage in prediction here, not just a philosophical preference.
Why it matters: the goal of a model is rarely to describe the sample you collected; it's to generalize to cases you haven't seen. Model comparison is the discipline of choosing for the future, not flattering the past. It's the bridge from classical statistics to machine learning, which takes this one idea and builds a whole field on it.
Problem 37 of the practice problems hands you four nested models with nothing but their log-likelihoods and parameter counts, and asks you to compute AIC and BIC yourself, weigh the best-fitting model against the simplest one, and say which you would actually report.
Common questions
What is the difference between AIC and BIC?
Both score models as fit-minus-complexity-penalty (lower is better), but BIC's penalty grows with sample size (k·ln n vs AIC's 2k), so it favors leaner models, especially in big data. Philosophically, AIC aims to minimize prediction error; BIC aims to identify the true model among candidates. In practice: report both, and take notice when they disagree.
Can I compare AIC across models fitted to different data?
Only if the models were fitted to exactly the same rows. AIC is a relative score with no meaning on its own, so the comparison is valid only when the likelihoods being compared refer to the same observations. The usual way people break this without noticing: one model includes a predictor with missing values, the software drops those cases without saying so, and the two models are now fitted to different sample sizes, which makes their AICs incomparable no matter how carefully you computed them. Check that n is identical in every model you rank, and if it is not, refit them all on the complete-case subset or handle the missing values first. The same rule governs BIC, and it is also why AIC cannot referee two models with different outcome variables or different transformations of the outcome.
What is a nested model?
One model is nested in another when it's a special case — obtainable by deleting predictors (setting their coefficients to zero). y ~ x₁ + x₂ is nested in y ~ x₁ + x₂ + x₃. Nesting matters because it licenses an exact significance test (the nested F-test, or likelihood-ratio test) for whether the extra terms earn their keep; non-nested rivals must be compared with AIC/BIC or cross-validation.