Prediction vs Explanation
You already know how to fit a regression. Machine learning uses the very same math, but often for a different purpose, and that change of purpose reshapes what counts as a "good" model. Getting this distinction straight is the single most useful idea for a stats student stepping into ML: it tells you which tools to reach for and which mistakes to fear.
Two questions, two cultures
In 2001 the statistician Leo Breiman described "two cultures" of modeling, and the split still organizes the whole field. Everything comes down to which question you're actually asking:
- Explanation (inference). Which variables matter, how much, and how sure are we? You want interpretable coefficients, standard errors, p-values, confidence intervals — a model you can put in a sentence and defend. This is the world of your stats courses: regression, hypothesis tests, effect sizes.
- Prediction. Given a new case, what's my best guess — and how wrong will I be? You care about error on data the model hasn't seen. Whatever machinery minimizes that error is fair game, interpretable or not. This is the world of ML: cross-validation, held-out test sets, "whatever works."
Neither culture is superior; they answer different questions. A drug trial needs explanation: the size and certainty of the effect, so a regulator can decide. A spam filter needs prediction — it doesn't matter why an email is spam, only that the label is right on tomorrow's inbox.
The same model, two different jobs
The catch that trips people up: the exact same linear model can serve either master. Fit a straight line and you can report its slope with a confidence interval (explanation), or use it to predict new points and measure the error (prediction). The model is identical. What differs is how you judge it, and that judgment pushes you toward different models when you're allowed to choose the complexity.
Explanation prizes parsimony: every extra term is another coefficient you must interpret, defend, and worry about. Prediction prizes generalization: an extra term is welcome the moment it lowers error on new data, however murky its meaning. Watch those two pressures pull in different directions on one dataset.
🎯 One dataset, two goals
The dots are a noisy sample from the dashed curve (the truth). Slide the flexibility dial, or let each goal pick for itself. Explanation pays an "interpretability tax" for every extra term, so it stops at the simplest faithful model. Prediction only watches held-out (cross-validated) error, so it keeps flexibility as long as error keeps falling. Same data — two different winners.
Notice the shape of the story. Explanation almost always settles on the plain straight line — one slope you can report and interpret, and it refuses to buy extra terms it can't justify. Prediction is happy to add curvature because it genuinely lowers held-out error. Both are correct answers to different questions. If you tried to force the explanation model to also be the best predictor, or demanded the prediction model hand you a clean interpretable coefficient, you'd be disappointed, not because the model failed, but because you asked it the wrong question.
When each mindset wins
A quick field guide for your own projects:
- Reach for explanation when the deliverable is understanding: does this therapy work, which predictors drive burnout, is the effect big enough to matter. You'll report coefficients, CIs, and effect sizes, and you'll keep the model small enough to interpret.
- Reach for prediction when the deliverable is a decision on new cases: flag at-risk students, triage scans, forecast demand. You'll split off a test set, tune with cross-validation, and judge the model purely by out-of-sample error.
Plenty of real projects want both (a model that predicts well and offers some insight), and that's fine, as long as you know which hat you're wearing at each step. The danger is doing one while silently believing you're doing the other.
Why the two cultures report different things
This is why ML papers rarely show p-values and stats papers rarely show test-set error — they're optimizing different quantities. A p-value answers "could this coefficient be zero?"; it says nothing about how well the model predicts tomorrow. A cross-validated error answers "how wrong will I be on new cases?"; it says nothing about whether any single coefficient is real. A model can predict beautifully with every coefficient uninterpretable (a random forest), or explain cleanly while predicting no better than a coin flip on genuinely hard problems. Ask for the number that matches your question.
Reading every coefficient as an effect
A common way to blur the two goals is to fit one multiple regression and then read its whole coefficient table as a list of effects. Epidemiologists named this the Table 2 fallacy, after the table where it usually turns up. The model was built to estimate one thing well, and only that one row means what the column heading claims.
Take a study of training (X) and performance (Y) in which motivation (Z) raises both. Motivation confounds the training effect, so you adjust for it, and the adjustment does its job: with data generated so the true training effect is 0.50, the unadjusted slope reads 0.68 and the adjusted one lands on 0.50. Now look one row down. Motivation's own coefficient in that same model is 0.40, while motivation's total effect on performance is 0.70, because part of that effect travels through training and the model has just held training fixed. Read the table as "the effect of each variable" and you have understated motivation by more than 40%.
Nothing is broken. Adjusting for a confounder buys an interpretable coefficient for the one predictor you chose the adjustment for, and it pays for that by turning every other row into a direct effect with your predictor blocked. Which coefficients you may interpret is a question about your causal diagram, not about the output. Wanting the effect of motivation as well means a second model with a second adjustment set.
Why it matters: before you fit anything, decide whether you're explaining or predicting. That one choice determines your success metric (a p-value / CI vs held-out error), how complex a model you should accept, and which pitfalls you must guard against — and it's the thread running through the rest of this course. Students arriving from the other direction, with a dissertation to plan and a test to pick, meet the same fork in Choosing Statistics for Your Dissertation.
Common questions
Is machine learning just statistics with a fancier name?
Same mathematics, genuinely different emphasis. Much of ML is built directly on statistical models (regression, logistic regression, and their relatives), but the two fields optimize different things. Classical statistics is usually doing inference: which variables matter, how large the effect is, and how certain we are (coefficients, confidence intervals, p-values). ML is usually doing prediction: how accurately it can guess the outcome for cases it has never seen, measured by held-out error. The cultures overlap and borrow constantly, but 'just rebranded' undersells the shift — ML happily uses uninterpretable models if they predict well and rarely reports p-values, while inferential statistics keeps models interpretable and rarely reports test-set error.
Can I do machine learning with only 80 participants?
You can fit the models; the honest difficulty is that you will not be able to tell whether they worked. Split 80 people 60/20 and your test score rests on 20 cases, so a model that gets 16 of them right scores 80% with a 95% interval running from 58% to 92%, which is compatible with 'excellent' and with 'barely better than guessing' at once. Flexible models also need data to earn their flexibility, so at this size a regression with two or three well-chosen predictors usually predicts about as well as anything fancier and hands you interpretable coefficients as a bonus. If you do go predictive, use repeated k-fold or nested cross-validation rather than a single split, so every case takes a turn in the test fold, and report the interval around your score rather than the score alone.
Can the same model be good for both prediction and explanation?
Sometimes: a simple, well-specified linear model often does a respectable job of both, but don't assume it. The two goals pull in different directions: explanation rewards parsimony (every extra term is another thing to interpret and defend), while prediction rewards whatever lowers held-out error (often more flexibility). When they conflict, pick the model that matches your real goal and say which goal it is. The classic mistake is to build a black-box predictor and then read its internals as if they were causal effects — a model optimized for prediction owes you no honest explanation.