Prediction vs Explanation
You already know how to fit a regression. Machine learning uses the same math, but often for a different purpose, and the purpose changes what counts as a "good" model. If you are coming to ML from statistics, this is the most useful idea to learn first, because it tells you which tools to use and which mistakes to watch for.
Two questions, two cultures
In 2001 the statistician Leo Breiman described "two cultures" of modeling. They differ in the question they ask:
- Explanation (inference). Which variables matter, how much, and how sure are we? You want interpretable coefficients, standard errors, p-values, confidence intervals: a model you can put in a sentence and defend. This is the world of your stats courses: regression, hypothesis tests, effect sizes.
- Prediction. Given a new case, what's my best guess, and how wrong will I be? You care about error on data the model hasn't seen. Whatever machinery minimizes that error is fair game, interpretable or not. This is the world of ML: cross-validation, held-out test sets, "whatever works."
Neither culture is superior; they answer different questions. A drug trial needs explanation: the size and certainty of the effect, so a regulator can decide. A spam filter needs prediction. It doesn't matter why an email is spam, only that the label is right on tomorrow's inbox.
The same model, two different jobs
The same linear model can serve either goal. Fit a straight line and you can report its slope with a confidence interval (explanation), or use it to predict new points and measure the error (prediction). The model is identical. Only the way you judge it changes, and when you are free to choose the complexity, the two ways of judging lead to different models.
Explanation values parsimony: every extra term is another coefficient you must interpret and defend. Prediction values generalization: an extra term is welcome if it lowers error on new data, even when its meaning is unclear. The widget below shows the two goals choosing different models on one dataset.
🎯 One dataset, two goals
The dots are a noisy sample from the dashed curve (the truth). Move the flexibility slider, or let each goal pick for itself. Explanation adds a fixed penalty, an "interpretability tax", for every extra term, so it keeps a term only if the term lowers the error by more than the penalty. Prediction looks only at held-out (cross-validated) error, so it adds flexibility as long as that error keeps falling. On the same data, the two goals can pick different models.
Explanation almost always settles on the straight line, with one slope you can report and interpret, because the curved terms do not lower the error by enough to cover their penalty. Prediction adds curvature because the curvature lowers held-out error. Both are correct answers to different questions. Don't expect the explanation model to also predict best, or the prediction model to give you one clean coefficient to interpret. Each model answers only its own question.
When each mindset wins
- Reach for explanation when the deliverable is understanding: does this therapy work, which predictors drive burnout, is the effect big enough to matter. You'll report coefficients, CIs, and effect sizes, and you'll keep the model small enough to interpret.
- Reach for prediction when the deliverable is a decision on new cases: flag at-risk students, triage scans, forecast demand. You'll split off a test set, tune with cross-validation, and judge the model purely by out-of-sample error.
Many real projects want both, a model that predicts well and offers some insight. That is fine as long as you know which goal you are pursuing at each step. The danger is doing one while believing you are doing the other.
Why the two cultures report different things
ML papers rarely show p-values and statistics papers rarely show test-set error, because they optimize different quantities. A p-value answers "could this coefficient be zero?"; it says nothing about how well the model predicts tomorrow. A cross-validated error answers "how wrong will I be on new cases?"; it says nothing about whether any single coefficient is real. A model can predict well although nothing in it can be read as an effect (a random forest), or explain clearly while predicting no better than a coin flip on a hard problem. Ask for the number that matches your question.
Reading every coefficient as an effect
A common way to blur the two goals is to fit one multiple regression and then read its whole coefficient table as a list of effects. Epidemiologists named this the Table 2 fallacy, after the table where it usually turns up. The model was built to estimate one thing well, and only that one row means what the column heading claims.
Take a study of training (X) and performance (Y) in which motivation (Z) raises both. Motivation confounds the training effect, so you adjust for it. The adjustment works: with data generated so that the true training effect is 0.50, the unadjusted slope is 0.68 and the adjusted slope is 0.50. Now look one row down. Motivation's own coefficient in the same model is 0.40, but motivation's total effect on performance is 0.70. Part of that effect runs through training, and the model holds training fixed. If you read the table as "the effect of each variable", you understate motivation's effect by more than 40%.
The model is not wrong. Adjusting for a confounder gives you an interpretable coefficient for the one predictor you chose the adjustment for. Every other coefficient becomes a direct effect, with the path through your predictor blocked. Your causal diagram tells you which coefficients you may interpret. The output itself cannot tell you. If you also want the effect of motivation, you need a second model with a second adjustment set.
Why it matters: before you fit anything, decide whether you're explaining or predicting. That choice determines your measure of success (a p-value or CI, or held-out error), how complex a model you should accept, and which pitfalls you must guard against. The rest of this course builds on it. If you are planning a dissertation and need to pick a test, Choosing Statistics for Your Dissertation starts from the same choice.
Common questions
Is machine learning just statistics with a fancier name?
The mathematics is the same, but the emphasis is different. Much of ML is built directly on statistical models (regression, logistic regression, and their relatives), but the two fields optimize different things. Classical statistics is usually doing inference: which variables matter, how large the effect is, and how certain we are (coefficients, confidence intervals, p-values). ML is usually doing prediction: how accurately it can guess the outcome for cases it has never seen, measured by held-out error. The two fields overlap and borrow from each other all the time, but calling ML rebranded statistics ignores this difference in goals. ML will use an uninterpretable model if it predicts well, and it rarely reports p-values. Inferential statistics keeps models interpretable and rarely reports test-set error.
Can I do machine learning with only 80 participants?
You can fit the models, but you will not be able to tell whether they worked. Split 80 people 60/20 and your test score rests on 20 cases. A model that gets 16 of them right scores 80%, with a 95% interval from 58% to 92%, which is compatible with both 'excellent' and 'barely better than guessing'. Flexible models also need a lot of data before their flexibility helps. At this sample size, a regression with two or three well-chosen predictors usually predicts about as well as anything fancier, and its coefficients can be interpreted. If you do build a predictive model, use repeated k-fold or nested cross-validation rather than a single split, so that every case takes a turn in the test fold. Report the interval around your score along with the score.
Can the same model be good for both prediction and explanation?
Sometimes. A simple, well-specified linear model often does a reasonable job of both, but don't assume it. The two goals pull in different directions. Explanation favors parsimony (every extra term is another thing to interpret and defend), while prediction favors whatever lowers held-out error (often more flexibility). When they conflict, pick the model that matches your main goal and say which goal it is. The classic mistake is to build a black-box predictor and then read its internals as if they were causal effects. A model optimized for prediction gives you no guarantee that its internals mean anything causal.