Train/Test Splits & Generalization
A predictive model is useful only if it works on data it has never seen. Machine learning has one golden rule: never judge a model on data it was trained on. A score on the training data tells you little about how the model will do on new cases.
The golden rule and the split
A flexible model can memorize its training data almost perfectly and still be useless on the next case. That is overfitting. To estimate how well a model generalizes to new data, you hold some data back:
- Training set: the model learns here (fits coefficients, grows trees, picks neighbors).
- Test set: set aside until the very end and used once, to report performance. If you look at it and then adjust the model, it is no longer a test set.
A typical split is 70–80% train, the rest test. For classification, stratify the split so each side keeps the same class proportions, otherwise a rare class can vanish from one side by chance. And when you need to compare or tune models, split a validation set off the training data (or use cross-validation on it), so the test set stays untouched.
Data leakage
Data leakage is when information from the test set gets into training. Nothing crashes and no warning appears. The reported score looks excellent, and the model then performs much worse on new data. Three leaks cause most of the problems:
- Preprocessing before the split. Scaling, or choosing "the most predictive features", using the whole dataset lets the test rows influence those choices. The test set helped build the model, so its score is biased upward.
- The same subjects on both sides. Duplicate rows, or several measurements from one person split between train and test, let the model recognize individuals it has already seen. The score goes up, but the model predicts new people no better.
- Tuning on the test set. Trying many models, thresholds or settings and reporting the one that scored best on the test set turns the test set into training data for your choices. The winning score is biased upward, because you chose it for being the highest.
The widget below runs a small predictive pipeline: a k-nearest-neighbors classifier on data where two features are weakly predictive and the rest are noise. Switch the leaks on and the pipeline's reported accuracy rises above its true accuracy on new cases.
🧪 The leakage laboratory
Each bar is an average over many simulated studies. With a leak-free pipeline the two bars match, and the test score can be trusted. Switch on a leak and the reported bar grows while the true bar (performance on new data) barely moves. The gap between them measures how far the leak overstates the model.
Each leak you add raises the reported bar while the true bar barely moves, and with all three on, the reported accuracy is far above the true one. Leaks like these let a model score 0.95 during development and 0.60 once it is used on new cases.
How to prevent leakage
One rule prevents all three leaks: every step that learns anything from the data must happen inside the training split, never before it. In practice:
- Split first, then preprocess. Fit your scaler and your feature selection on the training data only, then apply them to the test data. In scikit-learn, wrap the whole thing in a
Pipelineso cross-validation re-fits every step inside each fold. - Split by subject, not by row. If one person contributes several rows, keep all of that person's rows on one side (a "grouped" split). Split on the same unit you will make predictions for in practice.
- Touch the test set once. Do all comparing and tuning with cross-validation on the training data; unlock the test set only to report the final, single number.
Follow these rules and the test score is an unbiased estimate of how the model will do on new cases.
Your test score is an estimate too
Even a leak-free test score has sampling error. Test accuracy is a proportion measured on a sample, so it has a standard error like any other proportion, √(p(1 − p) ÷ n), and you should give it a confidence interval. Score 86% on 200 test cases and the 95% interval around it runs from about .81 to .90 (a Wilson interval, as R's prop.test computes it). Score the same 86% on 50 test cases and the interval runs from .74 to .93, wide enough to cover "clearly useful" and "barely worth deploying" at once. Getting that interval down to ±5 points takes roughly 200 test cases; ±3 points takes about 500.
So report the interval alongside the number, because a single accuracy hides how much of it is chance. And on a modest test set, treat small gaps between models as noise until shown otherwise: two models a point apart on 200 cases are indistinguishable. To compare two models, score them on the same cases and analyze the pairing. McNemar's test does this using only the cases where the two models disagree. Comparing two separate accuracies ignores that the models get many of the same cases wrong.
Why it matters: a leaked evaluation overstates the model, hides overfitting and can make you choose the wrong model, and its score drops as soon as the model meets new data. Only a test set left untouched until the end gives you a score you can base a decision on.
Common questions
What is data leakage?
Data leakage is any way that information from your test set (or from the future) leaks into training, so the model is graded partly on information it already had. The symptom is a high reported score and disappointing real-world performance. The three usual causes are: scaling or selecting 'the most predictive' features using the whole dataset before splitting; the same subject's rows appearing in both the training and test sets; and comparing many models or settings on the test set, then reporting whichever did best. The fix is a single rule: every step that learns anything from the data must happen inside the training split, never before it.
Do I still need a separate test set if I use cross-validation?
Yes, whenever you use cross-validation to make choices. Cross-validation on the training data is the right tool for comparing models and tuning settings. But once you use a CV score to choose something, that score becomes slightly optimistic, because you picked the option that scored best on it. A final test set, used once after every decision is made, gives an unbiased estimate for the model you chose. On small datasets people sometimes use nested cross-validation instead of a held-out set. The principle is the same: the number you report should come from data that influenced none of your decisions.
My test accuracy came out higher than my training accuracy. Is something wrong?
Usually not. The most common explanation is sample size. A test score is a proportion measured on a sample, so on 50 test cases the 95% interval around 86% runs from roughly .74 to .93. That is wide enough to include a training score several points lower. There are three other common causes. A model trained with regularization or dropout is handicapped during training and evaluated without that handicap. A lucky split can put the easier cases in the test set. And a cross-validated training score averages over models that were each fitted on less data than the final one. The gap is a warning sign when it is large and remains after refitting on several different random splits. Then look for leakage, most often duplicate rows on both sides of the split, which let the model recognize cases it has already seen.