Train/Test Splits & Generalization
A predictive model has exactly one job: to work on data it has never seen. So there is one golden rule, and breaking it is the most common way careful people fool themselves in machine learning — never judge a model on data it was trained on. The moment you do, you're measuring memory, not learning.
The golden rule and the split
A flexible model can memorize its training data almost perfectly and still be useless on the next case — that's overfitting. To get an honest estimate of how well a model generalizes, you hold data back:
- Training set: the model learns here (fits coefficients, grows trees, picks neighbors).
- Test set: locked in a drawer until the very end, opened once to report honest performance. If you peek and adjust, it's no longer a test set.
A typical split is 70–80% train, the rest test. For classification, stratify the split so each side keeps the same class proportions — otherwise a rare class can vanish from one side by chance. And when you need to compare or tune models, carve a validation set (or use cross-validation) out of the training data, so the test set stays pristine.
Data leakage: the #1 silent killer
Data leakage is when information from the test set sneaks into training. It doesn't crash anything or throw a warning; it just makes your reported score gorgeous and your real-world performance disappointing. Three classic leaks account for most disasters:
- Preprocessing before the split. Scaling, or choosing "the most predictive features", using the whole dataset lets the test rows influence those choices. The test set helped build the model, so it can't fairly grade it.
- The same subjects on both sides. Duplicate rows, or multiple measurements from one person landing in both train and test, let the model recognize individuals it "already met" — inflating the score without any real skill.
- Tuning on the test set. Trying many models / thresholds / settings and reporting the one that scored best on the test set turns your test set into a training set for your choices. You've optimized against the exam.
Below is a real (if tiny) predictive pipeline: a k-nearest-neighbors classifier on data where only a couple of features carry weak signal and the rest are noise. Flip the leaks on and watch the pipeline's reported accuracy pull away from its true accuracy on fresh cases.
🧪 The leakage laboratory
Each bar is an average over many simulated studies. With an honest pipeline the two bars match — the test score is trustworthy. Switch on a leak and the reported bar inflates while the true bar (performance on never-seen data) barely moves. That gap is self-deception, measured.
With every leak off, the reported and true bars sit on top of each other — that's what an honest evaluation looks like. Each leak you add lifts the reported bar while the true bar stays put, and stacking all three produces a triumphant number that is almost entirely fiction. This is exactly how a model posts 0.95 in the notebook and 0.60 in production.
How to split honestly
The leak-proof principle is one sentence: every step that learns anything from the data must happen inside the training split, never before it. In practice:
- Split first, then preprocess. Fit your scaler and your feature selection on the training data only, then apply them to the test data. In scikit-learn, wrap the whole thing in a
Pipelineso cross-validation re-fits every step inside each fold. - Split by subject, not by row. If one person contributes several rows, keep that whole person on one side (a "grouped" split). The unit you'll predict on in the wild is the unit you should split on.
- Touch the test set once. Do all comparing and tuning with cross-validation on the training data; unlock the test set only to report the final, single number.
Do this and the interactive's two bars stay glued together, which is the entire idea. An honest, modest number beats an inflated, dishonest one every time, because only the honest one survives contact with reality.
Your test score is an estimate too
One honesty problem survives even a leak-free pipeline. Test accuracy is a proportion measured on a sample, so it carries a standard error like any other proportion, √(p(1 − p) ÷ n), and deserves a confidence interval. Score 86% on 200 test cases and the 95% interval around it runs from about .81 to .90 (a Wilson interval, which is what R's prop.test hands back). Score the same 86% on 50 test cases and the interval runs from .74 to .93, wide enough to cover "clearly useful" and "barely worth deploying" at once. Getting that interval down to ±5 points takes roughly 200 test cases; ±3 points takes about 500.
Two habits follow. Report the interval alongside the number, because a lone accuracy hides how much of it is luck of the draw. And treat small gaps between models on a modest test set as noise until shown otherwise: two models a point apart on 200 cases are indistinguishable. When you genuinely need to compare two models, compare them on the same cases and analyze the pairing, with McNemar's test on the cases where the two disagree, rather than lining up two accuracies that share most of their mistakes.
Why it matters: a leaked evaluation doesn't just overstate a model; it recommends the wrong one, hides overfitting, and collapses the moment it meets real data. Keeping the test set genuinely untouched is the difference between a number you can stake a decision on and a number that only ever described your own dataset.
Common questions
What is data leakage?
Data leakage is any way that information from your test set (or from the future) sneaks into training, so the model is secretly graded on things it already saw. The symptom is a beautiful reported score and disappointing real-world performance. The three classic culprits are: scaling or selecting 'the most predictive' features using the whole dataset before splitting; the same subject's rows appearing in both the training and test sets; and trying many models or settings and reporting the one that scored best on the test set. The cure is a single rule: every step that learns anything from the data must happen inside the training split, never before it.
Do I still need a separate test set if I use cross-validation?
Ideally yes, whenever you make choices. Cross-validation on the training data is the right tool for comparing models and tuning settings — but the moment you use a CV score to choose something, that score becomes slightly optimistic, because you have optimized against it. A final test set, opened once after every decision is locked, gives an unbiased estimate of the model you actually settled on. On small datasets people sometimes use nested cross-validation instead of a held-out set, but the principle is unchanged: the number you report should come from data that influenced none of your decisions.
My test accuracy came out higher than my training accuracy. Is something wrong?
Usually not, and the most ordinary explanation is sample size. A test score is a proportion measured on a sample, so on 50 test cases the 95% interval around 86% runs from roughly .74 to .93, which comfortably covers a training score several points lower. Three other routine causes: a model trained with regularization or dropout is handicapped during training and evaluated without that handicap; a lucky split can hand the test set the easier cases; and a cross-validated training score averages over models each fitted on less data than the final one. It becomes a warning sign when the gap is large and survives refitting on several different random splits. At that point look for leakage, with duplicate rows landing on both sides being the usual culprit, because the model is recognizing cases it has already met.