Classification Metrics & the Accuracy Trap
When your outcome is a category (sick vs healthy, spam vs not, will-churn vs won't), "how good is the model?" has more than one answer. Reaching for plain accuracy is the single most common beginner mistake, because accuracy can look brilliant while the model fails, unnoticed, at the only job you cared about. Let's build the vocabulary that keeps you honest.
Every prediction lands in one of four boxes
A classifier takes a case and outputs a label: positive (has the condition, is spam, will churn) or negative. Compare that guess to the truth and there are exactly four outcomes, laid out in a confusion matrix:
- True positive (TP): flagged, and really positive. A hit.
- False positive (FP): flagged, but actually negative. A false alarm (Type I error).
- False negative (FN): missed, but actually positive. A miss (Type II error).
- True negative (TN): correctly left alone.
Every metric you'll ever meet is just some ratio of these four counts. Learn the boxes and the rest is arithmetic.
The metrics, and the question each one answers
- Accuracy = (TP + TN) / everything. "What fraction did I get right?" Intuitive, and the first thing to distrust.
- Recall (a.k.a. sensitivity, true-positive rate) = TP / (TP + FN). "Of the real positives, how many did I catch?" The metric a cancer screen lives or dies by.
- Precision = TP / (TP + FP). "Of everything I flagged, how much was right?" The metric a spam filter lives or dies by.
- Specificity (true-negative rate) = TN / (TN + FP). "Of the real negatives, how many did I correctly clear?"
- F1 = the harmonic mean of precision and recall, high only when both are high, so it resists the trick accuracy falls for.
Clinical papers give two of these a second set of names, which is worth knowing before you read one: precision is the positive predictive value (PPV), and its mirror image TN / (TN + FN) is the negative predictive value (NPV), the chance that a case you cleared really was negative. Sensitivity and specificity describe the test; PPV and NPV describe what a result means for the person holding it, which is why the two pairs answer such different questions in Bayes' rule problems.
Where accuracy lies: base rates
The whole problem fits in one sentence. If 5% of patients truly have a disease, a model that shouts "healthy!" at every single person is 95% accurate, and catches exactly zero sick patients. Its recall is 0%. Accuracy rewarded it for parroting the majority class; the metric you actually needed (recall) exposes it instantly. This is the accuracy trap, and it's a direct consequence of base rates. Watch it happen.
🎯 Threshold, base rate, and the four boxes
Two score distributions: healthy people and people who have the condition. The model calls everything to the right of the threshold "positive." Drag the threshold (or use the slider), pull the two groups apart, and (the key move) crank prevalence down to a rare disease and watch accuracy stay smug while recall collapses.
Two lessons fall straight out of the picture. First, the threshold is a dial, not a law of nature. Sliding it right buys precision at the cost of recall (fewer flags, but surer ones); sliding it left does the reverse. There is no universally "correct" threshold, only the one that matches which error hurts more. Second, a single number is never the whole story. Accuracy, precision, and recall can each be made to look great in isolation; you have to read them together, against the base rate, to know what the model really does.
Choosing the metric by which error hurts more
The right metric is a values question disguised as a statistics question. Ask: which mistake is worse here?
- A miss is catastrophic → chase recall. Cancer screening, fraud detection, structural safety checks. Better a few false alarms than one missed case, so you accept lower precision to push recall up.
- A false alarm is expensive → chase precision. Spam filtering, flagging accounts for suspension, recommending a costly intervention. A wrongly-deleted email or a wrongly-punished user is the error you most want to avoid.
- Both matter and the class is rare → F1 (or a cost-weighted score) gives a single figure that can't be gamed by ignoring the minority class.
Notice what accuracy never told you: which kind of mistake the model is making. Two models with identical 90% accuracy can have wildly different precision–recall profiles: one cautious, one trigger-happy. The confusion matrix shows you the difference; accuracy hides it. (Psychology has run on this same table since the 1950s: in signal detection theory the four cells are hits, misses, false alarms, and correct rejections, with a human as the classifier.)
Two single numbers that survive a rare class
When a reviewer wants one number anyway, two of them do the job accuracy cannot. Balanced accuracy is the plain average of recall and specificity, so each class contributes half the score however rare it is. Matthews correlation coefficient (MCC) is the Pearson correlation between the predicted labels and the true ones, coded 0 and 1: it runs from −1 to +1, sits at 0 for chance, and climbs toward 1 only when all four cells of the matrix are in good shape.
Run the lesson's own example through both. Of 1,000 patients, 50 have the condition. The model that calls everyone healthy scores 95% accuracy, 50% balanced accuracy, MCC 0. A real model that catches 30 of the 50 and raises 50 false alarms scores 93% accuracy, 77.4% balanced accuracy, MCC .44. Accuracy ranks the useless model above the useful one; the other two put it back where it belongs.
One footnote you will meet in software: MCC's denominator is zero whenever a model predicts a single class, as the parrot does, so R and Python hand back 0 by convention rather than by calculation. It is the right verdict on a model that has learned nothing, but it is a convention.
Why it matters: before you report "my classifier is 94% accurate," ask what the base rate is and which error is worse. On a rare outcome, accuracy is almost meaningless; a coin that always says "no" can beat it. Report the confusion matrix and the metric that matches the cost of being wrong, and treat the threshold as a decision you own. Next we sweep that threshold across every value at once to get the ROC curve and AUC.
Problem 31 of the practice problems builds the confusion matrix for a 95%-sensitive screening test on a condition affecting 8 people per 1,000, and finds where the false positives all came from.
Common questions
Why is accuracy misleading for rare outcomes?
Because a lazy model can score high just by ignoring the rare class. If 1 in 100 patients has a disease, a model that labels everyone 'healthy' is 99% accurate while catching zero cases — useless, yet it beats most honest models on accuracy alone. Accuracy weights every case equally, so when 99% of cases are one class, that class dominates the score and the rare class barely registers. For imbalanced problems, report recall (did we catch the positives?) and precision (were our positive calls right?) instead of, or alongside, accuracy.
My classes are very imbalanced. Should I oversample the rare class?
Reach for it last, not first. Oversampling the minority class (or SMOTE, which manufactures synthetic minority cases in between real ones) changes the base rate the model trains on, so the probabilities it returns stop describing your actual population: a model trained on a rebalanced 50/50 sample overstates everybody's risk. It tends to improve ranking a little, and sometimes not at all. Two cheaper moves come first. Leave the data alone and move the decision threshold, which costs nothing and stays entirely under your control. Or weight the classes inside the loss function (class_weight="balanced" in scikit-learn, a weights argument in most R fitting functions) so that missing a positive case costs the model more than missing a negative one. Whatever you do to the training set, never resample the test set, which exists to look like the population you will deploy on.
When should I use F1 instead of accuracy?
Use F1 when the positive class is rare and both kinds of error matter. F1 is the harmonic mean of precision and recall, so it is high only when both are high — it cannot be rescued by ignoring the rare class the way accuracy can. It also ignores the true negatives entirely, which is exactly what you want when true negatives are abundant and uninformative. If the two error types have clearly different costs, go further and pick the metric that matches the costlier error (recall for screening, precision for spam), or set the threshold to balance them deliberately rather than trusting any single summary number.