Classification Metrics & the Accuracy Trap
When your outcome is a category (sick vs healthy, spam vs not, will-churn vs won't), "how good is the model?" has more than one answer. Plain accuracy is not enough, because it can be high while the model fails at the one task you cared about.
Four outcomes: the confusion matrix
A classifier takes a case and outputs a label: positive (has the condition, is spam, will churn) or negative. Compare that guess to the truth and there are exactly four outcomes, laid out in a confusion matrix:
- True positive (TP): flagged, and really positive. A hit.
- False positive (FP): flagged, but actually negative. A false alarm (Type I error).
- False negative (FN): missed, but actually positive. A miss (Type II error).
- True negative (TN): correctly left alone.
Each metric in the next section is a ratio of these four counts.
The metrics, and the question each one answers
- Accuracy = (TP + TN) / everything. "What fraction did I get right?" Easy to understand, and often misleading.
- Recall (a.k.a. sensitivity, true-positive rate) = TP / (TP + FN). "Of the real positives, how many did I catch?" The metric that matters most for a cancer screen.
- Precision = TP / (TP + FP). "Of everything I flagged, how much was right?" The metric that matters most for a spam filter.
- Specificity (true-negative rate) = TN / (TN + FP). "Of the real negatives, how many did I correctly clear?"
- F1 = the harmonic mean of precision and recall, high only when both are high, so ignoring the rare class cannot raise it.
Clinical papers use other names for two of these. Precision is the positive predictive value (PPV), and its counterpart TN / (TN + FN) is the negative predictive value (NPV), the chance that a case you cleared really was negative. Sensitivity and specificity describe the test. PPV and NPV describe what a result means for the person who receives it, and unlike sensitivity and specificity they change with the base rate, as Bayes' rule problems show.
Base rates and the accuracy trap
If 5% of patients have a disease, a model that labels every person "healthy" is 95% accurate and catches no sick patients at all. Its recall is 0%. Accuracy gave it a high score for always predicting the majority class, and recall, the metric you needed, shows that it is useless. This is the accuracy trap, and it follows directly from base rates.
🎯 Threshold, base rate, and the four boxes
Two score distributions: healthy people and people who have the condition. The model calls everything to the right of the threshold "positive." Drag the threshold (or use the slider) and change how far apart the two groups are. To see the accuracy trap, lower the prevalence to that of a rare disease and move the threshold right: accuracy stays high while recall falls.
The widget shows two things. First, the threshold is your choice. Moving it right raises precision and lowers recall (fewer flags, but more of them correct), and moving it left does the reverse. No threshold is correct for every problem. The right one depends on which error is worse. Second, no single number tells you enough. Accuracy, precision and recall can each look good on their own, so read them together, against the base rate, to know what the model does.
Choosing the metric by which error hurts more
To choose a metric, ask which mistake is worse in your problem. The answer depends on what each error costs in your setting.
Where a miss is very costly, aim for high recall. In cancer screening, fraud detection and structural safety checks, a few false alarms are better than one missed case, so you accept lower precision to push recall up. Where a false alarm is expensive, aim for high precision instead. In spam filtering, in flagging accounts for suspension and in recommending a costly intervention, a wrongly deleted email or a wrongly punished user is the error you most want to avoid. When both errors matter and the class is rare, F1 (or a cost-weighted score) gives a single figure that a model cannot raise by ignoring the minority class.
Accuracy also does not tell you which kind of mistake the model makes. Two models with the same 90% accuracy can have very different precision and recall: one may flag few cases, and the other many. The confusion matrix shows the difference. (Psychology has used the same table since the 1950s: in signal detection theory the four cells are hits, misses, false alarms and correct rejections, and the classifier is a person.)
Two single numbers that survive a rare class
If a reviewer wants a single number, two measures work where accuracy fails. Balanced accuracy is the plain average of recall and specificity, so each class contributes half the score however rare it is. Matthews correlation coefficient (MCC) is the Pearson correlation between the predicted labels and the true ones, coded 0 and 1. It runs from −1 to +1, is 0 for chance-level prediction, and approaches 1 only when both false positives and false negatives are rare.
Apply both to the example from earlier. Of 1,000 patients, 50 have the condition. The model that calls everyone healthy scores 95% accuracy, 50% balanced accuracy, MCC 0. A model that catches 30 of the 50 and raises 50 false alarms scores 93% accuracy, 77.4% balanced accuracy, MCC .44. Accuracy ranks the useless model above the useful one, and the other two measures rank them correctly.
One detail you will meet in software: MCC's denominator is zero whenever a model predicts a single class, as the all-healthy model does. The formula is then undefined, so R and Python report 0 by convention. Zero is a sensible score for a model that has learned nothing.
Why it matters: before you report "my classifier is 94% accurate," ask what the base rate is and which error is worse. On a rare outcome, accuracy says almost nothing, because a model that always says "no" can score higher than a useful one. Report the confusion matrix and the metric that matches the cost of being wrong, and treat the threshold as a decision you make and justify. The next lesson moves that threshold across every value at once to draw the ROC curve and AUC.
Problem 74 of the practice problems builds the confusion matrix for a 95%-sensitive screening test on a condition affecting 8 people per 1,000, and finds where the false positives all came from.
Common questions
Why is accuracy misleading for rare outcomes?
Because a model can score high just by ignoring the rare class. If 1 in 100 patients has a disease, a model that labels everyone 'healthy' is 99% accurate while catching zero cases. It is useless, yet on accuracy alone it beats most models that try to find the cases. Accuracy weights every case equally, so when 99% of cases are one class, that class dominates the score and the rare class barely counts. For imbalanced problems, report recall (did we catch the positives?) and precision (were our positive calls right?) instead of, or alongside, accuracy.
My classes are very imbalanced. Should I oversample the rare class?
Try it last. Oversampling the minority class (or SMOTE, which creates synthetic minority cases between real ones) changes the base rate the model trains on, so the probabilities it returns no longer describe your actual population. A model trained on a rebalanced 50/50 sample overstates everyone's risk. Oversampling tends to improve ranking a little, and sometimes not at all. Try two simpler options first. Leave the data alone and move the decision threshold, which you can do at any time without refitting the model. Or weight the classes inside the loss function (class_weight="balanced" in scikit-learn, a weights argument in most R fitting functions) so that missing a positive case adds more to the loss than missing a negative one. Whatever you do to the training set, never resample the test set. It should look like the population you will use the model on.
When should I use F1 instead of accuracy?
Use F1 when the positive class is rare and both kinds of error matter. F1 is the harmonic mean of precision and recall, so it is high only when both are high. A model cannot get a high F1 by ignoring the rare class, as it can with accuracy. F1 also ignores the true negatives entirely, which suits problems where true negatives are plentiful and tell you little. If the two error types have clearly different costs, pick the metric that matches the costlier error (recall for screening, precision for spam). Or set the threshold to balance them on purpose, rather than relying on one summary number.