ROC Curves & AUC

The last lesson showed that every setting of the decision threshold gives a different precision and recall. Which setting do you report? You don't have to pick one: you can show them all. Plotting every threshold at once gives the ROC curve, the most widely used one-picture summary of a classifier, and the number calculated from it, AUC.

Sweep the threshold, trace a curve

As you move the threshold from "flag almost nobody" to "flag almost everybody," two rates from the confusion matrix change together:

• True-positive rate (TPR = recall = sensitivity): the fraction of real positives you catch.
• False-positive rate (FPR = 1 − specificity): the fraction of real negatives you wrongly flag.

Plot TPR (vertical) against FPR (horizontal) at every threshold and you get the Receiver Operating Characteristic curve, a name that comes from World War II radar operators deciding which blips were real. Press Sweep and the widget draws the curve: as the threshold moves down, the point travels from the bottom-left corner (flag nobody: 0, 0) to the top-right (flag everybody: 1, 1).

Reading the curve

The shape of the curve shows how good the classifier is:

• Top-left corner is perfection: catch every positive (TPR = 1) with no false alarms (FPR = 0). The closer the curve comes to that corner, the better.
• The diagonal is a coin flip. A curve on the 45° line means TPR = FPR at every threshold, so the model gains true positives only by accepting an equal rate of false ones. It has learned nothing.
• Below the diagonal means worse than chance, usually a sign that a label has been flipped somewhere (reverse the predictions of a model that is reliably wrong and you get a good one).

Because the curve covers every threshold, you can compare two models before you know which threshold you will use.

The curve also gives a default answer to the question the last lesson left open, which threshold to use. Youden's J picks the point furthest above the diagonal, the threshold that maximizes TPR − FPR, and it is what most software means by "the optimal cutpoint". For two normal groups with equal spread, that threshold is always the midpoint between the two means. At d = 1 the midpoint is a threshold of 0.50, giving TPR .69, FPR .31 and J = .38. Those are the widget's starting values, so you can read them off the plot. Treat J as a starting point, not a final answer: it counts a miss and a false alarm as equally costly, and the previous lesson showed that weighing those costs is your decision.

AUC: one number, one clean meaning

The Area Under the Curve reduces the ROC to a single number between 0 and 1. It has a concrete interpretation: AUC is the probability that the model gives a randomly chosen positive case a higher score than a randomly chosen negative one. An AUC of 0.5 means the model orders a random positive above a random negative only half the time, the rate you would get by guessing. An AUC of 0.90 means it ranks them correctly 90% of the time.

With two normal groups there is a formula. If the groups are separated by d standard deviations, AUC = Φ(d/√2). The same identity appears in signal detection theory, where the separation is called d′. So d = 1 gives AUC ≈ 0.76, d = 2 gives ≈ 0.92, and d = 0 gives exactly 0.50. Check them against the widget's readout. Roughly, 0.7 counts as modest, 0.8 as good and 0.9+ as excellent, but always judge an AUC against how hard the problem is.

Rare positives: use a precision–recall curve

ROC has a weakness when negatives are plentiful. Its x-axis, the false-positive rate, divides the false positives by the number of negatives. On a heavily imbalanced problem (say 1 positive per 1,000) you can have thousands of false positives and still a tiny FPR. The ROC curve then looks excellent while most of your positive flags are wrong. A precision–recall curve (precision against recall, summarized by average precision) responds strongly to those false positives, because precision counts only the cases you flagged. As a rule of thumb, ROC is fine when the classes are roughly balanced. Add the PR curve when the positive class is rare and is the one you care about.

Ranking well is not the same as being right

AUC uses only the order of the scores. Compress them, stretch them or halve them all, and as long as the order stays the same, AUC does not change. So a model can sort every case correctly and still report probabilities that are badly wrong. Take 100 patients, 20 of whom develop the outcome. Model A gives each of those 20 a score of 0.55 and everyone else 0.45, which sorts them perfectly: AUC 1.00. Model B says "20%" to all 100 and sorts nothing: AUC 0.50. Now score both with the Brier score (the mean squared distance between the predicted probability and what actually happened, where 0 is perfect). Model A gets .2025 and Model B gets .160, so on the Brier score the model that sorts nothing is better.

Model A lacks calibration: the property that among the cases you called 20%, close to one in five do develop the outcome. Calibration matters most when a number is given to a person, because "your risk is 8%" is a claim about frequency, not about rank. Recalibrate Model A to say 0.95 and 0.05 instead, and its Brier score falls to .0025 while its AUC stays exactly 1.00. The ordering did not change, only the probabilities.

For a write-up, check calibration with a calibration plot: predicted risk, grouped into bins, against the observed rate in each bin, with a 45° line for reference. And compare the Brier score with a benchmark. Predicting the base rate p̄ for every case scores p̄(1 − p̄), .160 in this example, and a model that cannot beat that number has added nothing to what you knew before you fitted it.

Problem 76 of the practice problems comes at this lesson from the psychophysics side. Given one participant’s hits and false alarms in a recognition-memory task, it asks for the area under their ROC curve, which turns out to follow from d′ alone.