ROC Curves & AUC

In the last lesson we saw that the decision threshold is a dial, and every setting gives different precision and recall. So which setting do you report? The honest answer: don't pick one — show them all. That single move produces the ROC curve, the most-used one-glance summary of a classifier, and the number distilled from it, AUC.

Sweep the threshold, trace a curve

Recall the two knobs from the confusion matrix. As you slide the threshold from "flag almost nobody" to "flag almost everybody," two rates move together:

• True-positive rate (TPR = recall = sensitivity): the fraction of real positives you catch.
• False-positive rate (FPR = 1 − specificity): the fraction of real negatives you wrongly flag.

Plot TPR (vertical) against FPR (horizontal) at every threshold and you get the Receiver Operating Characteristic curve — a name inherited from WWII radar operators deciding which blips were real. Press Sweep and watch it draw itself: as the threshold slides down, the point crawls from the bottom-left corner (flag nobody: 0, 0) up to the top-right (flag everybody: 1, 1).

Reading the curve

The shape tells you everything at a glance:

• Top-left corner is perfection: catch every positive (TPR = 1) with no false alarms (FPR = 0). The closer the curve hugs that corner, the better.
• The diagonal is a coin flip. A curve on the 45° line means TPR = FPR at every threshold, so the model gains true positives only by accepting an equal rate of false ones. It has learned nothing.
• Below the diagonal means worse than chance, usually a sign you've flipped a label somewhere (a model that's reliably wrong is a correctly-labeled model in disguise).

Because the whole curve is threshold-free, it lets you compare two models without first committing to an operating point — genuinely useful when you don't yet know what threshold you'll deploy.

The curve also offers a default answer to the question the last lesson left open. Youden's J is the point standing furthest above the diagonal, the threshold that maximizes TPR − FPR, and it is what most software means by "the optimal cutpoint". For two normal groups of equal spread the winner is always the midpoint between the two means, so at d = 1 it sits at a threshold of 0.50, giving TPR .69, FPR .31 and J = .38. Those are the slider's own starting values, so you can read them straight off the plot. Treat J as a starting point rather than a verdict: it prices a miss and a false alarm identically, and choosing between those costs is precisely what the previous lesson said you own.

AUC: one number, one clean meaning

The Area Under the Curve collapses the ROC into a single number between 0 and 1. Its beauty is a concrete interpretation: AUC is the probability that the model gives a randomly chosen positive case a higher score than a randomly chosen negative one. An AUC of 0.5 means the model orders a random positive above a random negative only half the time — pure chance. An AUC of 0.90 means it wins that ranking contest 90% of the time.

Our two-normal setup even has a closed form: with the groups separated by d standard deviations, AUC = Φ(d/√2), an identity borrowed from signal detection theory, where the same separation is called d′. So d = 1 gives AUC ≈ 0.76, d = 2 gives ≈ 0.92, and d = 0 sits exactly on 0.50; check the readout against the slider. A rough field guide: 0.7 modest, 0.8 good, 0.9+ excellent — but always judged against how hard the problem is.

When ROC flatters — reach for precision–recall

ROC has a blind spot: its x-axis, the false-positive rate, is forgiving when negatives are abundant. On a heavily imbalanced problem (say 1 positive per 1,000) you can rack up thousands of false positives and still show a tiny FPR, so the ROC curve looks gorgeous while your positive flags are mostly wrong. A precision–recall curve (precision vs recall, summarized by average precision) reacts sharply to those false positives because precision only counts the cases you flagged. Rule of thumb: roughly balanced classes → ROC is fine; rare positive class you care about → report the PR curve too.

Ranking well is not the same as being right

AUC reads only the order of the scores. Squeeze them, stretch them, halve them all: as long as the ranking survives, AUC does not move. So a model can sort every case correctly and still report probabilities that mean nothing. Picture 100 patients, 20 of whom develop the outcome. Model A gives each of those 20 a score of 0.55 and everyone else 0.45, sorting them flawlessly, AUC 1.00. Model B says "20%" to all 100, sorting nothing, AUC 0.50. Score both by the Brier score (the mean squared distance between the predicted probability and what actually happened, where 0 is perfect) and Model A gets .2025 while Model B gets .160. The model that ranks nobody beats the model that ranks everybody.

What Model A lacks is calibration: the property that among the cases you called 20%, close to one in five really does develop the outcome. It becomes the whole story the moment a number reaches a person, because "your risk is 8%" is a claim about frequency and not about rank. Recalibrate Model A to say 0.95 and 0.05 instead, and its Brier score falls to .0025 with AUC still exactly 1.00. Nothing about its ordering changed; only its honesty about magnitude did.

Two things to carry into a write-up. Check calibration with a calibration plot, predicted risk grouped into bins against the observed rate in each bin, with a 45° line for reference. And know the benchmark: predicting the base rate p̄ for every case scores p̄(1 − p̄), which is .160 in this example, so a model that cannot beat that number has added nothing at all to what you knew before it was fitted.

Problem 33 of the practice problems arrives at this lesson from the psychophysics side: given one participant’s hits and false alarms in a recognition-memory task, it asks for the area under their ROC curve, which turns out to follow from d′ alone.