8. Ranking Every Threshold at Once

How ROC curves and AUC grade a ranking

In the last chapter you judged your logistic-regression model at one threshold. You picked a cut, every image landed on one side of it, and you counted the results in a confusion matrix. That gave you precision, recall and F1. They were all honest numbers. But they describe one decision, and the model can make many.

The model does not hand you a decision. It hands you scores. For every image it gives a probability for each of the ten digits. If you line up all the images by how sure the model is that each one is a seven, you get a ranking, from most seven-like down to least. A threshold is a single cut through that ranking. Everything above the cut gets called a seven, and everything below does not. Move the cut and you get a different confusion matrix, but the ranking underneath stays the same.

This chapter is about measuring that ranking directly. The tools are the ROC curve and the number that summarizes it, AUC. ROC stands for receiver operating characteristic, a name left over from radar work, and AUC stands for area under the curve. By the end you will be able to build one by hand, compute it in scikit-learn on your MNIST validation predictions, and say when it tells you what you need and when a different curve, the precision-recall curve, tells you more. This is the next step in the first-models phase, right after the confusion matrix and thresholds.

The setup is the same as last time. You fit the scaled logistic-regression pipeline on the first fifty thousand training images and predict on the last ten thousand training images, your validation split. The test set stays shut.

Turning ten digits into one question

ROC curves work on yes-or-no questions. MNIST has ten answers, so first you shrink the problem. Pick one digit. Say it is seven. Every real seven is a positive. Every other image, whether it is a zero, a three or a nine, is a negative. This is called one-vs-rest: one class against all the rest.

The score for each image comes from predict_proba. Call predict_proba on the validation images and you get a table with ten thousand rows and ten columns. Each row adds up to one. The column for "7" holds the model's probability that each image is a seven. That single column is your ranking.

Notice what this does to the balance. Each digit is about a tenth of MNIST, so in a one-vs-rest view you have about one positive for every nine negatives. Keep that ratio in mind. It matters a great deal later.

Building a curve from eight images

Before touching ten thousand images, work with eight. Say the model scored them like this, sorted from highest to lowest probability of being a seven. A seven at 0.95. A seven at 0.85. A non-seven at 0.70. A seven at 0.60. Then four non-sevens at 0.45, 0.30, 0.20 and 0.10. That is three positives and five negatives.

You need two rates. The first is the true positive rate. It is the share of real positives that land above the cut. You already know it by another name: it is recall. The second is the false positive rate. It is the share of real negatives that land above the cut. In other words, false positives divided by all negatives. With three positives here, the true positive rate moves in steps of a third. With five negatives, the false positive rate moves in steps of a fifth.

Now sweep the threshold from high to low, letting in one image at a time.

Start with the cut above every score. Nothing is called a seven. No positives found, no negatives wrongly flagged. Both rates are zero. That is your first point, at false positive rate zero, true positive rate zero.

Lower the cut to let in the 0.95 image. It is a real seven. You have found one of three positives, so the true positive rate is one third. No negatives have slipped in, so the false positive rate is still zero.

Let in the 0.85 image. Another seven. True positive rate two thirds, false positive rate zero.

Let in the 0.70 image. This one is not a seven. The true positive rate stays at two thirds, but now one of the five negatives is flagged. The false positive rate becomes one fifth, which is 0.2.

Let in the 0.60 image. A seven. Now all three positives are found, so the true positive rate is one. The false positive rate stays at 0.2.

The last four images are all negatives. Each one you let in pushes the false positive rate up by a fifth: 0.4, then 0.6, then 0.8, then 1. The true positive rate stays at one the whole way.

Plot these points with the false positive rate across the bottom and the true positive rate up the side. The path starts in the bottom left corner. It climbs straight up to two thirds. It steps right to 0.2. It climbs again to the top. Then it runs flat along the top to the right-hand corner. That staircase is the ROC curve.

Read its shape. Each step up means the next image in the ranking was a positive. Each step right means it was a negative. A model that ranked every positive above every negative would go straight up the left edge to the top, then straight across. That is the perfect curve, hugging the top left corner. A model that ranks at random lets in positives and negatives mixed together, so its steps up and right come in roughly equal proportion, and the curve wanders along the diagonal from bottom left to top right.

Our eight images made one mistake in the ranking. A negative at 0.70 sat above a positive at 0.60. That one mistake is the single step to the right before the curve reaches the top.

The area, two ways

AUC is the area under that staircase. The whole square has area one, because both rates run from zero to one.

Break the area into two rectangles. From false positive rate zero to 0.2, the curve sits at height two thirds. That rectangle is 0.2 wide and two thirds tall, so its area is about 0.133. From 0.2 to 1, the curve sits at height one. That rectangle is 0.8 wide and one tall, so its area is 0.8. Add them and you get about 0.933. As a fraction, that is fourteen fifteenths.

Now look at the same eight images in a completely different way. Make every possible pairing of one positive with one negative. Three positives times five negatives gives fifteen pairs. For each pair, ask one question: did the model score the positive higher than the negative?

The seven at 0.95 beats all five negatives. That is five correct pairs. The seven at 0.85 also beats all five. Five more. The seven at 0.60 beats the four negatives below it but loses to the negative at 0.70. That is four correct and one wrong. Total: fourteen correct pairs out of fifteen.

Fourteen fifteenths again. This is not a coincidence. The area under the ROC curve always equals the share of positive-negative pairs that the model puts in the right order. Each step right in the staircase is a negative entering the cut, and the height at that step counts how many positives already sit above it. Adding up those heights, each a fifth wide, is the same as counting correct pairs.

This second reading is the one to remember. AUC answers a plain question: if you pick one real seven and one non-seven at random, how often does the model score the seven higher? An AUC of 0.5 means the answer is "half the time", which is a coin flip. An AUC of 1.0 means "every time", a perfect ranking. Our toy model, at 0.933, gets the order right about fourteen times in fifteen.

Two things are missing from this number, and both are on purpose. First, no threshold appears anywhere in it. The pair count never asks where you would cut. Second, the exact score values do not matter, only their order. If you replaced 0.95, 0.85 and the rest with 9, 8, 7 and so on down, every pair would come out the same, and so would the AUC. That makes AUC a clean measure of ranking. It also means AUC says nothing about whether your scores mean what they claim. A model that says 0.95 when it is right only six times in ten can still have a perfect AUC, as long as it puts things in the right order.

The same curve on MNIST

Now do it for real on your validation predictions. Everything below lives in scikit-learn's metrics module.

Start by getting the probability table. With your fitted pipeline, call predict_proba on the validation pixels. Call the result proba.

Next, find the right column. Your MNIST labels are strings, like "7", not numbers. The pipeline keeps its class order in an attribute called classes_. So find the position of "7" in that list, by turning classes_ into a Python list and asking for the index of "7". In MNIST the classes sort as "0" through "9", so the answer will be seven, but asking the model is safer than assuming.

Then build the yes-or-no truth. Compare your validation labels to the string "7". That gives you an array of True and False, one per image, with True for each real seven.

Now call roc_curve with two arguments: that True-and-False array, and the seven column from proba, which you get by taking all rows and just that one column. roc_curve only handles one yes-or-no question at a time. It wants a one-dimensional set of scores, not the whole ten-column table. If you hand it the multiclass labels directly, it raises an error. That is why you built the True-and-False version first. If you would rather pass the string labels, you can instead set pos_label to "7", which tells the function which label counts as positive.

roc_curve gives back three arrays: false positive rates, true positive rates, and the thresholds that produced each pair. That is the eight-image sweep you just did, run over ten thousand images. Each threshold is a cut, and each pair of rates is a point on the staircase. With this many images the staircase has so many small steps that it looks like a smooth curve.

To get the number, call roc_auc_score with the same two arguments. To see the picture, scikit-learn has RocCurveDisplay. Its from_predictions method takes the same truth and scores, computes the rates and the area, and draws the curve with the AUC in the legend. There is also a from_estimator method that takes the pipeline, the validation pixels and the labels and pulls out the probabilities itself. For a ten-class model, though, you want to control which column is plotted, so from_predictions on one column at a time is clearer. To compare several digits on one chart, call it once per digit and point each call at the same plotting axes.

Then do all ten digits at once. roc_auc_score can do the one-vs-rest split for you. Pass it the full validation labels and the full ten-column proba, and set multi_class to "ovr". Also pass labels set to the pipeline's classes_, so the function knows which column belongs to which digit. For this mode it insists on probabilities whose rows add up to one, which is what predict_proba gives. It then computes ten one-vs-rest AUCs and combines them. The default combination is average set to "macro", a plain unweighted mean of the ten. If you set average to None instead, you get all ten AUCs back as an array, which is the more useful view when you want to see which digit is weakest.

Expect your numbers to be high. A model like this one usually lands near 0.99 for macro AUC, with individual digits somewhere around there too. Your exact values depend on your run, so read your own.

That may look strange next to an accuracy of about 92%. How can a model that misses roughly one image in thirteen rank pairs correctly about ninety-nine times in a hundred? There are two reasons.

The first is that ranking is easier than cutting. Accuracy asks for the single right digit, which means the correct class must win against all nine others on that image. AUC for seven asks something gentler: is this seven scored more seven-like than that random non-seven? A seven the model mistook for a nine may still have a seven score well above most zeros, ones and sixes. It loses the accuracy point but wins most of its pairs.

The second is the one-to-nine balance. Most of the negatives for "is it a seven?" are easy. A zero looks nothing like a seven, and the model gives it a seven score near zero. Those easy negatives pile up at the bottom of the ranking, and they make up most of the pairs. The hard cases, the nines and ones that look a bit like sevens, are a small slice of the negatives. So even many hard mistakes barely dent the false positive rate, because it divides by all negatives, easy ones included.

That second reason is where AUC can mislead you.

When the ROC curve looks better than the truth

The false positive rate is false positives divided by all negatives. When negatives vastly outnumber positives, that bottom number is huge, and a lot of false alarms can hide in it.

Work it through on your validation split. Say about a thousand of your ten thousand images are sevens, leaving about nine thousand negatives. A false positive rate of one percent sounds tiny. But one percent of nine thousand is ninety false alarms. If the model finds, say, nine hundred and fifty real sevens at that same cut, then of everything it called a seven, about nine in ten were right. That is acceptable.

Now take a far rarer case, the kind you meet in fraud or rare-disease screening. Say one item in a thousand is positive, and you have a million items. That is a thousand positives and nine hundred and ninety-nine thousand negatives. The same one percent false positive rate now means almost ten thousand false alarms. Even if you catch every one of the thousand real positives, more than nine in ten of your alerts are wrong. The ROC curve at that point looks nearly identical to the MNIST case. The experience of the person reading the alerts is completely different.

The precision-recall curve exposes this. It plots precision against recall, both of which you met in the last chapter. Precision is true positives divided by everything flagged. Recall is true positives divided by all real positives. Neither one uses the true negatives at all. So the pile of easy negatives cannot dilute anything. Every false alarm comes straight out of precision.

In scikit-learn, precision_recall_curve works like roc_curve. Give it the True-and-False array for one digit and that digit's score column. It returns precisions, recalls and thresholds. Like roc_curve, it handles one yes-or-no question at a time.

The single-number summary is average_precision_score, often called AP. Walk down the ranking. Each time recall goes up, take the size of that rise and multiply it by the precision at that point. Add those pieces up. That is AP. It is a weighted average of precision, weighted by how much recall each step adds. It is computed this way, rather than by drawing straight lines between points and measuring the area, because straight-line shortcuts on a precision-recall curve tend to flatter the model. Unlike roc_curve, average_precision_score will also take the full ten-column proba and the labels and do one-vs-rest for you.

The two summaries also have different floors. For ROC, random guessing always scores 0.5, whatever the balance. For precision-recall, random guessing scores the share of positives. If a tenth of images are sevens, a random ranking has precision about 0.1 everywhere, so its AP is about 0.1. In the one-in-a-thousand case, the random floor drops to 0.001. That moving floor is the point. It reminds you how rare your positives are, which ROC keeps hidden. When you draw the curve with PrecisionRecallDisplay, you can turn on its chance line with plot_chance_level set to True, and it draws that floor for you.

Try it on a weak digit. Compute per-class F1 on your validation predictions as the last chapter showed, or use your numbers if you already ran it, and pick the lowest. Say yours is eight. Draw its ROC curve and its precision-recall curve side by side. The ROC curve will probably still bulge toward the top left, with an AUC near the high nineties. The precision-recall curve will likely sag more as recall approaches one. That sag is the cost of catching the last few hard eights: to reach them you lower the cut, and threes, fives and nines start flooding in above it. ROC shows that flood as a small move to the right. Precision-recall shows it as a real drop.

So which one should you use? Use ROC and AUC when you care about both classes, and the balance between them is stable. It is a fair, balance-free comparison of how well two models rank. Use precision-recall and AP when the positives are rare and each false alarm costs something real, like a person's time spent checking it. On MNIST one-vs-rest, at one to nine, both curves are informative and they mostly agree on which digits are hard. The gap between them grows as the positives get rarer.

One note on the macro average. Averaging ten one-vs-rest AUCs feels balanced, but each digit's "rest" is a different mix of digits. So even the macro number is shaped by what the negatives look like for each class. Look at the ten individual numbers, not just their mean.

What a high AUC does not promise

Here is a misunderstanding worth checking in yourself. Suppose your seven has an AUC of 0.99. Does that mean the model has a good threshold for sevens? Does it mean that when it says 0.8, the image is a seven about eight times in ten?

It means neither.

Go back to the eight-image example. Take every score and square it. The 0.95 becomes about 0.90, the 0.60 becomes 0.36, the 0.10 becomes 0.01. The order does not change, so every pair comes out the same and the AUC stays at fourteen fifteenths. But the probabilities now say something different. If you had a threshold at 0.5, it would now let in only the top two images instead of four. The ranking is identical. The decision you make from it is not.

So AUC tells you that a good cut exists somewhere along the ranking. It does not tell you where it is. Finding the cut is the threshold work from the last chapter: choose it from the confusion matrix and the costs of each kind of mistake, on validation data.

And AUC tells you nothing about whether a score of 0.8 means eighty percent. That property is called calibration. A model can rank perfectly and still be overconfident or underconfident in every number it reports. If someone downstream will read your probabilities as probabilities, you need to check calibration separately. A high AUC is a statement about order, and only about order.

Your turn

Here is the exercise. It stays on the validation split. The test set stays shut.

Before you compute anything, write down a guess. For each of the ten digits, predict its one-vs-rest AUC and its average precision. Then predict which digit will rank lowest on each.

Then compute them. Get all ten AUCs from roc_auc_score with multi_class set to "ovr", average set to None, and labels set to the pipeline's classes_. Get all ten average precisions by building the True-and-False array for each digit and calling average_precision_score on that column. A loop over the ten class names does it.

Now make three rankings of the digits, from strongest to weakest: one by AUC, one by AP, and one by per-class F1, computed as the last chapter showed. Compare them. Where do they agree? Where does a digit sit high on AUC but lower on AP? For any digit where the rankings disagree, look at its precision-recall curve and say in a sentence what it shows that the ROC curve did not. Check your answers against the reasoning here: AUC should be uniformly high and compressed near the top, AP should spread the digits out more, and F1 reflects one particular cut, so it can disagree with both.

Your rankings also feed straight into the error analysis exercise set last time. The digits that sit low on AP are where the model's false alarms concentrate.

The model you have used through all of this is logistic regression, which draws straight boundaries between digits in pixel space. The next model makes no boundaries at all. It labels a new image by finding the training images most like it. That is nearest neighbors, and every tool from this chapter will be ready to grade it.

What changed in scikit-learn and pandas this month

Two items from mid-October touch the tools you are using.

The first is pandas 3.1.0, released on October twenty-second and available now on the Python Package Index. It is a regular minor release with performance improvements, bug fixes, deprecations and compatibility updates. Nothing in it changes how you have used pandas so far, which has been to inspect MNIST. The practical point is about your lock file. Your project, built with uv, pins whatever pandas version you locked. It does not move to 3.1.0 on its own, and it should not move mid-experiment. If you want to know what you have, run uv tree and read the pandas line. Upgrade between experiments, rerun your notebook from a fresh kernel, and check that your numbers match.

The second item is not released yet, and that distinction matters. scikit-learn's development branch, labeled 1.10 dev, has a set of changes aimed at a release candidate targeted for November. You do not have these unless you install the development version, and you should not, for a course project. But one change is a good lesson in what a random seed does and does not promise.

The histogram gradient boosting models, HistGradientBoostingClassifier and its regressor partner, can pick a random subset of features at each split when you set max_features below one. The development branch changed how that subset is chosen. The result is that the same model, with the same random_state, can now follow different decision paths than it did before. Same code, same seed, different trees.

Back in the workspace chapter you saw that a seed makes one run repeatable on one set of library versions, not across versions. This is that warning made concrete. Gradient boosting arrives later in this phase, so here is the check to keep. When a scikit-learn upgrade lands, rerun one saved experiment on the old lock file and the new one, with the same seed, and compare the validation metrics, not the exact predictions. Small shifts are expected. A large change in validation AUC or accuracy from a library upgrade alone is a sign your result depended on luck in the random choices, and that is worth knowing before you build a project on it.

The same development branch also changes how QuantileTransformer estimates quantiles and how partial dependence plots handle missing values. Neither touches anything you have built yet. Both are reminders that preprocessing and inspection tools can shift between versions too, which is another reason the version lives in your lock file and not in your memory.