7. Where the Ninety-Two Percent Goes Wrong

Your logistic-regression model gets about nine digits in ten right. That sounds good, and against a floor of about eleven percent, it is. But that one number says nothing about the tenth digit. Which images does the model get wrong? What does it call them instead? And does it matter which mistakes they are?

This chapter answers those questions. You will open up the model's errors with a confusion matrix. Then you will measure them with three numbers called precision, recall and F1. All of them grow from one idea. A single accuracy score hides which mistakes a model makes, and which mistakes matter depends on the problem you are solving.

Opening the model's mistakes

First, a rule about where these numbers come from. In the last chapter's exercise, you split the training images into two parts. You fit on the first fifty thousand and checked on the last ten thousand. That last part is your validation set. Everything in this chapter is computed on validation predictions. The test set stays shut.

The reason is the same one that made us score the test set only once. In this chapter you will look hard at mistakes. You will notice patterns, and soon you will want to change things because of them. If you did that looking on the test set, every change would be shaped by those ten thousand images. The final test score would then measure how well you fit those images, not how well the model handles new handwriting. Validation data exists so you have somewhere to look closely. So if you have not made the split yet, make it now. Slice the training images at row fifty thousand, fit the same scaled pipeline on the first part, and call predict on the second part. Call the true labels y val and the predictions y pred.

Before touching MNIST, start small enough to count with your fingers.

Say you have a tiny classifier that only knows three digits: one, seven and nine. You show it twelve images, four of each. Here is what it says.

Of the four real ones, it calls all four a one.

Of the four real sevens, it calls two a seven. It calls one a one, and one a nine.

Of the four real nines, it calls three a nine and one a seven.

Count the correct answers. That is four plus two plus three, which is nine out of twelve. Accuracy is seventy-five percent. Now look at how much that number throws away. It does not tell you that every one was found. It does not tell you that sevens were the weak spot. And it does not tell you that sevens and nines get mixed up both ways.

A confusion matrix keeps all of that. It is a grid of counts. Each row is a true label, meaning what the image really is. Each column is a predicted label, meaning what the model said. The number in a cell is how many images had that true label and got that prediction.

For the three-digit case, the grid has three rows and three columns, in the order one, seven, nine. Read it row by row. The row for true ones reads four, zero, zero. The row for true sevens reads one, two, one. The row for true nines reads zero, one, three.

Two ways of reading it matter.

The first is the diagonal. That is the line of cells running from top left to bottom right, where the true row and the predicted column are the same digit. Those cells hold the correct answers: four, two and three. Add them and divide by the total, and you get accuracy back. So accuracy is just the diagonal squeezed into one number.

The second is everything off the diagonal. Every one of those cells is a specific kind of mistake. The cell in the seven row and the nine column holds a one. It means one real seven was called a nine. The cell in the nine row and the seven column also holds a one. Those two cells mirror each other across the diagonal. When both are filled, the model confuses that pair in both directions. When only one side is filled, the confusion runs one way.

Notice that row order matters. Rows are truth and columns are guesses. If you flip them, every reading below comes out backwards. scikit-learn uses this same layout, so it is worth fixing in your head now.

Precision and recall from the grid

The grid holds two different questions about each digit. Take seven.

The first question looks down a column. Of everything I called a seven, how many really were sevens? This is precision. Go down the seven column and you find zero, two and one. So the model said seven three times. Two of those were right. Precision for seven is two out of three, about sixty-seven percent.

The second question looks along a row. Of the real sevens, how many did I find? This is recall. Go along the seven row and you find one, two and one. So there were four real sevens. The model found two. Recall for seven is two out of four, fifty percent.

A short way to remember it: precision is about the model's claims, and recall is about the truth. Precision asks whether you can trust a seven when the model says one. Recall asks whether a real seven will get noticed.

There is some vocabulary that goes with this. When we focus on one class, a correct seven is a true positive. A real seven the model missed is a false negative. Something else that got called a seven is a false positive. Precision is true positives divided by true positives plus false positives. Recall is true positives divided by true positives plus false negatives. Both have the same top. They differ only in which mistakes go underneath.

Do the other two digits the same way.

For one, the column reads four, one, zero. The model said one five times, and four were right, so precision is eighty percent. The row reads four, zero, zero. All four ones were found, so recall is a hundred percent. This model is eager to say one. It catches every real one, but it also pulls in a seven by mistake.

For nine, the column reads zero, one, three. The model said nine four times and was right three times, so precision is seventy-five percent. The row reads zero, one, three, so three of four real nines were found. Recall is also seventy-five percent.

You often want one number per class that respects both questions. That number is F1. It is the harmonic mean of precision and recall. In words, take two times precision times recall, then divide by precision plus recall.

For seven, two times two thirds times a half is two thirds. Precision plus recall is two thirds plus a half, which is seven sixths. Two thirds divided by seven sixths is four sevenths, about point five seven. For one, two times point eight times one is one point six. Divide by one point eight and you get about point eight nine. For nine, both numbers are point seven five, so F1 is point seven five as well. When the two are equal, the harmonic mean is just that value.

Why the harmonic mean and not a plain average? Picture a lazy model that says seven only once, on the one image it is surest about, and gets it right. Its precision is a hundred percent. But say there are fifty real sevens. Then its recall is one in fifty, two percent. A plain average of a hundred and two is fifty-one. That sounds like half a decent model, but it is useless. The harmonic mean gives two times one times point oh two, divided by one point oh two, which is about point oh four. The harmonic mean is pulled down hard by whichever number is smaller. So a model cannot get a good F1 by being excellent on one question and hopeless on the other. That is the behavior you want from a single score that stands for both.

The real matrix on MNIST

Now apply this to your ten digits. scikit-learn has a function in its metrics module called confusion matrix. You pass it the true labels first and the predictions second: confusion matrix of y val and y pred. It returns a ten-by-ten array of counts, with rows as truth and columns as predictions, the same layout you just used by hand. Your labels are text strings, the character "5" and not the number five. The function sorts them, so the rows come out in the order zero through nine. You can make that order explicit by passing a labels argument with the ten strings.

A ten-by-ten grid of raw numbers is hard to scan. For a picture, use ConfusionMatrixDisplay, in the same module. Its from predictions method takes the same two arguments and draws the grid as a heat map. Pass cmap equals the string "Blues" to get a blue color scale, where darker cells hold bigger counts, and the numbers are printed inside. Older tutorials use a function called plot confusion matrix. That was removed from scikit-learn, and ConfusionMatrixDisplay is its replacement. If you have the fitted model instead of saved predictions, from estimator takes the model, the validation images and the validation labels, and predicts for you.

One option makes the picture easier to read. Pass normalize equals the string "true". Each row is then divided by its own total, so every row adds to one. A cell now reads as a share. It says what fraction of the real fours got called nines. That fixes a real problem. Digits appear in slightly different numbers, so raw counts mix up how often a digit shows up with how often it gets confused. With rows normalized, the diagonal cell for each digit is exactly that digit's recall, because it is the share of real examples the model found. Normalize equals "pred" divides down the columns instead, and then the diagonal is each digit's precision. You can set values format to a short string such as dot two f to show two decimal places.

Here is how to read your grid. I won't tell you your numbers. Your split and your solver settings decide them, and you should read them yourself.

Start with the diagonal. With the Blues color scale, it should be dark all the way down, since this model gets roughly nine in ten right. Look for the lightest diagonal cell, or simply the smallest number printed on the diagonal. That digit has the lowest recall, and it is your weakest class.

Then scan the off-diagonal cells for the biggest values. Most will be near zero. A handful will stand out. For handwritten digits, a few pairs are worth checking. Four and nine can look alike when the top of a four closes. Three and five share a lower curve. Seven and nine both have a stroke at the top and a long tail. Two and eight can blur when a two's loop is drawn tight. Find each pair and read both mirror cells. Is it one-way or two-way? A four called a nine is not the same mistake as a nine called a four, and the grid keeps them apart.

This is already more than accuracy gave you. It is also more than the per-class correctness you computed last chapter. That told you how well each digit did. The grid tells you what each digit was mistaken for, and that is where an explanation can start.

The full report

You could work out thirty numbers by hand from the grid: precision, recall and F1 for each of ten digits. scikit-learn's classification report does it for you. Call classification report with y val and y pred and print the result. You get one line per digit with four columns. They are precision, recall, F1-score and support. Support is how many real examples of that digit are in the validation set. It is the row total from your grid.

Check one line against the grid so you trust the report. Take a digit. Its recall in the report should match the diagonal cell of the row-normalized matrix. Its precision should match the diagonal of the column-normalized one. If they agree, you are reading both correctly.

Below the per-digit lines, the report prints three summary lines.

The first is accuracy, the same number you already had.

The second is the macro average. Macro means add up the ten per-digit scores and divide by ten. Every digit counts the same, however many examples it has.

The third is the weighted average. Each digit's score is weighted by its support, so common digits count for more.

On MNIST these two lines will be close. The digits are close to balanced. Ones are the most common at about eleven percent of training images, and fives the least common, but the gap is small. When every class has about the same support, weighting by support changes very little.

On other data they can split far apart. Say you have a three-class problem where one class makes up ninety percent of the data. The model does very well on it, with F1 near point nine five. The two small classes do badly, near point three each. The weighted average is mostly the big class, so it comes out around point nine. The macro average gives each class a third, so it lands near point five. Both are true. They answer different questions. Weighted asks how good a typical prediction is. Macro asks how good the model is on each class, including the rare ones. If the rare classes are the ones you care about, macro is the honest summary. A good weighted average can hide a class the model has almost given up on.

The report has one more option worth knowing. If a class never gets predicted, its precision has zero on the bottom. That cannot be computed, and scikit-learn gives a warning by default. The zero division argument picks what to use instead. Setting it to zero prints a zero. Setting it to NumPy's not-a-number marker leaves that class out of the macro average instead of counting it as zero.

When accuracy misleads

You have already met a model whose accuracy hid everything. The DummyClassifier from two chapters ago predicted one for every image. On the test set it scored about eleven percent. Its recall for ones was a hundred percent and its recall for every other digit was zero. Feed its predictions to classification report and you would see that at once. There would be one line with perfect recall, nine lines of zeros, and a macro F1 near the floor.

On MNIST that baseline's accuracy is low, so nobody is fooled. The danger comes when one class is rare.

Say you are building a detector where one case in a hundred is a positive. It could be a fraud check, a fault alarm, or a screening test. You have ten thousand cases, so a hundred are positive and nine thousand nine hundred are negative. Now take the laziest possible model, the same idea as the DummyClassifier. It always says negative. It is right on all nine thousand nine hundred negatives, so its accuracy is ninety-nine percent. Its recall on positives is zero out of a hundred. It never finds a single one. Its precision on positives can't even be computed, because it never makes a positive claim. A ninety-nine percent model that catches nothing is the exact case these new numbers exist for.

The general rule is this. Accuracy is the share of predictions that are right. When one class is rare, guessing the common class wins almost all of that share for free. So before you trust an accuracy figure, ask what the most-frequent baseline would score. Then look at recall and precision on the class that matters.

Moving the threshold

So far, every prediction has come from the model's top choice. Logistic regression gives ten probabilities per image, one for each digit, adding to one. Predict returns whichever digit has the highest. But you can choose the rule yourself, and doing that shows the trade between precision and recall directly.

Call predict proba on the validation images. You get a ten-thousand-by-ten array. Each row is one image, and each column is one digit's probability, in the order of the model's classes attribute. Pick one digit to focus on, say eight. Treat it as eight against everything else. That is called one-versus-rest.

The usual rule calls an image an eight when eight has the top probability. That might happen at forty percent if the other nine digits split the rest. A new rule: call an image an eight whenever its eight probability is at least some cutoff you choose. Otherwise, give it the most likely of the other nine digits. You can write that with NumPy's where function. Then compute precision and recall for eight alone. First turn both label lists into yes-or-no arrays: y val equals "8", and your new predictions equal "8". Then pass those two arrays to precision score and recall score.

Now move the cutoff and watch what happens.

Lower it to, say, twenty percent. More images pass. Some are eights the model was unsure about, so recall goes up and fewer real eights slip through. Others are threes and fives that looked a little like eights, so precision goes down and more false eights get in.

Raise the cutoff to, say, ninety percent. Only images the model is very sure of pass. Those are nearly all real eights, so precision goes up. But many real eights fall short and get missed, so recall goes down.

You can't push both up by moving the cutoff. You can only slide along the trade. A better model shifts the whole trade in your favor, but the cutoff only picks a point on it.

Which point is right is not a question about the model. It is a question about the cost of each mistake. Say a missed positive means a fraud payment goes through, and a false alarm means a person spends a minute checking a flag. Then missed cases cost far more, and you lower the threshold to raise recall. Now say each positive call sets off something expensive or annoying, like blocking a customer's card. Then false alarms cost more, and you raise the threshold to protect precision. Nothing in the data settles this. You settle it, from what the errors cost in the place the model will be used, and you write that reasoning down beside the number you report.

Choose the threshold on validation data, for the same reason as before. A cutoff tuned while watching the test set is a choice shaped by the test set.

Your exercise

This exercise is the first piece of the error analysis your first portfolio project will need. Do every step on validation data.

1. Look at your row-normalized confusion matrix and write down which digit you think has the worst recall, and roughly what it is, before you compute anything.
2. Run classification report, find the actual lowest recall, and compare it with your guess.
3. Find the validation images of that digit that the model got wrong, by keeping rows where the true label is that digit and the prediction is not.
4. Reshape ten to twenty of those rows to twenty-eight by twenty-eight and show each with matplotlib's imshow, with the predicted label as the title.
5. Write one sentence on what those errors have in common, such as a stroke style, a slant, a thickness, or one digit they keep being called.

Check yourself this way. Your sentence should point to something you can see in the images, and it should agree with the largest off-diagonal cell in that digit's row. If the images say "these look like nines" and the grid says they were mostly called sixes, look again. If some images look unreadable even to you, write that down too. Some errors come from the model, some from messy handwriting, and a few from labels a person may have gotten wrong. Telling those apart is what error analysis is for.

Python 3.15 is out

One piece of news connects to your project. Last chapter, Python 3.15's final release had slipped. Blocking bugs in the new lazy-import feature led the release manager, Hugo van Kemenade, to put out a third release candidate and move the final date to the ninth of October. The final release has now shipped.

It is the same release you heard about before, now finished. Lazy imports, from the proposal called PEP 810, add an opt-in way to write an import that waits. You write lazy import followed by the module name. Python then hands back a light stand-in and does not load or run the module until your code first uses it. Ordinary imports work exactly as they did before. That matters for tools that import many heavy libraries but only use some of them on a given run. UTF-8 is now the default text encoding for file reading and writing. There is a built-in frozendict, a dictionary that can't be changed after it is made. And there is Tachyon, a low-overhead sampling profiler under a new profiling package, which can attach to a program that is already running.

For you, the point is not to switch right away. Your notebook works on 3.14, and a fresh language version is exactly the kind of change that can quietly move results. The check from before now has something to run against. In a copy of your project, not the original, pin Python to 3.15 and run uv sync. Then run uv tree to confirm which versions of pandas and scikit-learn you actually got, since both have published wheels for 3.15. Restart and run the whole notebook. Compare its validation accuracy and its per-class numbers with your 3.14 run, and now compare its classification report too. Matching numbers mean the upgrade is safe for this project. Numbers that move are worth understanding before you switch, and the per-digit lines will show you where they moved.