Learning Rules From Examples Instead of Writing Them

When rules fail and examples work

The chapter opens with two small programs that look equally simple but aren't. Withholding tax from a paycheck is a rule someone already wrote down — you code it in six lines and check it by hand. Spam filtering looks the same size but collapses: every fix (bad spelling, hidden HTML, image text, padding with harmless prose) is immediately worked around, and the rules start fighting each other until nobody can say what the filter does. Yet a person can still sort spam from real mail in seconds. That gap — recognizable but not describable, with plenty of examples on hand — is where learning from data belongs instead of hand-written logic, a distinction laid out clearly by pieces comparing rule-based and machine learning approaches to language tasks and when to choose one system over the other. Handwritten digit recognition tells the same story without any adversary involved: loops that don't close, sevens with and without a bar, forced the field toward systems like MNIST that learn the rule from labelled pixels instead of hand-coded shape logic.

From there the chapter separates five words people use loosely and interchangeably — artificial intelligence, machine learning, deep learning, statistics, and data science — giving each one a problem it actually owns, from shortest-path search to A/B test confidence intervals to the messy plumbing of real datasets. It rejects the idea that these sit on a ladder with deep learning at the top, pointing out that a hand-written rule, a plain average, or a significance test is frequently the correct and more honest answer.

Turning a wish into a stated problem

The back half works a single example end to end: a vague request to "reduce churn" gets turned into a fully specified prediction problem — one subscriber, one month, features known before the fact (with a live demonstration of leakage via a cancellation-reason field filled in only after someone leaves), a yes/no target, historical labels drawn straight from billing records, and a baseline ("always say no," 96% accurate and useless) that any real model has to beat. It closes by turning to MNIST itself — sixty thousand labelled digit images reduced to raw pixel brightness with no notion of a line or curve — as the plainest way to see what a model is actually handed before it learns anything.

Here is a small program anyone reading this could write today.

A payroll system needs to work out how much income tax to withhold. The law says the first eleven thousand dollars of income is taxed at ten percent, and everything above that up to forty-four thousand is taxed at twelve percent. So for someone earning thirty thousand dollars, you take ten percent of the first eleven thousand, which is eleven hundred dollars. Then you take twelve percent of the remaining nineteen thousand, which is two thousand two hundred and eighty. Add them and you get three thousand three hundred and eighty dollars.

You could write that in six lines. Better, you could check it by hand. If the law changes next year, you change one number and the program is correct again. There is nothing to learn here, because the rule already exists, written down by someone else, and it is exact. A system that guessed at this number and came within a dollar or two would be worse than useless. It would be wrong in a way nobody could audit.

Now here is a second program, and it looks like it should be about as hard.

An email arrives. Decide whether it is spam.

The first attempt writes itself. Keep a list of words that spam uses and legitimate mail does not. If the message contains one of them, throw it away. This works for about a week. Then the senders start writing the word with a one in place of the i and an at-sign in place of the a. So you add those spellings to the list. Then they hide the word inside HTML markup, so the letters are broken up by tags that the mail program does not display but your rule can see. So you write a rule to strip the tags first. Then they send an image with the text inside the picture, where there are no letters at all. Then they pad the message with paragraphs of ordinary, harmless prose lifted from a news site, so that whatever else you measure, the message looks statistically normal.

Every fix you make is correct. Every fix you make is also immediately worked around, because on the other end of this there is a person who gets paid when your rule fails. And the rules interfere with each other. The rule you wrote to catch the misspellings starts throwing away a real message from a friend with an unusual name. The rule you wrote to catch the padding starts throwing away newsletters people asked for. After a few hundred rules, you no longer know what your own program does, and you cannot change one rule without breaking another.

That is the collapse. Not that the task is impossible, but that the rule you would need has thousands of exceptions and nobody, including you, can list them.

And yet notice something. You can still tell. Show a person a hundred messages and they will sort them into spam and not-spam in about two minutes, and they will agree with the next person on ninety-something of them. The knowledge is there. It just is not in a form you can type.

That gap is the entire subject of this course, and it is worth saying in one sentence, because everything else in this chapter hangs off it. Machine learning is what you reach for when you can recognise the right answer on examples but cannot write down the rule that produces it.

Tax withholding fails that test in the good direction: the rule is written down already, so write it. Spam fails it in the other direction: you can recognise the answer all day and never produce the rule. So instead of writing the rule, you hand the machine the recognising and ask it to produce the rule itself.

What you actually hand over is worth being precise about, because the shape never changes for the rest of this course, no matter how large the model gets.

You hand over a pile of examples. An example is one thing you want a decision about. Here, one email.

For each example, you hand over some measurements of it. Each measurement is called a feature. For an email, a feature might be the number of words in the message, whether the sender's address has ever written to you before, how many links it contains, how many of those links point somewhere other than where their text claims, or simply whether some particular word appears. Features are the only thing the machine gets to look at. If a feature is not there, the machine cannot use it, however obvious it is to you.

And for each example, you hand over the answer you already know. That known answer is called the label. Spam, or not spam. You know these because a human sorted them, or because users pressed a button, and either way somebody paid in attention for every one of them.

So the input to learning is pairs: features, and a label. Many thousands of them.

What comes out is called a model. A model is just a rule — but a rule that was fitted to those pairs rather than typed by a person. It takes features in and puts an answer out. When it does that on a message it has never seen, its output is called a prediction. The word is a little grand for what it is. It does not mean the model knows anything about the future. It means the model is producing an answer where you have not supplied one.

And here is the part that makes the whole thing hard, rather than merely fiddly. The model is only useful on examples nobody has seen yet. It is very easy to build a rule that gets every single one of your training examples right — memorise them, look each one up. That rule is worth nothing, because you already had those answers. The rule has to work on the next message, the one that has not arrived. Every technique in the rest of this course is, in one way or another, about that problem.

Handwritten digit recognition tells the same story as spam, from a completely different direction, and it is worth a moment because it shows the collapse is not about adversaries.

Postal services need to read the digits in a handwritten postcode. Nobody is trying to fool the machine. People are just writing. So you try rules again, and they are sensible rules: a zero is a single closed loop, an eight is two closed loops stacked, a one is a stroke with no loops and no branches, a four has a crossing. Then you meet real handwriting. Loops that do not quite close. Strokes that break in the middle because the pen skipped. A seven with a bar through it and a seven without. A one that leans so far it could be a slash. Every one of those is a special case, the special cases multiply against each other, and you are back where the spam filter was.

What worked instead was to give a system many thousands of small images of digits, each one already labelled with the digit a human says it is, and let it fit the rule itself from the raw pixels. The most-used collection of these is MNIST, a set of scanned handwritten digits with their labels, and it is still the first thing many people ever train a model on. Each example is one image. Its features are the brightness values of the pixels. Its label is a digit from zero to nine. Nobody ever tells the system what a loop is.

Both stories have the same shape. Recognising was easy, describing was impossible, examples were available. Where those three things are true together, learning from examples is the right tool. Where any one of them is false, it usually is not.

Five words for five different problems

The trouble with this field is that five words get used for it, often in the same sentence, often by people who are not being careful. Artificial intelligence, machine learning, deep learning, statistics, data science. Let me give each of them a problem it owns, and then say the thing about them that most people get wrong.

Artificial intelligence is the oldest and broadest of the five. It is the ambition of building systems that perceive something about their situation and then act sensibly toward a goal. That is a wide enough description to include a great deal that does no learning at all. A program that finds the shortest route across a city is doing artificial intelligence, and it works by systematically searching possible routes and keeping the cheapest, not by studying past journeys. A program that fills in a timetable so that no teacher is in two rooms at once is doing artificial intelligence, and it works by narrowing down possibilities until only consistent ones remain. Give either of those tasks to a system that learns from examples and you have made it worse. The shortest route is a thing you can compute exactly, and a model that predicts a route would sometimes hand you a road that does not connect.

Machine learning is one approach inside that broader ambition: the approach where the rule comes from data instead of from a person. That is what the spam filter and the digit reader have in common, and it is the whole of what the word means. It owns exactly the problem we started with — recognisable, undescribable, and supplied with examples.

Deep learning is one family inside machine learning. Its models stack many layers, and each layer transforms what came out of the layer beneath it, with every one of those transformations learned rather than designed. What that buys you is that you can feed in raw, unhelpful input — pixel brightnesses, a waveform, a sequence of characters — and the early layers will build up their own useful measurements from it, instead of you having to invent good features by hand. That is why the digit problem went to this family. Nobody had to define a loop, because the layers built up something loop-like on their own out of pixels. The cost is that this needs a lot of examples and a lot of computation, and that what those middle layers are doing is genuinely hard to inspect.

Statistics is not a subfield of any of these. It is an older discipline, and it asks a different question. Not "what answer should I give for this new case" but "what does this sample let me claim, and with how much confidence." Suppose you change the sign-up page on a website, and over two weeks the new version converts at four point one percent while the old one converts at three point eight. Statistics owns that problem, and it owns it because the interesting question is not prediction at all. The question is whether that gap is real or whether you would see a gap that size fairly often from random noise alone with that many visitors. That is what a confidence interval or a significance test is for. A predictive model will happily give you an answer here and it will be answering the wrong question, because a model that minimises prediction error is free to lean on anything that correlates, including the time of day the two versions happened to be shown.

Clinical trials belong to statistics for the same reason, with more at stake. You want to know whether a drug works, which means you need random assignment, a stated hypothesis, and an honest account of how likely your result is if the drug does nothing.

Data science is not a theory at all. It is the working practice of getting value out of data end to end: finding the data, pulling it out of wherever it lives, discovering that a third of the rows have the date in the wrong format, deciding what to do about the missing values, describing what is actually in there, and then explaining the result to someone who will make a decision with it. It uses statistics and it uses machine learning as tools inside that pipeline. It owns the problem of the messy real supply line, which is most of the work on most projects and is almost none of the material in most courses.

They overlap, and honestly. Linear regression is taught in statistics courses and in machine learning courses, and it is the same mathematics in both; what differs is what you do with it. A statistician tends to care what the fitted coefficients mean about the world and whether the assumptions behind them hold. A machine learning practitioner tends to care whether the predictions are good on data the model has not seen. Both of those are legitimate, they are not the same goal, and the same equation serves both.

Now the misconception, said directly, because it will cost you real time if you carry it.

These five are not five rungs of a ladder with deep learning on top. That picture says statistics is what people did before they had computers, machine learning is what you use when you cannot afford deep learning, and deep learning is what you use when you are serious. Every part of that is wrong.

The correct answer to a problem is frequently a hand-written rule. Payroll was one. The correct answer is frequently a plain average with an honest error bound on it, because the question was "what is our typical response time" and adding a learned model to that adds nothing but opacity and something new to break. The correct answer is frequently a significance test, because the question was whether a difference is real. And when the answer is machine learning, the correct answer within machine learning is often a small, simple, inspectable model, because it is nearly as accurate as the large one, it trains in a second, and you can see what it is doing.

Choosing correctly is a skill, and it is a harder one than running any particular algorithm. It has no shortcut and no rule of thumb. It comes from having stated the problem clearly enough to see what kind of problem it is.

Which is the work we are going to do now.

Turning a wish into a stated learning problem

People do not bring you learning problems. They bring you wishes. Here is a real one, in the form it usually arrives:

"We want to use AI to reduce customer churn."

That sentence has no unit, no input, no output, no data and no standard. It is not a task, it is a mood. Every one of those five gaps has to be filled before anyone writes a line of code, and filling them is what the rest of this chapter does, out loud, on this exact sentence.

Say the business is a subscription service. People pay monthly, and some of them stop.

First: what is one prediction about? This is the unit, and getting it wrong quietly ruins everything downstream. It is tempting to say "churn" is the thing being predicted, but churn is not a thing, it is a rate. One prediction has to be about one identifiable something at one identifiable moment. So: one prediction is about one subscriber, on the first day of one month. That is the unit. If we have a subscriber who has been with us for two years, that person contributes twenty-four separate examples, one per month, not one.

Second: what do we know at the moment of prediction, and is it really known then? This is where most beginner projects die, and they die silently, which is worse.

For a given subscriber on the first of the month, we can look up how many months they have been paying, which plan they are on, what they pay, how many times they logged in during the previous thirty days, how many support tickets they opened in the previous ninety days, and whether their last payment failed. All of that is in our billing and application records, dated, and all of it was recorded before the first of the month. Those are our features.

Now the trap. Our records also have a field called cancellation reason, filled in by a support agent when someone leaves. It is enormously informative. It is also filled in after the person cancels. If we hand that to the model, the model will learn that a filled-in cancellation reason means cancellation, it will score almost perfectly in testing, and it will be worth exactly nothing in use, because on the first of the month for a subscriber who has not cancelled, that field is empty and always will be. Information that only exists after the answer is called leakage, and the test for it is one question asked about every single feature: on the morning I actually have to make this prediction, will this value be sitting in front of me? Cancellation reason fails. A support ticket count fails too if it is counted over the coming month rather than the past one. The same measurement can be legitimate or fatal depending on the window it covers, so the window has to be part of the feature's definition.

Third: what exactly is the target? We say: whether this subscriber cancels at any point in the next thirty days. That is a category, and it has two values, cancelled or not. It is not a number.

That choice is not cosmetic. It decides the method, the way you measure success, and what the output even means. If we had instead defined the target as how many more months this subscriber will stay, that would be a number, and it would take a different family of models and a different notion of being wrong — being off by two months is worse than being off by one, whereas with a category you are either right or you are not. Both versions are answerable. They are different problems and they cannot share a solution. Pick one, in writing, before anything else.

Fourth: where do the labelled examples come from, and how many are there? Notice we do not need anyone to sit down and label anything, because the past already did it. For every subscriber and every past month, the billing records tell us whether they cancelled in the following thirty days. The label was recorded as a side effect of running the business, and that is why this problem is worth attempting at all.

Let us count. Say we have eight thousand subscribers and two years of usable history. Eight thousand times twenty-four is roughly a hundred and ninety-two thousand subscriber-months. That is our supply of examples, and it is plenty. But now count the labels, not the examples. If about four in a hundred subscribers cancel in a given month, then about four percent of those rows are labelled cancelled. Four percent of a hundred and ninety-two thousand is about seven thousand seven hundred. So we have a hundred and eighty-odd thousand examples of nothing happening, and seven or eight thousand of the thing we actually care about.

Which brings us to the fifth thing, and the one people skip.

What result would be useful, and what result would be no better than the obvious guess?

Start with the obvious guess, because it is startling. A model that ignores every feature and always says "this person will not cancel" is right about ninety-six percent of the time. It is right ninety-six percent of the time because ninety-six percent of the rows are people who did not cancel. It requires no data, no features and no training. You can implement it by returning the same answer forever.

So if we build something clever and it comes back at ninety-five percent accurate, we have done worse than a rule that says nothing. And if it comes back at ninety-six point five percent, we have not learned whether it is any good, because we do not know whether that half a point is real or noise. That fixed always-say-no rule is what we will call a baseline: the cheapest defensible thing, whose score you must beat before you are allowed to be pleased.

The baseline also tells us that accuracy is the wrong measure here, which is a conclusion we could not have reached without computing it. It is not that accuracy is a bad idea in general. It is that on this problem, with these proportions, it cannot distinguish a useful model from a rule that does nothing.

So define usefulness from the actual decision instead. Suppose the retention team can meaningfully contact two hundred subscribers a month. Then what we want from the model is a ranking, and the question we ask of it is: of the two hundred subscribers it puts at the top, how many actually cancel in the next thirty days? Pick two hundred subscribers at random and, at a four percent rate, about eight of them will be leavers. That is the floor. If the model's top two hundred contains sixty leavers, the team is reaching seven or eight times as many of the right people as random selection would, and that is a result you can put in front of the person who owns the budget. If it contains twelve, the model is doing something, but not enough to change anyone's week.

Notice what we have and have not established. Sixty out of two hundred would establish that the model ranks better than chance on our historical records. It would not establish that contacting those people reduces cancellations, because we would have shown who is likely to leave, not that any intervention changes their mind. That is the statistics question from earlier, it needs a controlled comparison, and it is a separate piece of work with a separate design. Being clear about which of the two you have proved is the difference between an honest project and a misleading one.

This last step is not paperwork. A project with no stated baseline cannot tell success from noise. It has no way to know whether ninety-six percent is triumph or embarrassment. And the sequence matters: the baseline has to be written down before you see the model's score, because afterwards you will find yourself constructing a standard that the number you happen to have clears.

So the wish has become a statement, and it fits in a paragraph. One prediction is about one subscriber on the first of a month. The features are tenure, plan, monthly price, logins in the past thirty days, support tickets in the past ninety days, and whether the last payment failed, all of them known on that morning and none of them drawn from after it. The target is a category: cancels within thirty days, yes or no. The labelled examples come from two years of billing history, roughly a hundred and ninety-two thousand rows, of which about four percent carry the cancelled label. The baseline is the rule that always says no, which is ninety-six percent accurate and useless, and random selection, which finds about eight leavers in two hundred. Useful means finding several times that many in the top two hundred, and success at that means the ranking works, not that anyone was retained.

Nothing there is about technology. That is the point. This same shape — unit, features known at prediction time, target and its type, source and quantity of labels, baseline and standard — is what you write whether the thing that eventually gets fitted is three lines of library code or a system with millions of parameters. The shape does not change with the size of the model. Skipping it does not get you to the interesting part faster; it gets you to a number you cannot interpret.

Before we leave this, take the same wish and turn it into a different problem, so you can feel which choices are yours. Keep the same business, but suppose the finance team asks instead how much revenue we will lose to cancellations next quarter. Work through the five: the unit is no longer a subscriber, the target is now a number rather than a category, and the baseline is no longer a fixed answer but something like last quarter's figure. Write those five lines out. If your target came out as a number and your baseline came out as a comparison against last quarter, you have understood the exercise.

Now, one thing worth putting in front of you today, connected to everything above.

The single most valuable thing a beginner can go and look at is not a model. It is a labelled dataset, opened up and read.

Take MNIST, the collection of handwritten digits from postal scanning. Its structure is exactly the vocabulary from earlier, made physical. One example is one small greyscale image of a single digit. Its features are the brightness values of its pixels, laid out as a grid twenty-eight cells wide and twenty-eight tall, so seven hundred and eighty-four numbers per example, each between fully dark and fully bright. Its label is a single digit from zero to nine, written down by a human who looked at that image. There are sixty thousand of these set aside for training and ten thousand held apart for testing, and that separation is deliberate rather than incidental: the ten thousand exist so that the score you report is a score on examples the model never saw, which is the only score that means anything.

It comes bundled with several standard Python libraries, so loading it is a single function call, and the thing to do first is not to train anything. Print one example. Look at the seven hundred and eighty-four numbers. Then display that same row as an image and see the digit appear. Then print its label and confirm it matches what your eyes say.

Do that once and something clicks that no explanation delivers. The model is not shown a digit. It is shown seven hundred and eighty-four numbers with no notion that they sit next to each other, no notion of a line or a curve, no idea that a shape exists, and it is asked to produce a label. Everything the field has built to solve that comes from the difficulty you can see in that one printed row.

Then count what it took to make. Sixty thousand images, each looked at by a person who wrote down what they saw. That is the real cost of learning from examples, and it is why the churn problem above was attractive: nobody had to sit and label anything, because the business had been labelling it all along without noticing.