3. What Signal the Machine Actually Gets

The one question behind four names

Supervised, unsupervised, reinforcement, self-supervised are not four families of algorithms. They are four answers to one question: what does the system get told, and when. Everything here follows from that.

Answers already written down

MNIST as the worked case: pixels paired with a digit a person chose. The correct answer sits beside every guess, which makes the feedback immediate — and paid for in human labour. Sixty thousand images means sixty thousand decisions, and the price climbs when the judgment is a chest X-ray two radiologists may read differently, an email that is angry or merely brisk, or a cyclist boxed frame by frame.

Throw the labels away

The same pile of pixels with nothing attached. What remains is structure: which images resemble each other, and how much of the seven hundred eighty-four dimensions is redundant. The awkward part gets stated plainly — with no answer key there is no clean test. Six customer groups or eight? Nothing in the data holds an opinion, so you judge by downstream usefulness or by whether someone who knows the domain recognises the groups.

A number that arrives at the end

Tic-tac-toe as Sutton and Barto set it up in Reinforcement Learning: An Introduction: pick a free square, receive nothing, and after three to five of your own moves one number arrives. It says how things went, not what you should have done — the credit assignment problem. Values propagate backward along the path actually walked, one step at a time. And because the learner only sees consequences of moves it chose, it must sometimes act against its own current judgment to find out.

Targets hidden in the data

Cover a word and ask for it. A five-hundred-word page becomes roughly five hundred examples, and nobody annotated anything. It uses supervised machinery on invented targets, and it usually is not the last step — pretraining is followed by fine-tuning on far fewer real labels.

What you have, not what sounds advanced

Loan repayment records, uncategorised support transcripts, and a home-screen recommender that can be read either way. The four settings are not a ranking.

GLM-5.3, read through its reward

Z.ai's GLM-5.3 release keeps GLM-5.2's 743-billion-parameter base with no further pretraining and scales the reinforcement learning: verified multi-step environments. Reported jumps on Terminal-Bench 3.0 and DeepSWE are evidence about those benchmarks and nothing else. The open weights were held back around two weeks after cyber capability scaled faster than expected.

Two chapters ago we drew a line between problems where somebody can write the rule down and problems where only examples can reveal it. Last chapter we followed that line through seventy-five years of arguing about who supplies the rule. Both times, one item kept getting pushed forward: the four learning settings. Supervised. Unsupervised. Reinforcement. Self-supervised. Now it is time to take them properly.

Here is the thing to hold onto before any of the four names, because it makes the whole subject smaller than it looks. These are not four families of algorithms. They are four answers to a single question: what does the system get told, and when does it get told it?

That question sounds almost too plain to be the organizing idea of a field. It is the organizing idea anyway. When you sit down with a real problem — a spreadsheet, a folder of images, a running service, a pile of text — the first useful thing you can do is ask what feedback is actually available. Not what model you would like to use. Not whether the problem sounds like deep learning. Just: what does the world hand back, and how soon? The answer decides which setting you are in, and the setting narrows what methods can even apply. Everything else in this chapter follows from that.

Let me start with the setting we already own.

When somebody already wrote the answers down

We have already formulated a supervised problem in this course, and we already looked at MNIST closely enough to know what one example is made of: a small grid of pixel brightnesses, and next to it a single digit that a person decided the image showed.

That second part is the whole definition of supervised learning. Every training example arrives paired with the answer. The input, and the target. The pixels, and the four. The learner's job is to find a function from one to the other, and it can check itself constantly, because for every example it guesses, the correct answer is sitting right there. Guess seven, the truth was four, so something was wrong and by how much is measurable. That immediacy is what makes supervised learning the easiest setting to reason about, and the reason we teach it first.

Now say the cost out loud, because it is easy to skip past. Somebody had to label every one. Sixty thousand training images means sixty thousand human decisions about what digit was written. Not sixty thousand hard decisions — a person can do a digit in a second — but sixty thousand of them, made by someone, recorded somewhere, before any learning could happen at all.

Digits are the cheap end. Consider what labelling costs when the judgment is harder. Deciding whether a chest X-ray shows early pneumonia takes a trained radiologist, and two radiologists may disagree. Deciding whether a customer email is angry or merely brisk takes a person who reads it, and different readers will split. Deciding whether a stretch of road video contains a cyclist means somebody drawing a box around the cyclist, frame by frame. Every labelled example in supervised learning is a small purchase, and the price varies from a second of anyone's attention to an hour of an expert's.

So supervised learning has the best signal in the business, and it pays for that signal in human labor. Hold that trade in your head, because the other three settings are all, in one way or another, responses to it.

What is left when the answers are gone

Take MNIST and throw the labels away. Sixty thousand grids of pixel brightnesses, and nothing telling you what any of them is. What could a system possibly learn now?

Not what digit an image shows. That word "four" came from a person, and you have deleted the person. Whatever structure the pixels contain, none of it says "the culture that wrote this calls this shape four."

But something is still there. Some of those images resemble each other far more than they resemble others. If you had a notion of how similar two images are, you could sort the pile into groups where each group's members look alike and the groups look unlike each other — and you would probably end up with something in the neighborhood of ten groups, because the data really was produced by ten kinds of thing. You could also notice that the seven hundred eighty-four pixel values are enormously redundant: the corners are almost always dark, neighboring pixels almost always agree, and the real variation in the collection lives in far fewer dimensions than seven hundred eighty-four. Compressing the data down to those few dimensions, without losing much, is also something you can do with inputs alone.

That is unsupervised learning. The learner gets inputs and nothing else, so the only thing it can look for is structure in the inputs: what groups with what, what varies with what, what could be described more compactly, what looks unlike everything else in the pile.

And now the honest part, which matters more than any technique. There is no answer key, so there is no clean way to tell whether the result is good.

Sit with that, because it is genuinely awkward. In supervised learning you hold back examples, predict their labels, and count how many you got right. The score means something because the labels came from outside the model. In unsupervised learning, suppose your system sorts a million customers into six groups. Are six groups correct? Would eight have been better? Is that group in the corner a real kind of customer or an artifact of how you measured similarity? Nothing in the data answers those questions, because the data never contained an opinion about them. You end up judging the output by whether it is useful for something downstream, or by whether a person who knows the domain looks at the groups and recognizes them. Those are real forms of evidence, but they are softer than a test-set accuracy, and pretending otherwise is how unsupervised results get oversold.

Notice that the shift from supervised to unsupervised was subtraction. We removed the signal. The next setting does something more interesting: it keeps a signal, but changes its shape.

A number that arrives late and says nothing about what to do

In reinforcement learning, the system does not sit and receive examples. It acts. It is inside something — a game, a simulation, a machine, a market — and at each moment it chooses one of the actions available to it. Then, sometimes, a number comes back. That number is called a reward, and it is very nearly the whole of the feedback.

Two things about that number are strange compared with a label, and both of them are the substance of this setting.

First, the number says how well things went. It does not say what you should have done. A label says "four." A reward says "one." Nobody has told you which of your choices earned the one.

Second, it often arrives long after the choices that produced it.

Let me make both of those concrete with one small task, one that Richard Sutton and Andrew Barto use in Reinforcement Learning: An Introduction, in the chapter where they first show what this setting is. The task is tic-tac-toe. Three by three board, one player against an opponent.

Here is what the agent does. On its turn, it looks at the board and picks one of the squares that is still empty, and places its mark there. That is the entire action set: the free squares. Nothing more exotic than that.

Here is what comes back during the game. Zero. Not a small number, not a hint, not a nudge — nothing at all. There is no bonus for taking the center, no penalty for wasting a move, no partial credit for having two in a row. Play a move, the board changes, the opponent replies, and no feedback of any kind has occurred.

Then the game ends, and exactly once, a number arrives. A win pays one. A loss pays zero, or in the version where the two players' payoffs mirror each other, minus one. A draw sits in between: zero, or a half, depending on how you set it up. That is the signal. One number, at the end.

Now count the gap. A full game runs somewhere between five and nine turns in total, which gives our player three to five decisions of its own. So the move that opened the game is separated from the only feedback in the entire episode by several of its own later moves and several of the opponent's replies. The signal is not late by a rounding error. It is late by the whole game.

And here is why that is hard, stated as concretely as I can. Suppose the agent takes the center on its first move, and on its fourth move it completes a line and wins, and the number one comes back. What does the one tell it? It tells it that the sequence, taken as a whole, worked. It does not tell it whether the center opening was the reason. Maybe the center was decisive. Maybe the center was irrelevant and the opponent simply blundered on its third move, handing over a game that any opening would have won. Maybe the first three moves were mediocre and the fourth was a nice piece of play that rescued them. The scalar one is identical in all three cases. It cannot distinguish them, because a single number carries no breakdown.

This is called the credit assignment problem, and now the name should be self-explanatory: an outcome has to be assigned back across the actions that led to it, and the outcome itself does not say how to divide it.

So how does anything get learned? Sutton and Barto's tic-tac-toe player works like this. Instead of trying to score moves directly, it keeps an estimate for each board position of how likely it is to win from there. Call that a value for the state. At the end of a game, the value of the final state is adjusted toward what actually happened — you won from there, so that position was better than you thought. But the adjustment does not stop at the last position. The position before it gets nudged toward the position that followed it. And the position before that gets nudged toward the one that followed it. Over many games, the information that arrived once, at the end, seeps backward along the paths the agent actually walked, until early positions carry a sensible estimate of where they tend to lead. That is temporal-difference learning, and we will build it properly much later. What matters right now is the shape of the trick: a single late number becomes usable by being propagated back through the states that preceded it, one step at a time, over many repetitions.

One more thing follows from a reward-only signal, and it is worth naming even at map level. The agent's data depends on its own behavior. A supervised learner gets whatever dataset you hand it. A reinforcement learner only ever sees the consequences of moves it chose to make. If it never takes the center, it never learns what the center leads to. So it has to sometimes do things it does not currently believe are best, purely to find out. That tension has a name too — exploration against exploitation — and it exists in this setting and not the others, because in this setting the learner's choices decide what it gets to observe.

Labels the data already contained

The fourth setting is the one that explains the last several years, and its idea is almost cheeky.

Suppose you want supervised learning's excellent signal without paying for human labels. So you take unlabelled data, hide part of it, and train the system to predict the hidden part from the part it can still see. The target is the piece you covered up. Nobody wrote it down. It was already in the data, and you created a prediction problem by concealing it from yourself.

That is self-supervised learning. Targets manufactured out of the input.

Ordinary text is the clearest case, and it is the one the previous chapter already put in front of you when it got to large language models. Take any sentence. Show the model the words up to some point and ask it for the next word. Then slide along and do it again. The answer is never in doubt, because the next word was sitting there in the text the whole time. You did not annotate anything. You covered a word with your hand.

Now do the arithmetic on how much training data that gives you, because the consequence is the entire reason this setting matters. A page of text with five hundred words is not one training example. It is roughly five hundred of them, one for every position where you can stop and ask what comes next. And the supply of pages is not something anyone has to produce for you. Books, code, documentation, transcripts, forum arguments, product manuals — all of it was written for its own reasons by people who had never heard of your model, and all of it is already labelled data for the next-word problem.

So put the two settings side by side. Supervised learning on images needs a person per example, which means a serious labelling effort buys you tens of thousands of examples, and a very large industrial one buys you millions. Self-supervised learning on text needs nobody, which means the ceiling is however much text you can collect and afford to compute over. That is the difference between waiting on annotators and consuming everything that has been written. It is not a difference in cleverness of algorithm. It is a difference in what the training signal costs.

Two honest qualifications, because the four settings are cleaner as definitions than as boxes.

The first: self-supervised training is supervised machinery pointed at invented targets. Once you have covered a word and asked for a prediction, the situation inside the learner is exactly the supervised situation — here is an input, here is the correct output, here is how wrong you were. The same loss, the same gradient-based fitting, the same everything. What is new is only where the target came from. So if you find yourself asking whether self-supervised learning is a kind of supervised learning, the answer is that it uses supervised learning's mechanism and solves supervised learning's cost problem. It sits in its own category because of the origin of the label, not the mathematics of the fit.

The second: it usually is not the last step. A model pretrained by predicting hidden text has learned an enormous amount about how language goes together, but it has not been told what you want it to do. So the common pattern is to pretrain self-supervised on a great deal of unlabelled data, then fine-tune on a much smaller set of real labelled examples for the actual task — this kind of email is a complaint, this answer is the one a person preferred, this document belongs in that category. The expensive human labels come back, but you need far fewer of them, because the model is no longer starting from nothing. The settings compose. They are not rival camps.

Deciding which setting a problem is in

The definitions are the easy half. The skill worth having is diagnostic, and it is two questions asked in order.

What data actually exists right now? And what feedback will the world return if the system does something?

Not what data you wish existed. Not what feedback would be nice. What is there.

Try three cases.

First. A bank has two hundred thousand past loan applications, and for each one it knows whether the borrower repaid. It wants to judge new applications. What exists is inputs with outcomes attached, one outcome per application, sitting in a database. That is supervised, and the fact that the outcomes came from events rather than from an annotator's opinion does not change the setting. A label is a target attached to an example; it does not have to have been typed by a labeller.

Second. A company has four years of customer support transcripts and no categories at all. Somebody wants to know what people write in about. What exists is inputs only. Nobody has said what the categories are, which means nobody can say whether your categories are right. That is unsupervised, with the honest caveat attached: you will be judging the output by whether the support team recognizes the groups and can use them, not by a score.

Third, and this is the one I want you to get wrong first. A streaming service wants to pick which show to put at the top of the home screen. It has years of history: what it showed each person, and whether they clicked.

Read that as supervised and you have a perfectly reasonable problem. Input: a person and a show. Target: clicked or not. Historical data, ready to fit. Many production recommenders are built exactly this way and they work.

But look at the second question. What feedback does the world return when the system acts? It puts a show at the top, and what comes back is a click or no click — a number, after the action, with no statement of which show would have been the better choice. And what the system chooses now decides what data it will have later, because a show that never reaches the top of anyone's screen generates no evidence about itself at all. That is the reinforcement shape: act, get a number, and shape your own future observations.

So which is it? Both descriptions are legitimate, and which one you should adopt depends on what you can build and measure. Treated as supervised, it is far simpler, and it is honest about learning from what already happened. Treated as reinforcement, it can account for the fact that the system's own choices are narrowing what it will ever learn, but you now need a way to explore and a way to evaluate a policy you have not fully deployed, and both are hard. The point of the case is not to hand you the answer. It is that classifying a problem is a judgment about what signal you can obtain and act on, not a lookup on the surface of a description. The one thing you should refuse to do is decide it by which word sounds more advanced.

Which brings me to the misconception, stated plainly because it does real damage to beginners.

These four are not a ranking. There is no ladder with supervised at the bottom for simple tasks, then unsupervised, then reinforcement, with self-supervised at the top where the impressive models live. That story is tempting because the largest and most publicized systems were pretrained self-supervised, and because "supervised" sounds like the training-wheels version. It is still wrong.

The setting is dictated by the signal available to you. If you have inputs paired with the outcomes you care about, you are in the supervised setting, and using it is not a lesser choice — it is the setting with the strongest feedback anyone gets, which is exactly why it remains the workhorse of applied machine learning. If you have inputs and no outcomes, self-supervised methods will not conjure the outcomes you did not collect; they will give you a model that predicts hidden parts of your own data, which is valuable but is not the same as predicting the thing you were asked about. And reinforcement learning is not the advanced tier of the others. It is what you are forced into when the only feedback the world gives you is a late number about how things went, and its difficulty is a consequence of that poor signal, not a mark of prestige.

So when you meet a new problem, do not ask which setting is best. Ask what you have, and what the world will tell you, and let those two answers pick the setting for you.

That is the last of the conceptual map. We have separated the words, walked the history, formulated a supervised problem, and now sorted out what kind of feedback a learner can get. What we have not done is run a single line of code. Next chapter, that changes: Python, a notebook, and version control, set up through a small data exercise you can rerun and get the same result twice.

One release, read through its training signal

A short piece of current news, chosen because it is a clean illustration of the setting we spent the longest on.

On the fourteenth of August, 2026, Z.ai released GLM-5.3. The interesting thing about it is what did not change. The base model underneath is the same mixture-of-experts architecture as GLM-5.2, at seven hundred forty-three billion parameters, with no further pretraining. Same weights to start from, same size. What Z.ai scaled up was the reinforcement learning afterward — more compute spent on it, a wider variety of tasks, and many more synthetic task environments with multi-step dependencies, hidden state and separate programs that verify whether the job actually got done.

You should be able to see why that verifier matters, given the last hour. A synthetic environment where an independent checker can confirm success is a machine for producing exactly the signal reinforcement learning needs: the agent acts over many steps, and at the end a number says whether it worked. Not what it should have done. Whether it worked. Z.ai's reported gains are concentrated in long, autonomous, multi-step work, which is what that signal rewards.

Now the reading discipline. The numbers reported are real measurements of particular tasks, and they measure those tasks only. On Terminal-Bench 3.0, which tests running commands and using tools in a terminal across multiple steps, the reported score went from four point six to twenty-eight point three. On DeepSWE version one point one, which is resolving real software issues end to end, it went from forty-six point two to sixty-six point nine. Large jumps, and the honest way to state what they establish is narrow: on those benchmarks, on those task types, under whoever's harness produced the numbers. They are not evidence that the model writes better prose, explains a concept more clearly, or costs less to run. A benchmark is evidence about the thing the benchmark does.

And separate announcement from availability, because this release makes the distinction unusually visible. You can use GLM-5.3 now through Z.ai's application programming interface, through their coding plan, and in their ZCode environment. The open-weight release — the downloadable model — was held back at launch for roughly two weeks of safety evaluation, after its capability at building cybersecurity exploitation chains improved faster than expected. So at launch, "released" meant one thing through a paid interface and a different thing for anyone waiting to run the weights themselves. Get in the habit of asking which one a headline means.

One thing to look at, if you want fifteen minutes with this rather than a paragraph. Find the description of Terminal-Bench and read what a single task in it consists of: what the model is given, what it is allowed to do, and how the benchmark decides the task was completed. You are looking for the verifier — the thing that turns a long sequence of commands into one number at the end. That is the reward we spent this chapter's middle unpacking, in a real system rather than on a tic-tac-toe board.