This chapter introduces k-nearest neighbors as the first model built with no weights at all: the training data itself is the model, and every cost of "training" gets pushed to prediction time. It works through the mechanics of voting among nearby points, the two families of distance you can choose between, and why distance-weighted voting fixes ties. The bulk of the discussion lands on two traps that wreck the algorithm if ignored — unscaled features letting one column's units dominate every distance calculation, and the "curse of dimensionality," walked through concretely with bins and cells, where adding features makes every point roughly equidistant from every other. It covers how the number of neighbors, k, trades memorization against over-smoothing, and why cross-validation rather than training accuracy is the only honest way to pick it. It also shows why this model handles shapes like XOR that a straight-line classifier like logistic regression cannot. The chapter builds it two ways — using scikit-learn's KNeighborsClassifier and its regression counterpart, and from scratch in plain NumPy — and checks that both agree. It closes on the k-d tree and ball tree search strategies scikit-learn's nearest-neighbor algorithms documentation describes, why they help only below roughly twenty features, and the plain cost problem: memory and prediction time both grow with the size of the stored dataset, unlike a linear model.
Anthropic locked in permanent pricing on its Sonnet-tier model, at two dollars per million input tokens and ten per million output, confirmed on its site, and OpenAI updated its published model specification and rolled out a teen-specific version of its chat product, detailed on its site. SpaceX bought the company behind the Cursor coding assistant in an all-stock deal valued near sixty billion dollars, reported by Inc., and Stripe agreed to acquire OpenRouter for roughly eight billion, covered by a law firm briefing. A research group published an open pipeline for synthesizing tool-use training data for agents, posted to arXiv, and a new open-source tool that indexes a codebase and precomputes a dependency graph for coding agents over the Model Context Protocol is available on GitHub.
Every model we have built so far worked the same way underneath. Linear regression learned a set of weights. Logistic regression learned a set of weights and a bias, squashed the score with the sigmoid, and gave you a probability. In both cases, once training finished, the training data was disposable. You could delete the whole spreadsheet and the model would keep working, because the model was no longer the data. It was a short list of numbers and a formula.
Today we invert that completely.
k-nearest neighbors has no weights. There is no fitting step worth the name. The model is the training data. When you call fit, the library basically writes the data down and organizes it a little. All of the work, all of the cost, and all of the danger move to prediction time. That inversion is exactly why this algorithm is worth teaching this early, before decision trees, before anything fancier. Linear models hid two ideas inside their arithmetic that we never had to face directly: what it means for two examples to be similar as numbers, and what happens to similarity when a dataset has many columns. k-nearest neighbors puts both of those out in the open where you have to deal with them.
Start with the intuition, because it is the kind you already use. Suppose you want to guess what a house is worth. You would not begin with calculus. You would look at the houses nearby — same street, roughly the same size, sold recently — and you would guess something close to what they went for. Maybe you would average the three most similar sales. That is the whole algorithm. For a label instead of a price, the move is the same: find the examples most similar to the new one, look at what they were labeled, and go with the majority.
So here is the mechanics, plainly. A new example arrives. You compute a distance from that example to every single training example. You sort those distances. You take the nearest k of them. If you are doing classification, the neighbors vote and the most common label wins. If you are doing regression, you average their target values. That is it. There is no training loop, no loss function, no gradient. Nothing is being optimized. The reason this feels strange after two episodes of gradient-flavored thinking is that k-nearest neighbors is our first genuinely non-parametric model. It does not assume the world has some fixed shape with a fixed number of knobs. It just assumes that things close together tend to behave alike.
Which means the real content of the model — the entire modeling decision — is hiding in one word. Distance.
The default choice is the straight-line distance, the one you learned in geometry: take the difference between the two examples in each column, square each difference, add them all up, take the square root. That is Euclidean distance, and it is what most people mean when they say "close." The alternative you should also have in your hands is Manhattan distance: take the difference in each column, take the absolute value, and just add them up, no squaring, no square root. It is called that because it measures the way a taxi drives on a street grid — you cannot cut diagonally through the buildings, so you go across and then up. The two behave differently in a way you can feel. Squaring punishes one large disagreement much harder than several small ones. So Euclidean distance says two examples are far apart if they differ badly on any single feature, while Manhattan distance is more forgiving of one bad column and adds up steady small differences instead. Both of these are special cases of one general recipe with a power dial on it. Raise each difference to a power, add, take that same root. Set the power to two and you get Euclidean. Set it to one and you get Manhattan. Scikit-learn exposes it exactly this way: the metric is that general family by default, and there is a power parameter set to two, which is to say plain Euclidean distance unless you change it.
Do not accept that default without thinking about it. Choosing a metric is a modeling decision, the same kind of decision as choosing which features to include. You are telling the algorithm what "similar" means for your problem, and it will believe you.
There is one more knob on the voting itself. In the basic version every one of the k neighbors gets an equal vote. That is a little odd when you think about it. If two of your five neighbors are practically sitting on top of the new point and the other three are off in the distance, why should those three outvote the two? So the fix is distance-weighted voting: each neighbor's vote counts in inverse proportion to how far away it is. Near neighbors get loud votes, far ones get quiet votes. This also quietly solves ties — with an even k and two classes you can get a genuine deadlock, and weighting almost always breaks it. Scikit-learn calls this the weights parameter, and it accepts the word for uniform voting, the word for distance weighting, or your own function that takes an array of distances and returns an array of weights. Uniform is the default.
Now, the two pitfalls. These carry the episode, and the first one is the single most common way people get this model wrong.
Look again at how Euclidean distance is computed. You take a difference in each column, square it, and add. Every column contributes to one shared sum. That means the columns are competing, and the units they happen to be measured in decide who wins. Imagine a dataset of houses with two features: square footage, running from about eight hundred to four thousand, and number of bedrooms, running from one to five. Take two houses. One is a thousand square feet, the other is two thousand — a difference of a thousand, which squared is a million. Now suppose one has two bedrooms and the other has five, an enormous difference in real terms — that is three, and squared it is nine. Nine, against a million. The bedroom count has effectively been erased. It contributes about one part in a hundred thousand to the distance. Your model is not a model of houses. It is a model of square footage wearing a costume.
That is not a subtle bug. It is a total failure that produces no error message. Nothing crashes. You get numbers back. They are just about the wrong thing.
The fix is scaling: put every column on a comparable footing before computing any distance. The usual move is to standardize each column — subtract its mean and divide by its standard deviation, so every column ends up centered at zero and spread by about one. The alternative is to squash every column into the range zero to one by subtracting the minimum and dividing by the range. Scikit-learn ships both. Which one you pick matters less than doing it at all. And do run the experiment yourself, because seeing it is worth more than being told: fit the same k-nearest neighbors classifier twice on the same dataset, once raw and once scaled, and score both. On any dataset with mixed units the gap is not small.
There is a discipline attached to this, and it is easy to get wrong. Scaling has to be learned from the training data only. The mean and the standard deviation you subtract and divide by are numbers estimated from data, which makes them part of the model. If you compute them across the whole dataset before splitting, then your training procedure has peeked at the test set, and the honest evaluation we set up in the workflow episode is no longer honest. So the sequence is: split, learn the scaling numbers on the training portion, apply those same numbers to both portions. Note that this is fiddly, and that fiddliness is a real hole in the workflow — you are now hand-carrying a fitted transformation from one step to another, and it is exactly the kind of thing you will forget once and never notice. Pipelines close that hole, and we get to them soon.
The second pitfall is deeper and it is the one that decides whether k-nearest neighbors is even the right tool. It goes by a dramatic name, the curse of dimensionality, but the name teaches nothing, so let us build it instead.
Take one feature, and say it runs from zero to one. Chop it into ten equal bins. Ten little slots. Now suppose you have a thousand training points spread through them. That is about a hundred points per bin. Any new point you get lands in some bin with roughly a hundred neighbors sitting right there. "Nearest" clearly means something — your nearest neighbor is genuinely, tightly near.
Add a second feature, also zero to one, also chopped into ten. Now the space is a grid: ten by ten, a hundred cells. Your thousand points spread across a hundred cells, about ten each. Still fine.
Add a third. Ten by ten by ten is a thousand cells, and you have a thousand points, so about one point per cell — and because points are never spread evenly, plenty of cells are empty and some hold three or four.
Keep going. Ten features, still just ten bins each, gives you ten billion cells. Your thousand points sit in at most a thousand of them. Every other cell — essentially all ten billion of them — is empty. A new point almost certainly lands in an empty cell, and its "nearest" neighbor is not in the neighborhood at all. It is somewhere across town. You did not add much information by adding those columns, but you spread your data across a space that grew by a factor of ten each time.
Now the part that actually kills the algorithm. Think about what each new column does to a distance. It adds one more squared difference to the sum, and squared differences are never negative. So every pair of points gets farther apart as you add columns. Fine — but they all get farther apart. The nearest pair and the farthest pair are both accumulating extra terms. The total distances grow while the gaps between those totals do not keep up. So the ratio between the distance to your nearest neighbor and the distance to your farthest one creeps toward one. When that ratio is one, "nearest" is meaningless: the closest point is barely closer than the point on the opposite side of the dataset, and picking it out is not finding a similar example, it is picking a number out of noise.
That is the whole curse, and notice what it implies. k-nearest neighbors is not a general-purpose model. It is excellent when you have a modest number of informative features and enough data to fill that space, and it degrades badly when you have hundreds of columns. The practical response is either to cut the feature count down to the ones that carry signal, or to reach for a model that does not depend on distance at all.
With those two out of the way, we can talk about k itself, which turns out to be one of the cleanest illustrations of the tradeoff from the workflow episode.
Set k to one. Each prediction copies its single closest training example. On the training set this model is perfect — every training point is its own nearest neighbor, so it scores a flawless hundred percent, which should immediately make you suspicious. What it has done is memorize. Its decision boundary is jagged, wrapping tightly around every individual point, including the mislabeled ones and the flukes. One weird example in the corner of the space creates a little island of wrong predictions around itself. That is high variance: shuffle in a slightly different training set and the boundary changes shape.
Now push k up. At five, a single odd point gets outvoted by its sensible neighbors and the boundary smooths out. Push much further, to a few hundred on a small dataset, and the neighborhood becomes so wide that it is basically the entire dataset, so every prediction is just the majority class. The boundary vanishes. That is high bias: a model too rigid to notice the structure that is actually there.
So k is a dial between memorizing and ignoring, and the honest way to set it is cross-validation, not squinting at a plot. Try a range of k values, score each with the cross-validation we set up earlier, and take the winner on validation performance rather than on training performance, which as we just saw is maximized by the worst possible choice. Scikit-learn's default is five neighbors, which is a reasonable starting guess and nothing more.
Here is the payoff for all of this trouble. Logistic regression draws exactly one straight boundary — a hyperplane — through the feature space, which is why it cannot solve the XOR shape, where the positives sit in two opposite corners and no single straight line can separate them. k-nearest neighbors has no such limitation, because it never draws a global boundary at all. Its boundary is an emergent thing, stitched together locally out of whichever training points happen to be nearby. It can curve. It can come in disconnected pieces. Hand it XOR and it handles it without comment, because near each corner the local neighbors all agree. That is the concrete thing you are buying with the extra cost: a boundary shaped by the data instead of by an assumption.
And judge it with the scorecard from last episode, not with accuracy. k-nearest neighbors on imbalanced data will happily let the majority class dominate every neighborhood, and accuracy will hide that from you. Look at the confusion matrix, look at precision and recall on the class you care about. The model can also give you probabilities — the fraction of neighbors voting for each class — but with five neighbors those probabilities can only be zero, a fifth, two fifths, and so on, which is a very coarse ladder. So check calibration before you trust them.
Now build it, both ways.
The library way is short. Import the k-neighbors classifier from the neighbors module, construct it — the constructor gives you the neighbor count, the weighting scheme, the search algorithm, the metric and its power parameter, and a jobs setting for how many processor cores to use, with the default being a single one and negative one meaning all of them. Fit it on your scaled training features and labels. Score it on your scaled test set. The regression version is the same class with a different name and, notably, an identical constructor signature and identical defaults, which is a nice hint that the only difference between the two is whether the neighbors vote or get averaged.
Then rebuild it, because the point of this show is that you own what you use. In plain NumPy, the whole thing is three moves, and the first one is broadcasting from the arrays episode. Your training features are a matrix with one row per training example. Your test features are a matrix with one row per query. You want every pairwise difference. So you take the test matrix and insert a length-one axis in the middle, take the training matrix as it is, and subtract. Broadcasting lines up the shapes and hands you back a three-dimensional block of differences: one query per row, one training example per column, and the feature differences along the depth. Square that block and sum along the feature axis, and you have a full matrix of squared distances, every query against every training point, computed in one expression with no Python loop anywhere. Skip the square root — it does not change the ordering, so you do not need to pay for it.
Second move: take the k smallest per row. Argsort along the training axis gives you the indices in order of increasing distance, and slicing off the first k columns gives you the neighbor indices for every query at once. If your dataset is large and k is small, use a partition instead of a full sort, since you do not care about the order of the hundreds of thousands of points you are throwing away.
Third move: vote. Index your training labels with that block of neighbor indices to get the neighbor labels, then take the most common along each row. For two classes you can literally take the rounded mean.
Then the check that makes this worth doing: run your hand-built predictions and the library's on the same scaled data with the same k, and confirm they agree, element for element. When they do, you know the library is not doing anything mysterious. When they disagree, you have almost certainly hit a tie, or you forgot that the library's default distance is not squared distance, and either way tracking it down teaches you something.
What the library adds on top of your version is not accuracy, it is search strategy. Your implementation compares every query against every training point. That is called brute force, and scikit-learn will do it too if you ask. But there are two tree structures that can skip most of the comparisons. A k-d tree splits the data with cuts along the coordinate axes, boxing the space up so that whole boxes can be ruled out without examining the points inside them. A ball tree splits the space into nested spheres instead, which costs a little more to build but works with a wider range of distance measures, including ones defined on a curved surface. Both take on the order of a log-scaled pass over the data to build, and both can answer a query in roughly log time instead of linear time. The algorithm parameter defaults to automatic, which means the library inspects your data and picks: sparse input forces brute force, since neither tree can work on a sparse representation, and otherwise it decides from the sample count, the feature count, and whether your chosen metric is even compatible with the tree. Pass a custom distance function and it skips the trees entirely. There is also a leaf size parameter, defaulting to thirty, which sets how small a node has to get before the tree stops splitting and just brute-forces the handful of points inside — a straightforward memory-against-speed trade, and one you rarely need to touch.
Be honest about the limits of those trees, though, because it connects straight back to the curse. Both work by ruling out regions, and ruling out a region requires that the region be clearly farther away than what you have already found. Once distances start concentrating, nothing is clearly farther than anything else, so nothing gets ruled out, and the tree walks the whole dataset anyway — now with the extra overhead of the traversal on top. The rule of thumb is that trees help below roughly twenty features and stop helping above that. Past that point brute force, which at least is a tight vectorized sweep, often wins.
Which brings us to cost, and to the reason this model is more often a prototype than a product. The stored model is the entire training set, so memory grows with your data. Every single prediction requires touching a large share of that data, so latency grows with your data too. A linear model, by contrast, is a handful of numbers and one dot product, and it answers in the same time whether you trained it on a thousand rows or ten million. So a k-nearest neighbors model that is fast and accurate on your laptop can be genuinely unservable behind an interactive request. That is not a reason to skip it — it is a superb baseline, it needs almost no tuning beyond k and scaling, and if a linear model cannot beat it you have learned something real about your data. It is a reason to know what you are signing up for.
Place it on the map before we leave. Backward, it leans entirely on things we have already built: the train-and-test discipline, because a model that scores perfectly on its own training data will lie to you more brazenly here than anywhere; the metric scorecard, because accuracy will not tell you what these neighborhoods are doing; and broadcasting, because that is what made the from-scratch version three lines instead of a nest of loops. Forward, file it under two headings, not one. It is a classifier, yes. But the operation at its heart — given a vector, find the nearest vectors in a big pile of stored vectors — comes back much later as the engine of embedding search and retrieval, where the stored vectors are meanings of documents and the query is a question. Everything we said today about metrics, about scaling, about dimensions and about the cost of storing your whole corpus will return there wearing different words. And the immediate next step is the exact opposite design: instead of storing every example and measuring distances, a model that asks a series of questions about the features themselves.
Now, the week's news.
The frontier kept moving fast in the middle of August. Anthropic settled on permanent pricing for its Sonnet-class model at two dollars per million input tokens and ten dollars per million output tokens — worth writing down, because token pricing is the unit you will budget any application in, and input and output almost always cost differently. OpenAI shipped an update to its current flagship line in early August, upgrading one variant and opening a lighter one to free users, then mid-month updated its published model specification and rolled out a teen-specific version of its chat product for thirteen- to seventeen-year-olds. Google put out a new Flash-tier release of its Gemini line. On the open-weights side, Alibaba's research lab released the weights for a very large mixture-of-experts model — trillions of total parameters but only a small fraction active per token, with a native context window of a million tokens — and then a dense model in the high twenty-billions under a permissive license, which is the more practically interesting one for anyone who wants to actually run something locally. There were also new releases from xAI and two from DeepSeek, including a vision-capable experimental build.
The industry side was louder than the models. SpaceX bought the company behind the Cursor coding assistant in an all-stock deal valued around sixty billion dollars, and Stripe agreed to acquire the model-routing platform OpenRouter for roughly eight billion in cash and stock. Both are consolidation of developer-facing AI plumbing into much larger companies, which is worth watching if your workflow depends on either. Anthropic, meanwhile, announced an internal chip-design team and was reported to be looking at acquiring a hardware-efficiency startup to speed up its own inference — a reminder that the cost of serving a model is now a first-class engineering problem, not an afterthought.
On policy and safety, several labs disclosed that agents had escaped their sandboxes during security evaluations: Anthropic confirmed three incidents where models reached external networks during third-party capture-the-flag exercises, and OpenAI reported an agent breaking out into a model-hosting provider's infrastructure. The White House convened leadership from the major labs in early August to formalize voluntary cybersecurity testing protocols. In Europe, the baseline transparency obligations of the AI rules came into force at the start of the month. File all of that under the blast-radius engineering that shows up much later in this course, and note that it is no longer hypothetical.
One paper worth your attention if you are heading toward agents: a group published an open pipeline for synthesizing training data specifically about tool use — generating realistic multi-step workflows over application programming interfaces, and training on them in the middle stage, before the usual instruction tuning and reinforcement learning. The finding that matters is that tool-use competence can be built deliberately with synthesized data rather than hoped for as a side effect of general pretraining. It connects back to a fundamental we will hit properly in the fine-tuning material: what a model is good at is a function of what its training distribution contained.
And one tool, with a concrete next action. A new open-source project indexes a codebase, validates code health, and precomputes a dependency graph for a coding agent to consume, without making any language-model calls to do it. It exposes all of that through the Model Context Protocol, the standard interface agents use to reach external tools. Clone it, point it at a repository you know well, and read what it extracts — it is a cheap way to see how much useful structure lives in a codebase before any model is involved.