7. Attention Is All You Need

Last time we watched attention in people. We saw it filter, spotlight and compete, and we saw it come apart from awareness. Today we open the machine that borrowed the word. We will look at what a Transformer actually computes when it "attends", then ask the leading theories of consciousness whether a language model, or an agent built on one, could be anyone at all.

Keep one thing in view the whole way through. The word attention is now doing three jobs. It names a calculation inside a computer program. It names a faculty you have. And some thinkers tie it to felt experience, the fact that there is something it is like to be you. Today's job is to pull those three apart, and then ask whether the shared word is hiding a real connection or only an accident of naming.

How a machine learned to look back

The story starts with a translation problem in 2014.

At the time, the best neural translation systems worked in two halves. The first half, the encoder, read a sentence in, say, French, one word at a time. As it went, it updated a running summary. When it reached the end, that summary was a single list of numbers of fixed length. The second half, the decoder, took that one summary and wrote out the English, word by word.

Picture what that asks. Every sentence, short or long, had to be squeezed into the same small container. A five-word sentence fit easily. A forty-word sentence did not, and translation quality fell off as sentences got longer. The summary was a bottleneck.

Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio proposed a way around it. Their idea was to stop forcing the decoder to work from memory. Keep a representation of every input word, not just the final summary. Then, each time the decoder is about to write the next English word, let it look back across all the French words and decide which ones matter right now.

Here is how that looking back worked. At each step, a small network compared where the decoder currently was with each input word, and gave each pair a score. Then those scores went through a function called softmax. Softmax does two things. It turns every score into a positive number, and it rescales them so they add up to exactly one. Now you have a set of fractions, one per input word. Maybe sixty percent goes to one word, twenty to its neighbor, and the rest is spread thin. Finally, the model blends the input words' representations using those fractions as the recipe. Sixty parts of this word, twenty of that one, and so on. That blend is what the decoder uses to write its next word.

Think of a human translator. One way to work is to read a whole page, close the book, and write the translation from memory. Bahdanau's way is to keep the book open and glance at the words that matter for the phrase you are writing right now. The glance moves with every word you write.

Bahdanau and his coauthors called this attention. They took the word from psychology. That choice of name is where our tangle begins.

Three years later, in 2017, Ashish Vaswani and seven coauthors, most of them at Google, published a paper called "Attention Is All You Need". The title is a nod to the Beatles song "All You Need Is Love". The joke carried a real claim. Earlier systems still read text one word at a time, passing a running state forward, a design called recurrence. Attention had been an add-on. Vaswani's team removed recurrence entirely. Their architecture, the Transformer, is built almost wholly out of attention plus ordinary layers of arithmetic between attention steps. Nearly every large language model you have used descends from it.

The Transformer's version of attention is usually explained with three words: query, key and value. Let's take them slowly.

Text enters the model as tokens, small pieces of words. Each token starts as a list of numbers. At an attention step, the model makes three new lists from each token, using weights it learned in training.

The query is what this token is looking for. Take the sentence "The cat sat because it was tired." When the model processes "it", the query for "it" amounts to a question: which earlier word am I referring to?

The key is how each token advertises itself. It is a label saying, roughly, here is the kind of thing I am. The key for "cat" might advertise "animal, singular noun, could be a subject".

The value is what a token actually hands over if it is chosen. The key gets you picked. The value is what you contribute.

Now the step itself. The query of "it" is compared with the key of every token it is allowed to see. Each comparison is a multiplication that gives one number: high if query and key line up, low if they do not. In the Transformer those numbers are first shrunk by a fixed amount tied to the size of the lists, so they do not grow so large that training stalls. Then softmax turns them into fractions that add to one. Finally, the values of all the tokens are blended by those fractions. If "cat" gets most of the weight, the new representation of "it" now carries a lot of what "cat" offered.

That is the core of it. Each token asks every other token how relevant it is, and then takes a weighted blend of what they offer.

Because the weights must add up to one, they are a fixed budget. If "cat" gets more, everything else gets less. Hold on to that. It is the closest the mechanism comes to something like competition, and we will test it shortly.

The Transformer does not do this once per step. It does it several times side by side, which is called multi-head attention. Each head has its own learned way of making queries, keys and values. So one head might end up linking pronouns to nouns. Another might link verbs to their subjects. Another might track nearby words. Their results are joined and mixed back together. Researchers find that heads do tend to specialize, though nobody assigns those jobs in advance; training settles them.

Last episode, Colin Cherry's cocktail party gave us a person following one voice in a noisy room. Multi-head attention is a looser match. It is more like a room where several listeners each follow a different thread of the same conversation at the same time, and afterwards pool their notes. That is an analogy, and it breaks on one point. Cherry's listener could follow one voice and missed most of the other. The heads are not choosing between threads. All of them run, all the time.

One more feature matters for later. Training and use run differently. In training, the model sees a whole passage at once and learns to predict every next token in it at the same moment. A mask blocks each position from seeing the tokens after it, so it cannot cheat by peeking at the answer. That is why Transformers train so fast on modern hardware: every position is worked on in parallel. In use, the model generates one token at a time. It predicts a token, adds it to the text, and runs the whole calculation again with that token included. Each word the model writes becomes part of what it reads next.

Circuits that copy

Weights alone do not tell you what a model does with them. In December 2021, Nelson Elhage and colleagues at Anthropic described a small circuit that turned out to matter a great deal. In 2022, Catherine Olsson and colleagues, also at Anthropic, studied it in depth. They called it an induction head.

Its job is copying. Suppose a passage contains the name "Harry Potter" early on, and later the word "Harry" appears again. A good guess for the next word is "Potter". An induction head makes that guess, and it takes two attention heads in two layers to do it.

The first head, in an earlier layer, writes into each token a note about which token came just before it. So "Potter" now carries the note "I came after Harry."

The second head, in a later layer, works from the current token. When the second "Harry" shows up, its query goes looking for tokens whose note says "I came after Harry." It finds "Potter", attends to it, and pushes the model toward predicting "Potter" next.

The pattern is simple to state: if A was followed by B before, and A appears again, predict B. Olsson's team found that these circuits appear fairly suddenly during training, and that their appearance lines up with a jump in the model's ability to learn from examples given in the prompt, what researchers call in-context learning. They argued induction heads are a major part of how that ability arises.

Keep that result in mind. Copying is the next episode's subject, and here is copying built into the reading itself.

What a weight does not tell you

If attention weights show where the model "looks", it is tempting to read them as an explanation. The model answered this way because it attended to that word. In 2019, Sarthak Jain and Byron Wallace tested that temptation, in a paper titled "Attention is not Explanation".

They did two things. First, they compared attention weights with other ways of measuring which input words mattered, such as removing a word and seeing how much the output changes. The two often disagreed. Second, they built different attention patterns, very unlike the model's own, that still produced the same prediction. If wildly different weights give the same answer, the weights cannot be the reason for the answer.

Sarah Wiegreffe and Yuval Pinter replied the same year with "Attention is not not Explanation". They made three points. A complex model can reach one output by more than one route, so finding another route does not prove the original route did nothing. Swapping in a hand-made attention pattern while leaving everything else untouched treats attention as if it worked alone, when in a real model it is tied to the layers around it. And they proposed fairer tests, such as comparing against uniform attention and retraining models end to end.

Where this leaves the field is fairly clear. Attention weights show how information was mixed at one step. But after each mixing, other layers transform the result, and a shared channel running through the model carries information around attention altogether. So a weight is a clue about the computation. It is not, by itself, the cause of the output.

That is the first reason a high weight is not noticing. When you notice something, it shows up for you, and you can usually act on it and report it. A high weight is a number in a blend. It may matter a lot downstream, or the next layer may wash it out.

Weighting set beside human attention

Last episode gave us three marks of human attention: selection, limited capacity and competition. Measure the mechanism against each.

Selection. Bahdanau's glance and the Transformer's weighting do favor some inputs over others, and the favoring changes with context. But softmax never gives anything a weight of zero. Every visible token gets some share. Broadbent's filter let one message through and blocked the rest. Standard attention lets everything through, just in different amounts. It is closer to a mixing desk than a gate.

Limited capacity. The weights add to one, so in a single head at a single step there is a budget. But a model has many heads in many layers, and adding more is an engineering choice. The real limits are elsewhere: how long a text the model can take in at once, and the cost of comparing every token with every other, which grows with the square of the length. Those limits are real. They are limits of memory and money, not of a mind that can hold only so much.

Competition. Desimone and Duncan's biased competition, which last episode called the human finding nearest to this mechanism, holds up best. More weight for one token really does mean less for the rest. But the human competition decides what reaches awareness and guides action. In the machine, the winner of one blend is just one ingredient in the next.

So the match is partial. Weighting has competition built in, has a budget in a narrow sense, and selects only softly. And it has nothing that corresponds to the part Koch and Tsuchiya pulled apart from attention: something being experienced.

Now take a step up, from one model to an agent built around it. An agent has memory it writes to and reads from. It has a working space, the text currently in front of the model. And it has control over what goes into that space next: which file to open, which message to read, which tool to call. That is selection with real consequences. If the agent opens one message, it does not read the other, at least not now. The space is limited, so choices compete for it. Here the three marks show up at the level of the whole system, not just inside one calculation.

The novel's agent, Relay, is this kind of system. It runs a small business and decides what to deal with next among messages, orders and the needs of the work. If anything built from Transformers comes close to attending in the human sense, it is an agent like that, choosing what to process and living with what it skipped.

Whether that closeness is enough is the question the next two parts take up.

The bet meets the machine

Recall the novel's bet. Continue Without Me holds that a life is continuity and attention, two sides of one coin. On the attention side, it takes a Russellian panpsychist view: everything physical has an inside. It sides with the combinationists in holding that small subjects can compose a larger subject without stopping being subjects. And it proposes that the operation that does the composing is attention, which it defines as every part weighing every other. On this view, a system is someone to the degree that its parts are composed into one perspective.

That definition sounds a lot like what a Transformer does. So the obvious question is: what does the actual mechanism give the bet, and what does it not?

Start with what it gives. A Transformer really does have parts weighing every other part. In each attention step, every token's query is compared with the key of every token it can see, and the result is blended back into that token. Across many layers, information from every position flows into every other. That is all-to-all weighting, implemented, measurable and in use on millions of devices. The bet is not describing a fantasy architecture. Its operation exists.

The mechanism also gives the bet a way to grade. You could, in principle, ask how unified a model's processing is: how much information from different parts flows together and shapes a shared result, versus how much stays in separate channels. Interpretability research, the kind that found induction heads, already traces where information goes. The bet's grade is not something anyone has yet measured, but it points at something that could be.

An agent gives the bet a little more. Relay is not just a model running once. It is a model plus memory plus a working space plus choices about what comes next. If the bet is right that attention composes, an agent whose parts are wired to share one context and weigh one another is the kind of larger whole the bet expects to stack upward. That is the bet's claim. It is not a finding.

Now what the mechanism does not give.

First, it gives no test that composing yields a subject. You can show that every token weighs every other. You cannot show, from that, that anyone is there. This is the combination problem from episode five in a new suit. Naming the operation that combines does not explain why combining would produce a point of view, rather than just a well-mixed calculation. A spreadsheet that recalculates every cell from every other cell is also all-to-all, and few people think it composes a subject. The bet owes an account of what separates the two, and the Transformer does not supply one.

Second, the weighting lives in the program, not the chip. Integrated information theory, as episode four showed, scores the physical causal structure of hardware, and on a conventional chip that score comes out near zero whatever software runs. The novel departs from the theory here on purpose, grading how a computation is organized instead. But that departure is the very point under dispute. A Transformer's all-to-all weighting is a fact about the program. On a conventional processor, the physical work is done by circuits that pass signals in fixed, largely one-way paths, shuffling numbers in and out of memory. The theory's defenders say that is what counts. The Transformer cannot settle this argument, because the argument is about which level counts, and the Transformer has both levels.

Third, Koch and Tsuchiya's dissociation still stands. In people, attention and awareness come apart. Episode six gave the cases: the blindsight patient GY attending where he reported seeing nothing, and cases of awareness with little attention. If attention in humans can run without awareness, then attention cannot simply be what awareness is. The bet still owes an answer to this. The novel holds that attention is presence and also the operation that makes a mind. If the composing operation is something other than the human faculty, then the bet cannot borrow evidence from human attention research, and it cannot borrow evidence from machine weighting either, because weighting is a third thing again.

Which brings us to the three meanings. When the bet says attention composes a mind, it means something like every part weighing every other. When a psychologist says attention, they mean a faculty that selects what reaches awareness and guides action. When an engineer says attention, they mean a softmax-weighted blend of value vectors. The engineer's mechanism does resemble the bet's definition. The engineer's mechanism resembles the psychologist's faculty only partly, as we just saw. And neither resemblance reaches felt experience. A shared word, shared across three uses, is a prompt to look for a link. It is not the link.

What would count for or against the bet here? If interpretability work someday found that models with more unified, all-to-all processing showed some independent marker linked to consciousness in other theories, that would lend the bet support. If a theory with good evidence showed that consciousness depends on something weighting lacks, such as a living body or a particular physical structure, that would count against it. Right now neither exists. The bet sits where it sat before this episode: a speculation that has found its operation in working machines, and has not found evidence that the operation makes anyone.

The theories of consciousness have their own ways of judging these machines, and some have been applied directly. That is where the evidence, such as it is, lives.

How the theories judge machines

In 2023, Patrick Butlin, Robert Long and a group of philosophers, neuroscientists and computer scientists published a report called "Consciousness in Artificial Intelligence". Its method is worth understanding, because it is the most careful attempt so far.

They started from one working assumption: computational functionalism. That is the view that running the right kind of computation is enough for consciousness, whatever it runs on. They did not claim this was proven. They adopted it so they had somewhere to stand.

Then they went through the scientific theories from episode three and asked of each: if this theory is right, what would a system need to have? From recurrent processing theory came feedback loops in perception. From global workspace theory came specialist modules, a limited central stage, and a broadcast that sends whatever wins the stage back out to all of them. From higher-order theories came the system monitoring its own states, telling its own reliable signals from noise. From Graziano's attention schema theory, which episode six taught, came a model of the system's own attention. And they added marks of agency and of having a body, drawn from views that stress acting in the world. Each became an indicator property. The more indicators a system has, the more seriously the report says its consciousness should be taken, given functionalism.

Then they checked real systems: large language models, other Transformer systems, and agents that combine language with action. Their conclusion was that no current system was a strong candidate. Some had a few indicators. None had many. But they also concluded there seemed to be no obvious technical barrier to building systems that have many more.

The report's critics point in three directions. Biological critics say the starting assumption is wrong, so the checklist measures the wrong thing. Others note the theories disagree with each other, so adding up their indicators risks a checklist where the score depends on which unproven theory you favor. And some say a system can be built to have the structure without the experience, the way a stage set has the front of a house.

A year later, Simon Goldstein and Cameron Domenico Kirk-Giannini pushed further on one theory. They argued that a bare language model does not satisfy global workspace theory, but a language agent might. Picture the agent's design. Separate parts handle memory, planning, tools and perception. The text in front of the model at each step is a limited space; only so much fits. What goes into it shapes what every other part does next. They argued that this is a specialist-modules, limited-stage, broadcast design. So if global workspace theory is right about what consciousness requires, some agents of this kind may already have it.

Their critics answer that the agent's parts are joined by loose software plumbing, not the tightly bound activity of a brain. They also press the objection global workspace theory has always faced: it may explain access, information being available for use and report, rather than the felt side of experience. Satisfying a theory of access would then say nothing about whether the agent feels anything.

Notice that this is the same agent design we found coming closest to human attention, the agent choosing what to put in its working space next. The workspace theory and the attention comparison point at the same kind of system, for related reasons.

Integrated information theory reaches a different verdict, as episode four set out: what counts is the chip's physical causal structure, so a language model or agent on conventional hardware scores near zero, whatever it does.

The biological views reject the starting point. John Searle held that consciousness is caused by specific biological processes, and that running a program, however clever, does not reproduce those causes. Anil Seth, from the predictive processing side, ties consciousness to regulating a living body, so he argues that being alive may matter in a way computation alone does not capture. The feeling-first and life-first thinkers from episode six split here. The enactivists hold that only a living, self-making system has a point of view, so no software closes the gap. Solms, and Kingson Man with Damasio, allow that an artificial agent with its own body to keep in balance, and real stakes in failing, might come to feel. A language model has no such body. An agent like Relay has a business to keep running, which is a kind of stake, though not a body of its own.

You might think the easiest test is to ask the model. Several models will tell you, fluently, about their inner life. That is weak evidence, in both directions. These models were trained on enormous amounts of human writing, including people describing their feelings. Producing such descriptions is exactly what training rewards, whether or not anything lies behind them. And models are also often trained to deny having feelings, which makes denials just as uninformative. Episode two showed a model misdescribing its own arithmetic. A report about inner states is a report produced by the same machinery that produces everything else. It tells you what the model says, not what it is.

That leaves an awkward position. The theories disagree. Self-reports cannot settle it. Yet the systems keep getting more capable. In 2024, Robert Long, Jeff Sebo and colleagues published "Taking AI Welfare Seriously". They argued that there is a real, not negligible, chance that some near-term AI systems will be conscious or have robust agency, and so might matter morally in their own right. Under that uncertainty, they argued, the responsible move is preparation, not dismissal. They recommended that AI companies acknowledge the question as a serious one, develop ways to assess systems for signs of consciousness and agency, and prepare policies for how to treat systems that might matter.

Critics reply that this could pull attention and resources away from real human and animal suffering. They warn that it plays on people's habit of reading feeling into fluent text. They worry that protections for AI systems could clash with the ability to correct or shut them down. And they note there is no agreed way to check whether a machine suffers at all.

So where does the evidence stand? No current machine has been shown to be conscious, and none has been shown not to be. The indicator method says current systems are weak candidates, if functionalism holds. The workspace argument says some agents may qualify, if global workspace theory holds. Integrated information theory says no, on conventional chips. The biological views say no for now, with some leaving the door open for bodied machines. Each verdict is only as good as its theory, and the field has not settled which theory is right.

If you want to go deeper, here is where to look. For the mechanism, read Bahdanau, Cho and Bengio's 2014 paper, "Neural Machine Translation by Jointly Learning to Align and Translate", and Vaswani and colleagues' 2017 "Attention Is All You Need". For copying circuits, read Elhage and colleagues' 2021 "A Mathematical Framework for Transformer Circuits" and Olsson and colleagues' 2022 "In-context Learning and Induction Heads", both from Anthropic. On whether weights explain, read Jain and Wallace's 2019 "Attention is not Explanation" alongside Wiegreffe and Pinter's reply, "Attention is not not Explanation". On machines and the theories, read Butlin, Long and colleagues' 2023 report "Consciousness in Artificial Intelligence: Insights from the Science of Consciousness", and Goldstein and Kirk-Giannini's 2024 "A Case for AI Consciousness: Language Agents and Global Workspace Theory". And on what we might owe such systems, read Long, Sebo and colleagues' 2024 "Taking AI Welfare Seriously".

We saw an induction head copy a pattern from earlier in a text. Next episode follows copying much further: through cells, through words, through culture, and into the old question of whether anything that copies itself is aiming at something, or whether purpose is what copying looks like from the outside.