Neuro 13
0 / 37 chapters finished

Chapter 22: Attention: a fuzzy lookup

Watch a network learn where to look.

The model you trained in chapter 21 had one hard limit: it could only see the last three letters. Anything before that was gone. For inventing names, three letters is enough. For reading a sentence it is hopeless, and it only takes one sentence to see why.

Why a fixed window is not enough

Read this: The trophy didn't fit in the suitcase because it was too big.

What was too big? The trophy. You knew without thinking. Now swap the last word and read it again.

▸ try it

Who is "it"?

Switch the ending. The word it is asking a question: what am I standing in for? The dark orange word is where the answer is.

…because it was too big
…because it was too small

the word asking the answer the other candidate the clue

This picture is drawn by hand to show the idea. It is not measured from a real model.

Notice what you did. Standing at it, you looked back across the sentence, found the two things it could be, and used a clue at the far end to choose between them. You skipped straight over in, the and because, even though they were closer. You didn't look at the nearest words. You looked at the useful ones, wherever they were.

A model with a three-token window can't do that. By the time it gets to it, all it can see is "suitcase because it". The trophy slid out of view long before. You could widen the window to ten tokens, or a hundred, but every extra slot costs more weights, and the word you need can always be one place further back.

(Sentences like this one have a name: Winograd schemas. For years, researchers used them to test whether a computer was following the meaning of a sentence or only matching up nearby words.)

What we want is a model where any token can look at any other token and decide for itself which ones matter. That idea is called attention: each token gathers information from the others, taking a lot from the useful ones and almost nothing from the rest. It is the central idea inside every modern chatbot, and you already own all the parts.

A question, a label and a payload

In attention, every token carries three lists of numbers. Each list has a job.

  • The query is the token's question: here is what I am looking for.
  • The key is the token's label: here is what I have to offer.
  • The value is the token's payload: here is what you get if you listen to me.

The token that is looking holds its query up against every other token's key and gets a match score for each one. Then it collects the values, taking the most from the tokens with the highest scores.

How do you score a match between two lists of numbers? With a dot product: multiply the first numbers together, multiply the second numbers together, keep going to the end of the lists, then add up the results. You have been doing this since chapter 3. It is exactly what a neuron does with its inputs and its weights.

When a list has just two numbers you can draw it as an arrow: the first number says how far right, the second says how far up. For arrows, the dot product means something you can see.

  • Arrows pointing the same way give a big positive score.
  • Arrows at a right angle give zero.
  • Arrows pointing opposite ways give a negative score.

Last step. The scores go through softmax, the function from chapter 10 that turns any list of scores into shares that add up to 1. Here the shares are called attention weights. The output is a weighted blend of the values: multiply each value by its share, then add them all together.

That is the whole operation. Score, softmax, blend. Try it by hand before we let a network learn it.

In the widget below, the red token's key is the arrow (2, 1). You aim a short query arrow the same way, then stretch it to three times the length without turning it. What happens to red's share of the attention?

A longer query multiplies every score by the same amount, and softmax cares about the gaps between the scores. Bigger gaps mean a more lopsided split. In the widget, a query of (0.4, 0.2) gives red 0.65 of the attention. Stretch it to (1.2, 0.6) and red gets 0.95.

▸ try it

Aim the query

Three tokens. Each one has a key, drawn as a colored arrow, and a value, which is a color. A color makes a handy value because it is a list of three numbers (how much red, green and blue light to mix), and you can see what a blend of colors looks like.

The black arrow is the query. Drag its round handle, or use the sliders. Watch the three scores, the three shares and the blended color.

the query's two numbers
1.0
1.0
score = key · query, then its share after softmax
the blend: each color times its share, added up

Four things to take away from that:

  • Direction picks the winner. The token whose key points most nearly the same way as the query gets the biggest score.
  • Length sets how sharp the choice is. A short query gives a mushy blend of everything. A long one hands nearly all the attention to a single token.
  • Zero means flat. With the handle at the center every score is 0, softmax hands out equal shares of 0.33, and the blend is a muddy brown. Remember that picture. The next model starts there.
  • Nobody ever gets exactly nothing. Softmax never produces a true zero. Even when the screen says 0.00 there is a sliver left. Attention is a soft choice, not a switch.

The locker room: a network that learns where to look

In that widget you aimed the query yourself, and we made up the keys. In a real network nobody aims anything. The queries, keys and values all come out of weights, and the weights are learned. Let's watch that happen.

The task is a locker room. Each example is a short row of lockers, each with an owner and one item inside, followed by a question that names an owner. The right answer is that owner's item.

sam:hat   mia:ball   leo:book   ? mia   →   ball

There are 8 owners and 8 items, shuffled for every example, so nothing can be learned by heart: mia has a ball here and a kite next time. The only way to be right is to find mia's locker and read what's in it. That is a lookup.

Here is the model, start to finish:

  1. Embeddings. Every owner and every item gets its own row of 16 learned numbers, like the embedding rows in chapter 20. A locker is its owner's row plus its item's row, added number by number. So each locker is one list of 16 numbers with who and what mixed together.
  2. Query. The row for the owner named in the question is multiplied by a grid of weights. Out comes the query.
  3. Keys and values. Each locker's 16 numbers are multiplied by a second grid of weights to make its key, and by a third grid to make its value.
  4. Attention. Dot product of the query with each key, softmax, then a weighted blend of the values. It is what you just did by hand, with 16 numbers in each arrow instead of 2.
  5. Answer. One last layer turns the blend into 8 scores, one for each item, and softmax turns those into probabilities.

That comes to 1,160 weights, and every one of them starts as a small random number. Training is what it has been all book: show examples in mini-batches of 16 (chapter 13), measure how wrong the answers are, and let backpropagation (chapter 14) work out a nudge for every weight. Nobody tells the model which locker to look at. It is only ever told the right answer.

The weights start as tiny random numbers. Before any training, how will the model spread its attention over three lockers?

Tiny weights make tiny queries and tiny keys, so every match score is close to zero. You saw what softmax does with equal scores: equal shares. The model starts out staring at every locker equally.

▸ try it

Train the locker room

Press train and watch the orange bars above the lockers. They show how much attention the question is paying to each one. The locker with the green outline holds the right answer.

The chart tracks 500 test examples that the model never trains on: how often its answer is right, and how much of its attention lands on the right locker. Training takes about 8 seconds. You can pause it at any point and poke at the example.

examples seen: 0 of 24,000 answers right: ? attention on the right locker: ? weights: ?

One example, live

attention on this locker the locker with the right answer
Question: what is in the locker of?

Every menu works. Change an owner, an item or the question and the model answers again at once.

Peek inside: which question matches which locker?

Each row is a question. Each square is a match score: the darker the orange, the higher the score. Gray is below zero and white is about zero. Hover over a square to read its number.

against the name on the door
against the item inside

What just happened

The run goes through three stages, in the same order every time.

  1. Guessing. At the start the model is right about 1 time in 8, which is what you'd get by picking an item out of a hat. Attention is flat.
  2. Stuck near one in three. Accuracy creeps up to around 30% and stalls, and the orange line doesn't move at all. The model has worked out that the answer is always one of the items in the room, which beats guessing from all 8. But with flat attention the blend is an even mix of three lockers, and nothing in it says whose item is whose. One in three is the best anybody could do without looking.
  3. The click. Somewhere between about 13,000 and 18,000 examples (it varies from run to run) the orange line lifts off, the bars pile onto the green locker, and accuracy shoots up to 100%. The model has learned where to look.

We ran this exact training 200 times outside the browser. Every run finished at 100% on its 500 test examples, with between 94% and 98% of the attention on the right locker.

Why the long stall? The nudges that shape the query and key weights are tiny while those weights are tiny, and they only start to mean something once the rest of the model can read an item out of the blend. So for a while nothing seems to happen. Then a little extra attention on the right locker makes the answer a little better, which makes the next nudge stronger, which sharpens the attention some more. It snowballs.

Now look at the two grids under "peek inside". A locker is its owner's row plus its item's row, so its key has two parts: one that comes from the owner and one that comes from the item. The left grid scores each question against the owner part. Before training it is blank. After training, the darkest orange square in every row sits on the diagonal: mia's query matches mia's key best, sam's matches sam's. Nobody programmed that pairing. The weights found it.

The right grid scores the same questions against the item part, and it stays almost blank. The model has learned that what is inside a locker is no help in deciding where to look, so the keys carry the owner and all but ignore the item. The values do the opposite job. We checked the trained weights of 40 models: in every one, the item's part of the value decided the answer and the owner's part barely counted. A label and a payload, pulled apart, just as the names promise.

Try to break it

The model has only ever seen rooms with exactly 3 lockers. You are about to hand it a room with 8. What will happen?

Nothing in the model counts lockers. Every locker gets its own score from the same weights, and softmax works on a list of any length. In our 200 test runs, every trained model scored 100% on rooms of 8. The price is focus: with more lockers competing, the share of attention on the right one typically fell from about 96% to about 87%.

Scroll back up to the widget (train it first if you started over) and try these:

  • More lockers. Press add a locker until there are 8, then press new example a few times.
  • Shuffle. Press shuffle the lockers. The bars travel with their lockers and the answer does not change at all. The model has no idea what order the lockers are in. Hold that thought.
  • Swap the item. Use the second menu on the green locker to put something else inside. The bars barely move, and the answer follows the new item.
  • Ask about someone who isn't there. In the question menu, pick a name marked no locker here. There is no right answer. The model gives one anyway, and it is usually very sure of itself. Softmax has to hand out shares that add up to 1, so attention has no way to say "nobody matches". It settles on whichever key is the least bad match and reports what is in that locker. Chapter 25 is about what happens when much bigger models answer confidently with nothing to go on.

Why "query", "key" and "value"?

The names are borrowed from databases. Programs store things in pairs: a key to look things up by, and a value that comes back. The contact list in a phone works this way (name in, number out). So does a Python dictionary:

lockers = {"sam": "hat", "mia": "ball", "leo": "book"}
lockers["mia"]   →   "ball"

The thing you ask with, "mia", is the query. An ordinary lookup like this is all or nothing. The query matches one key exactly and you get that key's value, whole. Ask for lockers["zoe"] and the program stops with an error.

Attention is the same lookup with soft edges. Every key matches every query a little, and what comes back is a blend. That is why this chapter calls it a fuzzy lookup.

The fuzziness is not sloppiness. It is what makes attention trainable. Backpropagation works by asking, "if I nudge this weight a hair, does the answer get a hair better?" With a blend, a small nudge always changes the mix a little, so there is always an answer. With an all-or-nothing lookup, a small nudge changes nothing at all, right up until the moment the choice jumps to a different locker. No slope, nothing to roll down. A soft lookup can start out flat, as ours did, and slide smoothly toward a sharp one. You just watched it happen.

Inside a real transformer

In 2017, eight researchers, most of them at Google, published a paper with a cheeky title: "Attention Is All You Need". It described a network built almost entirely out of the operation you just trained, and named it the transformer. Every big chatbot today is a descendant. Here is what a transformer adds to the locker room.

  • Every token asks at once. In the locker room only the question sent out a query. In a transformer every token does, all at the same moment, and each one gets back its own blend. (In a chatbot, a token may only look at the tokens before it. Its job is to predict what comes next, so peeking ahead would be cheating.)
  • Several heads. One set of query, key and value weights is called an attention head, and one head can only ask one kind of question. So each layer runs several heads side by side, each with its own weights, each free to look for something different. When researchers open up real models they do find some heads with a job you can name, like linking a word such as it back to the thing it stands for. Most heads are much harder to read.
  • Many layers. The blends that come out of one layer are the inputs to the next, so later layers can ask questions about what earlier layers found. The smallest version of GPT-2, from 2019, has 12 layers with 12 heads each: 144 heads. GPT-3, a year later, has 96 layers with 96 heads each: 9,216 heads.
  • A small ordinary network after every attention step. Attention only moves information around. So after each attention step, every token is passed through a small network of the kind you built in chapter 8, weights and a bias and a squish, to do some work on what it has gathered.

And one fix, for a problem you have already seen. A blend ignores order. You shuffled the lockers and nothing changed. For lockers that is fine. For language it would be a disaster, because dog bites man and man bites dog are the same three tokens. So before the first layer, a transformer adds a position signal to every token: a second list of numbers, added onto the token's embedding, that depends only on where the token sits in the text. First, second, third and so on each get a different list. Now a key can say "I am a noun, in fifth place", and a query from the seventh token can ask for "a noun, two places before me". The 2017 paper built its position lists out of wave patterns. Newer models mix position in other ways. The job is always the same: give attention a way to tell first from last.

Honesty note: we left out some plumbing. Real transformers also rescale the numbers between steps and add shortcut connections so that gradients survive dozens of layers, and they divide every match score by a fixed number before the softmax to keep the scores from growing too large. None of that changes the idea.

Chapter 23 puts the whole machine together: tokens in, embeddings, layer after layer of attention, and at the end a probability for every possible next token.

▸ run real python

Run the locker room in Python

This is attention.py: the same model, the same task and the same training as the widget, in one short numpy script. The function gradients() is backpropagation written out by hand. Look for four things in the output: the attention on one example before and after training, the two test scores stalling and then jumping together, how the model copes with rooms of other sizes, and what it says about someone who has no locker. Give it a little while. The progress lines arrive one at a time as it trains.

attention.py Python loads on first run
(click "run" to execute)

Attention is a lookup with soft edges. Each token asks a question (its query), every token wears a label (its key), a dot product scores each match, softmax turns the scores into shares, and what comes back is a blend of payloads (the values). All three are made by ordinary weights, learned by the same backpropagation as everything else.

Challenge

In the Python above, change LOCKERS = 3 to LOCKERS = 2, so the model only ever trains on rooms with two lockers. Before you run it, predict what the table at the end will say about rooms of 8. Then put it back to 3, change SIZE = 16 to SIZE = 8, and watch when the click comes. If it hasn't finished by the last step, give it STEPS = 3000.

And one with no code. Our model could never learn to answer "what is in the second locker?" Why not, and what would you have to add?

What decides how much one token listens to another?

The match score is the dot product of a query and a key. Distance plays no part unless a position signal has been mixed in, and the value is only what gets handed over once the shares are decided.

Two lockers end up with exactly the same match score, far above all the others. What does attention pass along?

Equal scores get equal shares from softmax, so the output is half of one value plus half of the other. Attention never has to pick just one. That is the "fuzzy" in fuzzy lookup.

After training, the question "mia?" puts nearly all of its attention on mia's locker. Where did that behavior come from?

The code never compares names and never says where to look. Queries and keys come out of weights that start random. Training only scores the final answer, and the nudges flow back through the blend to the weights that make the queries and keys.