Neuro 13
0 / 37 chapters finished

Chapter 20: Tokens: turning text into numbers

Build a tokenizer's dictionary one merge at a time.

Every network in this book so far has eaten numbers. Piano done? 1. Petal length? 4.7. But what you type into a chatbot is not a number. It is "tell me a story about a dragon who likes pizza", and a network can't multiply a weight by the word dragon.

So before a language model does anything at all, your text has to be turned into numbers. The trick is to chop the text into chunks and give every kind of chunk its own id number. One chunk is called a token. The full list of chunks a model knows is its vocabulary (we'll mostly call it the dictionary), and the tool that does the chopping is a tokenizer.

That leaves one question, which sounds boring and turns out not to be: what should the chunks be?

Three ways to chop

One token per letter. The dictionary is tiny: a few dozen letters, digits and punctuation marks, and nothing you type can ever be missing from it. But the lists get long (the dragon sentence is 46 characters, so 46 tokens), and a single letter carries almost no meaning. The model has to learn to spell before it can learn anything else. Chapter 21 chops text this way on purpose, because its job is small.

One token per word. Now the sentence is only 10 tokens and each one means something. The catch is the dictionary. English has hundreds of thousands of words, and people add more every day: names, slang, typos. A word that isn't in the dictionary has no id, so the model cannot see it at all. Chapter 24 uses a whole-word dictionary, and you'll watch this problem bite.

One token per piece. Somewhere in between. Common words get a token of their own. Rare words are spelled out of smaller pieces. And the single characters stay in the dictionary as a last resort, so nothing is ever missing. Almost every modern language model works this way.

LettersWhole wordsPieces
Dictionary sizetinyenormous, and never finishedwhatever size you pick
Tokens in the dragon sentence4610between the two
A brand-new word?no probleminvisiblespelled in pieces
The catchlong lists, and each token means very littlethe dictionary always has holessomebody has to choose the pieces

Nobody chooses the pieces. They are learned.

You might picture a team of language experts writing down the useful pieces: ing, un, tion. That is not how it's done. The pieces are worked out from a big pile of text by a counting recipe called byte-pair encoding, or BPE. There's no neural network in it. The recipe is four lines long:

  1. Start with every single character as its own token.
  2. Count every pair of tokens that sit next to each other in the text.
  3. Take the most common pair, glue it into one new token, and add that token to the dictionary.
  4. Go back to step 2. Stop when the dictionary is as big as you want.

Further down, you'll run this recipe on a 635-word story about a dragon who loves pizza. Which pair of neighbors do you think gets glued together first?

It's h + e, which sit side by side 142 times in the story. t + h is a great guess and comes third with 116: every "the" contains both pairs, but "he" also turns up in she, her, they and cheese. Second place goes to something you might not think of as a pair at all: a space followed by a t, 120 times. And z + z only ever appears in pizza, 12 times.

Here it is by hand on two words, banana bandana. The dot · is how we'll draw a space. A space is a character like any other, and in this kind of tokenizer it rides on the front of the word that follows it.

The text right nowNeighbors, countedWhat gets glued
b a n a n a
· b a n d a n a
a+n: 4   n+a: 3   b+a: 2
everything else: 1
a + n → an
b an an a
· b an d an a
b+an: 2   an+a: 2
everything else: 1
A tie, so take the pair that shows up first:
b + an → ban
ban an a
· ban d an a
an+a: 2
everything else: 1
an + a → ana
ban ana
· ban d ana
every pair: 1Nothing appears twice. Stop.

Three merges, and banana went from six tokens to two: ban and ana. Nobody told the recipe those were handy chunks. It found them by counting.

Why "byte-pair"? A byte is the unit a computer stores text in: a number from 0 to 255. Every plain English letter, digit and punctuation mark is exactly one byte. A lowercase a is byte 97 on practically every computer on Earth, and a space is byte 32. So "start with every single character" really means "start with the 256 possible bytes", which is why the dictionary you're about to build starts with 256 entries rather than zero. (The recipe itself was invented in 1994 as a way to make files smaller. Language models borrowed it about twenty years later.)

▸ try it

Build the dictionary

Press merge once and watch three things: which pair wins, the new token it becomes, and the story changing color underneath as letters fuse into pieces. Each colored block is one token. Then press play, or drag the slider to jump to any point. Your own sentence, at the bottom, is chopped with the dictionary exactly as it stands at that moment.

0
dictionary: 256 tokens (256 bytes + 0 learned) the story: 3,427 characters → 3,427 tokens

The most common neighbors right now

Counted across the whole story. The orange row is the pair that gets glued next.

The story, chopped with your dictionary

Your own sentence

46 characters → 46 tokens

see the pieces you have learned so far (0)

Newest first. Ids 0 to 255 are the single bytes, so the first learned piece gets id 256.

read the training story, or swap in your own text (635 words)

The dictionary is learned from exactly this text and nothing else. Change it, press relearn, and you get a different dictionary. Line breaks count as spaces.

What just happened

With the dragon story, the first seven merges are he, ·t, ·the, ·a, er, in and ·s. Look at the third one: after only three merges, the (with its space) is already a single token. Whole words keep arriving after that, most common first: ·and at merge 13, ·dragon at merge 44, ·pizza at merge 48, ·strawberry at merge 125.

After 313 merges the recipe stops by itself, because no pair of neighbors appears twice any more. By then the story has shrunk from 3,427 tokens to 1,024, which is about 3.3 characters per token. The same text, as a list less than a third as long.

GPT-2, a real language model from 2019, built its dictionary with this same recipe from a mountain of internet text. Its first nine merges were ·t, ·a, he, in, re, on, ·the, er and ·s. Every one of your first seven is on that list. A 635-word story about a pizza dragon and a big slice of the internet agree on what English is made of.

Three things worth poking at, with the slider all the way to 313:

  • The dot matters. Type dragon on its own and you get four pieces, even though the story is full of dragons. That's because the story is full of ·dragon, with a space in front, and to the tokenizer that is a different string of bytes. Real tokenizers have exactly this quirk. It also means merges never reach across a gap: the end of one word is never glued to the start of the next.
  • Words the story never used. pterodactyl comes out as seven pieces. Nothing is missing; it just costs more tokens. Try the pizza emoji too. It isn't in the story, so it falls apart into the four raw bytes a computer stores it as (the gray dashed pills). This is why nothing you type into a real chatbot is ever "not in the dictionary".
  • Capital letters. the dragon is 2 tokens. THE DRAGON is 10. The story hardly ever shouts, so the dictionary never learned how. A dictionary is only good at text that looks like what it was learned from.

Once a dictionary is built, it is frozen. It gets made before the neural network's training starts and never changes after that. A word invented next year will simply be spelled out of pieces that already exist.

Honesty note: real dictionaries are learned from billions of words, not 635, and real tokenizers add a few extra rules about punctuation. Some models use close cousins of this recipe with names like WordPiece and SentencePiece. The idea is the same in all of them: the pieces are counted out of real text, never chosen by hand.

An id is only a row number

Your dictionary gave ·dragon the id 299 and ·pizza the id 303. Does that make a pizza slightly more than a dragon? No. The ids are just the order the pieces happened to be created in. Id 299 is no more "close to" id 303 than locker 299 holds the same stuff as locker 303.

So how does a number like 299 come to mean anything? Through a second lookup. The model owns a big grid of weights called the embedding table, with one row for every token in the dictionary. The id says which row to pull out, and that row, a list of numbers, is what the network actually computes with. The row is called the token's embedding. (A list of numbers has a math name, a vector, and a grid of them is a matrix. You'll see both words a lot from here on.)

So the whole way in is text → pieces → ids → rows of numbers. The way out is the mirror image: the network produces a score for every id in the dictionary, one id gets picked, and the same dictionary is read backwards to turn that id into text. The network itself never touches a letter.

The rows are weights like any others in this book. They start out random, and training nudges them, exactly as in chapter 13, until tokens that get used in similar ways end up with similar rows. All of the meaning lives there.

▸ try it

From text to numbers, and back

This box follows the dictionary you built above, which has 256 tokens right now. Type here or up there; both boxes hold the same sentence. Then scroll up, move the merges slider, and come back to see what changed.

Lookup 1: what the model actually receives


  

Lookup 2: each id pulls out its row of the embedding table

Each row here is 16 numbers, drawn as colors: blue is positive, red is negative, darker is bigger. These rows are random. Nothing has been trained, so similar tokens do not have similar rows yet.

And back: the same dictionary, read the other way

When a model writes, it never outputs letters. It outputs one id. Pretend you're the model and pick one:

The first lookup is real: those are the ids your dictionary produces, and reading them backwards always gives your sentence back exactly. Look for a token that appears twice in your sentence. It pulls out exactly the same row both times: same id, same row, wherever it sits. The second lookup is a stand-in, though. In chapter 21 you'll watch rows like these being learned from scratch, and in chapter 24 you can explore thousands of learned word rows and see which words ended up as neighbors.

Why chatbots miscount the r's in strawberry

For a while, the most famous way to embarrass a chatbot was to ask it how many times the letter r appears in strawberry. Some very capable models said two. (It's three.)

You can now see why. Go back to the first widget, merge all the way to 313, and press strawberry. You get four pieces, st ra w berry, which the model receives as [328, 271, 119, 553]. Now press a strawberry. With a space in front, the whole word is one token: id 380. GPT-2's real dictionary behaves the same way. Without the space it sees st raw berry; with the space, a single token.

Now count the r's in 380. You can't. They aren't in there. The model never receives ten letters. It receives a row of numbers that stands for "the strawberry token", and anything it knows about how that token is spelled, it had to pick up second-hand from text that happened to talk about spelling.

Newer chatbots usually get this one right, partly because they have learned to spell a word out one letter at a time before answering. But the reason it was ever hard hasn't gone away, and it shows up again with rhymes, anagrams and writing a word backwards: a model sees tokens, not letters.

Why everything is counted in tokens

Tokens are what a model actually reads and writes, so they are also how its work gets measured.

  • Limits. A model can only read so many tokens at once. That limit is its context window, and for today's big chatbots it runs from about a hundred thousand tokens to a million or more. Paste in something longer and the extra does not fit. Chapter 23 comes back to this, because it explains a lot about what a chatbot "remembers".
  • Prices. Companies that rent out these models charge by the token: so much per million tokens read, and so much per million written.
  • Speed. A chatbot writes one token at a time, so its speed is measured in tokens per second.

A handy rule of thumb for ordinary English: one token is about four characters, or roughly three-quarters of a word. Your little dictionary reached 3.3 characters per token on its own story, which isn't far off.

Special tokens

A few entries in a real dictionary don't stand for text at all. They are signals, called special tokens. The most important is end-of-text. In training, it marks where one document stops and the next one starts. And when a chatbot is writing, it is how the answer ends: at some point the token the model picks is end-of-text, and the app stops asking for more. Chatbots have a few other special tokens too, marking whose turn it is to speak. You'll meet them in chapter 23.

GPT-2's dictionary shows the whole recipe in one number. It has exactly 50,257 tokens: 256 bytes, plus 50,000 merges, plus 1 end-of-text token. Yours has 256 bytes plus 313 merges. It's the same recipe. They just kept pressing the button.

A language model never sees letters or words. It sees a list of id numbers for pieces of text, chopped by a dictionary that was itself learned by counting. An id is only a row number; the meaning lives in the row of learned numbers it points to.

Challenge

Open "swap in your own text" in the first widget and paste in something of your own: a page from a book you like, or a few paragraphs in another language. Press relearn. Which whole words show up first now, and what do they tell you about the text? (Letters with accents, and whole alphabets like Greek or Japanese, are stored as two or three bytes each, so you'll see them start out as raw bytes and get glued back together by the first merges.) Then type an ordinary English sentence and compare its token count with what the dragon dictionary gave you. Which dictionary would you rather pay by the token with?

Why do language models chop text into pieces instead of whole words?

New words, names and typos never stop arriving, and a whole-word dictionary can't see any of them. Pieces fall back to smaller pieces, and finally to single bytes, so nothing is ever missing. It usually costs a few more tokens than whole words would, not fewer.

In your dictionary, ·dragon has id 299 and ·pizza has id 303. What does that tell you about the two words?

The ids are just the order the pieces were created in. Meaning lives in the embedding rows, and those have to be learned.

A friend runs the same BPE recipe on a pile of Spanish text instead. What comes out?

The recipe is the same, but the pieces are counted out of the text you give it. Different text, different counts, different dictionary. BPE doesn't know or care what language it is looking at.