Chapter 21: A language model you can train
Teach a network to invent names, one letter at a time.
Here's a game. I'm thinking of a name. It starts with q. What's the second letter?
You said u, and you didn't have to look anything up. You've read enough names, like Quinn and Quentin, to know what tends to come next. Now a harder one. The name so far is em. Next letter? It could be m (Emma), i (Emily) or e (Emery). You can't be sure. But you could say which letters are likely and which are silly. Nobody's name starts with emq.
That skill has a name. A language model is anything that looks at the text so far and says how likely each possible next piece is. A chatbot is a language model. So is the thing you are about to build. Yours works on names, one letter at a time, and you will train it yourself, on this page, starting from random weights. By the end it will be inventing names that nobody has ever had.
Turning a list of names into practice questions
Our data is a file called names.txt: 8,020 first names given to babies in the United States, taken from public government records. We set 802 of them aside as held-out names, which the model never trains on, so we can test it honestly later (the habit from chapter 15). That leaves 7,218 to learn from.
A network learns from examples: a question, and the right answer. A list of names doesn't look like that. Here is the trick that turns it into one. Slide a little window along each name, and at every stop ask: given these letters, what comes next? The window is three letters wide. In chapter 20 you met the context window, the most tokens a model can see at one time. Ours is three tokens. That is tiny on purpose, and it will matter later.
Two details. First, the tokens here are single letters, the simplest of the three ways to chop from chapter 20. Second, a name has to stop somewhere, so we add a 27th symbol, the end marker, which means "the name ends here". We draw it as a dot. It is a tiny version of the end-of-text token that tells a chatbot it has finished. The dot also fills the window at the start of a name, before there are three letters to look at. So the dictionary has exactly 27 tokens: 26 letters and the dot.
Slice a name into questions
Type any name. Each row is one training example: the three purple boxes are what the model gets to see, and the orange box is the answer it should give. Watch the window slide one letter per row.
A four-letter name makes five examples: one for each letter, plus one for the ending. All 7,218 training names together make 50,038 examples.
Notice that nobody had to label anything. The text is its own answer key: cover up the next letter, and the covered letter is the right answer. Every language model gets its training examples this way, chatbots included. They slide along billions of pages instead of a list of names.
First try: just count
Before we use a network, let's try the simplest model that could possibly work. Look at only one letter, and count what came after it in the training names. How often was a followed by n? By r? By the end of the name? There are 27 symbols that can come first and 27 that can come next, so the whole model is a table of 27 × 27 = 729 counts.
Before you look at the table: which letter do the most names in the list end with?
It's n. Of the 7,218 training names, 1,633 end in n (Aiden, Logan, Madison, Jordan), ahead of 1,372 that end in a. You can check: in the table below, find the row for n and look at its first square.
The counting table
Each row is the letter you are on. Each column is what came next. The darker the square, the more often that happened. The dot row at the top is the start of a name, and the dot column on the left is "the name ended". Move over the squares (or tap them) to read the counts. Look along the q row. Then compare the row for j with the row for n.
↓ this letterwhat came next →
Loading the names…
Each row of that table, divided by its total, is a list of 27 probabilities: after this letter, here is how likely each symbol is to come next. After q, the table says u 89% of the time (47 times out of 53). That makes the table a language model already, with no network anywhere. Engineers call it a bigram model, because it only ever looks at pairs of letters.
The button shows how a language model writes. Start on the dot row. Roll a weighted die to pick the next letter, so that likelier letters win more often. Jump to that letter's row and roll again. Stop when the die lands on the dot. Picking at random, with likelier options picked more often, is called sampling.
The names it invents are mostly bad: things like scrnan and amomeelimaraylll. Every pair of neighboring letters is believable, but the table can only see one letter back. It has no idea it just wrote three consonants in a row, or that the name is already sixteen letters long. Even so, this is far from random typing. It starts names with the letters names start with, and it stops them on the letters names stop on.
Keeping score
We need a fair way to say whether one model is better than another. Take the 802 held-out names and walk along them one letter at a time. At each step, look at the probability the model gave to the letter that really came next. That is the same score you used for the snack picker in chapter 10: the loss is −log of that probability, so a confident right answer costs almost nothing and a surprise costs a lot. Average it over every letter of every held-out name.
There is a handy way to read a loss. Type it into a calculator and press the e^x button, and it turns into a number of choices. A model that knows nothing gives each of the 27 symbols the same chance, 1 in 27. Its loss is 3.30, and e3.30 is 27. So a loss of 3.30 means "as unsure as picking among 27 equally likely symbols". A loss of 0.69 means picking between 2, a coin flip. A loss of 0 means 1 choice: total certainty. (Engineers call this number of choices the perplexity.)
| model | held-out loss | as unsure as picking among |
|---|---|---|
| knows nothing | 3.30 | 27 symbols |
| knows only which letters are common | 2.83 | 16.9 symbols |
| the counting table (one letter back) | 2.45 | 11.6 symbols |
| the network you are about to train (three letters back) | ? | ? |
No model can reach 0 here, and that is not a failure. After em, even a perfect model can't know whether the name is going to be Emma or Emily. Names are only partly predictable.
Honesty note: when we score the counting table we first add 1 to every count. Otherwise a pair it never happened to see would get a probability of exactly 0, and one such pair in the held-out names would make its loss infinite.
Why not just count three letters?
The counting table is weak because it has one letter of memory. So give it three! The trouble is the size of the table. It would need a row for every possible three-letter window: 27 × 27 × 27 rows of 27 squares, which is 531,441 squares. Our training names put a count in only 10,937 of them, about 2%. The rest stay empty, so for most windows the table has nothing to say. And it can't notice that emm and amm are nearly the same situation. To a table, they are two unrelated rows.
A network can share what it learns. If two letters behave alike, it can give them similar numbers, and then similar windows lead to similar predictions, including windows it has never seen.
Honesty note: with a window this short, the giant table is not a bad model. Counted the same simple way, it scores 2.31 on the held-out names, and with more careful handling of its empty squares it can reach 2.13, which is better than the small network you are about to train. Tables lose as the window grows: every extra letter of memory makes the table 27 times bigger, while a network hardly grows at all.
The network
You already own every part of this:
- Three letters go in, as ids from 0 to 26.
- Each id picks its row of the embedding table, exactly as in chapter 20. Our rows are very short: just 2 numbers per symbol.
- The three rows are laid side by side, making 6 numbers, and fed to a hidden layer of 64 neurons (chapter 8) that squish with tanh (chapter 17).
- An output layer gives 27 scores, one for each symbol, and softmax (chapter 10) turns them into 27 probabilities.
The numbers in the picture come from one of our own training runs, with the window on emm. Notice that both m's pull out the same row. Yours will come out different.
That is 54 + 448 + 1,755 = 2,257 weights in all. This design is a famous one. Yoshua Bengio and his colleagues published it in 2003, using words where we use letters, and it is the ancestor of today's language models.
Training is what you know from chapter 13 and chapter 14. All 2,257 weights start out random. One step is: grab a mini-batch of 32 random examples, see how surprised the network is by the right answers, pass the blame backwards, and nudge every weight a little. We take 20,000 steps. The learning rate starts at 0.3 and shrinks steadily to 0.01, so the network takes big strides early and careful ones at the end.
The network starts with random weights. What will the "names" it writes look like before any training?
With random weights, every one of the 27 symbols gets nearly the same chance, and that includes the end marker. So it draws letters at random and only stops when it happens to draw the dot (a 1-in-27 shot each time) or reaches the 24-letter limit we set.
Train a name inventor
Press train and watch three things. The chart: the loss should dive, and the grey line is the counting table's score, the one to beat. The ten names: every so often the network writes ten fresh ones, and early on the page pauses for a blink so you can read them. The letter map: every symbol is drawn at the two numbers in its embedding row. They start in a random clump in the middle.
Ten names it is making up right now
The letter map
orange vowelsblue consonants• end marker
A symbol's place on the map is its row of the embedding table: first number across, second number up. Point at a letter to read its row. Click the map to zoom in on the crowded middle, and click again to zoom back out.
How it sounded along the way
What just happened
The loss. It started at 3.30, the "knows nothing" score: 27 choices. It dropped under the counting table's 2.45 after about 2,000 steps, and finished near 2.25. (We ran this 20 times; the held-out loss always landed between 2.24 and 2.27. Yours will differ a little, because every run starts from different random weights.) Press ex on 2.25 and you get about 9.5. Seeing three letters back has taken the model from 11.6 choices to about 9.5.
The two lines. The loss on the names it trained on ends near 2.21, and on the held-out names near 2.25. Those are close. This network is not memorizing its homework the way the one in chapter 16 did. It is finding habits that hold for names it has never seen.
The names. Read the "How it sounded along the way" strip from top to bottom. At step 0 it types letter soup. A hundred steps later the names have a sensible length and a mix of vowels and consonants. By the end, many of them could pass for names on a class list. Some are still monsters, and there is a reason for that, which we will get to.
The letter map. Look where the orange letters ended up. Nobody told the network what a vowel is. Its only job was to guess the next letter. But a, e, i, o and u do similar jobs in names (they tend to come after consonants and before them), so training pushed their rows toward the same part of the map. In all 20 of our runs, one straight line could separate the five vowels from all 21 consonants. Now find y. In half of our runs it was the consonant sitting closest to the vowels, which is fair for the letter in Lily and Kyle. This is what chapter 20 meant by "tokens that get used in similar ways end up with similar rows", and you just watched it happen.
Ask it, then let it write
The network gives you 27 probabilities for whatever window you show it. To write a name, it does what the counting table did: sample one symbol, stick it on the end, slide the window along, and ask again. Predict, sample, append, repeat.
There is one knob on the sampling, called temperature. Every score is divided by the temperature before softmax. Below 1, the gaps between scores stretch, so the favorites win even more often. Above 1, the gaps shrink, so long shots get a real chance. At exactly 1, you get the probabilities the network learned.
Before you touch the slider: what will the names be like at temperature 0.2?
The weights really haven't changed, but the dice have. At 0.2 the most likely letter wins almost every time, so the network keeps walking down the same few well-worn paths. In one of our runs, 3,000 names at temperature 0.2 contained fewer than 200 different names.
Ask it
Train the network above first. Then type the start of a name. The bars show the chance the network gives each symbol of coming next. Press sample one letter again and again to watch it write one letter at a time, or let it finish ten names for you. Then move the temperature and watch the bars change shape.
Three experiments worth doing:
- Slide the temperature to 0.2, then to 2.0, and invent ten names at each. At 0.2 you get the same few safe names again and again. At 2.0 the bars flatten out and the names fall apart into things like dnoer and zey. Chatbots have this same knob, and chapter 23 comes back to it.
- Type a long start, like christoph. The greyed-out letters are invisible to the network. It sees oph and nothing else. That is why it sometimes writes a monster like janiyaslitee: it can't see how long the name has already become, so it doesn't know it is time to stop.
- Clear the box and type q. The counting table knew that u follows q 89% of the time. Your network is probably much less sure. Across 40 runs, ours gave u anywhere from 2% to 68%, and usually less than 30%. Why would a network lose to a table? Because q shows up only 53 times in 50,038 examples, and the network has just two numbers to describe each letter. A flat map doesn't have room to keep 27 symbols apart. Look at yours: there is a crowd of blue consonants in the middle, sitting almost on top of each other, and to the network those letters are nearly the same letter. Now find q. In our runs, the networks that did well on this test were the ones where q had ended up far away from every other letter. Real language models give each token hundreds or thousands of numbers instead of two, and this is one reason why.
Honesty note: every so often, random letters spell something rude. This page quietly throws those names away and draws another. Everything else you see is exactly what the network sampled.
Same code, different data
Here is the same network again: same 2,257 weights, same 20,000 steps, not one line of code changed. The only thing you can change is the list it learns from. The second list is 1,389 dinosaur names (1,250 to train on). For the third, you bring the words.
Where the lists come from: the first names are from public records of the United States Social Security Administration, by way of the names file in Andrej Karpathy's makemore project. The dinosaur names are from Wikipedia's "List of dinosaur genera", shared under the Creative Commons Attribution-ShareAlike 4.0 license.
Pick the data
Choose a list and train. Try the dinosaurs first. Then paste in a list of your own: everyone in your class, the teams in your league, towns near you, creature names you made up. Fifty words or more works best.
Your list never leaves your computer. The training happens right here in this page, and nothing is sent anywhere. (Your browser remembers the list so that it is still here if you reload.) We lowercase everything and keep only the letters a to z.
Ten it is making up right now
Dinosaurs. The same code now writes things like jeterosaurus, chanosaurus and micerator. Of the 1,250 names it trained on, 522 contain saur, and it noticed. In our runs, a network shown the window aur gave more than 90% to u, and after rus it was 99% sure the name was over. Its held-out loss lands near 1.8 (the counting table gets 2.17 on this list). Dinosaur names are easier to predict than baby names, because they are built from the same few Greek and Latin pieces.
A short list. Try the 129 foods. The names are a treat: in our runs it served up pomato, goosemary, roseberry and pumpling. But look at the chart. The blue line dives toward 0.8 while the orange held-out line climbs past 4 and often past 5, far worse than knowing nothing. That is overfitting, straight out of chapter 16. With only 116 words to learn from, 2,257 weights are enough to nearly memorize them. Between a third and a half of what it wrote in our runs were plain copies of words on the list, and most of the rest were memorized chunks glued together.
The code has no idea what a name, a dinosaur or a food is. Everything the model seems to know came out of the list. A model is made of its data.
It invents. It does not remember.
You might picture the model as a filing cabinet: it stored the 7,218 names, and now it pulls them back out. Let's check that. We can compare every name it writes against the list, which is something the model itself can't do.
The first-names network writes 200 names. How many of them will be copies of names it trained on?
In our runs it was about one in twelve. Roughly nine names in ten were ones the network had never been shown.
New, or real?
This uses the first-names network you trained in "Train a name inventor". It writes 200 names at temperature 1 and looks each one up. Purple names are new: they are not on our list at all. Grey ones copy a name from the training list. Gold ones are the special case: real names from the 802 we held out, which the network was never shown and came up with anyway.
In our runs about 90% of the names were new, 7 to 10% copied a training name, and about 1% were gold: real names it had never seen. And "new" only means "not on our list". When we checked the new names against a bigger list of 29,494 real names, which we had kept away from the network, about one in ten turned out to be somebody's real name.
So the filing-cabinet picture is wrong, and you can see why from the sizes. The training names contain 42,820 letters. The network has 2,257 weights. There is no room in there to keep the list. There is only room for habits: vowels and consonants take turns, names like to end in n or a, ll is fine and qz is not. When it writes, it follows those habits to somewhere new. Sometimes that somewhere is a name a real person already has.
Now the uncomfortable part. The model has no idea which of its names are real. The badges came from our list, not from the network. From the inside, a real name and an invented one are the same kind of thing: a string of likely letters. A chatbot is in the same position with facts, and chapter 25 looks at what that leads to.
From here to a chatbot
What you built is a real language model. A chatbot runs the same loop: predict a probability for every token, sample one, append it, repeat until the end token comes up. The differences are a matter of size.
| your name inventor | a big chatbot | |
|---|---|---|
| tokens in the dictionary | 27 (letters and a dot) | about 100,000 (pieces of words) |
| how far back it can see | 3 tokens | hundreds of thousands of tokens |
| numbers in each embedding row | 2 | thousands |
| weights | 2,257 | billions |
| learned from | 7,218 names | a large part of everything people have written |
| how it writes | predict, sample, append, repeat | predict, sample, append, repeat |
One of those rows can't be fixed by making our network bigger: how far back it can see. Our design lays the rows for three letters side by side and wires each position to the hidden layer separately. Stretch that to a thousand tokens and the first layer becomes enormous, and it is still rigid: it learns one set of weights for "the letter two places back" and a different set for "the letter three places back", when what matters in real writing might be a word forty places back in this sentence and four hundred back in the next. What you want is a network that can look over everything so far and pick out the parts that matter right now. That trick is called attention, and it is chapter 22.
Run the name inventor
This is names.py: the same network as the widget, in about 100 lines of numpy. It reads names.txt, shows the examples it cuts from the first name, scores the counting table, then trains and prints the loss every 2,000 steps along with one name the network writes at that moment. It finishes with twenty invented names, the q test, and the letter map drawn in text.
A language model does one thing: given the text so far, it gives a probability to every possible next token. Writing is that one thing in a loop: predict, sample, append, repeat. What comes out follows the habits of the training data. It is invented on the spot, not looked up.
Two numbers per letter was a tight squeeze. In names.py, change EMBED = 2 to EMBED = 10 and HIDDEN = 64 to HIDDEN = 100, and run it again. What happens to the held-out loss, and to the chance it gives u after q? (You lose the letter map, because ten numbers can't be drawn on a flat page.) Then go the other way: set CONTEXT = 1, so the network sees one letter, the same as the counting table. Which of the two does better now, and can you say why?
Where did the 50,038 training examples come from?
Text is its own answer key. Cover the next letter and you have a question with a known answer, so a plain list of names is all the training data a language model needs.
You turn the temperature down from 1.0 to 0.2 and the names get safer and more alike. What changed inside the network?
Temperature is a setting on the dice, not on the weights. The network computes exactly the same scores either way.
A friend says: "It's not inventing anything. It saved all the names and picks one at random." What is the best evidence against that?
A shuffled copy of the list would also come out in a different order every time. What settles it is that most of the output was never in the list, and that the network is far too small to store it. It learned habits and it follows them.