Neuro 13
0 / 37 chapters finished

Chapter 23: How AI chatbots actually write

Next-token prediction, sampling, and the training that makes it polite.

You now own all the parts. Chapter 20 turned text into token ids. In chapter 21 you trained a network to predict the next letter and let it invent names. Chapter 22 showed how a network learns where to look. A chatbot is those three ideas, made enormous.

The kind of network inside is called a transformer, a design built around attention that was introduced in 2017 in a paper titled "Attention Is All You Need". A very big transformer trained on text is a large language model, or LLM. ChatGPT, Claude and Gemini are all LLMs. This chapter follows one message through one of them, and then deals with the part most people get wrong: what a chatbot knows, and doesn't know, while it is talking to you.

The only job: predict the next token

Take a mountain of text. Cover up everything after some point. Ask the model which token comes next. Uncover the answer, measure how wrong the guess was, and nudge the weights. Repeat, trillions of times. That is the whole first stage of training an LLM. It's called pre-training.

It is chapter 10's snack picker with a much longer menu. Instead of three snacks there is one output for every token in the dictionary, which means 100,000 choices or more. Softmax turns the scores into probabilities exactly as it did there.

Notice that nobody has to label anything. In chapter 15 a person had to write down which kind of flower each row was. Here the text is its own answer key: the right next token is always sitting there, one step ahead. That is what lets these models learn from more text than any team of people could ever label.

A chatbot writes you an answer that is 200 tokens long. How many times did the whole network run to produce it?

Once per token. Each trip through the network produces exactly one token. Step through the pipeline below to see what a single trip involves, then the widget after it to see the trips add up.

▸ try it

One trip through the network

Your message is "tell me a story about a dragon who likes pizza". Press next stage to follow it through the four things that happen before a single token of the reply appears. Watch what the text has turned into at each stage.

1. Tokenize
2. Embed
3. Attention
4. Predict

Stage 1: chop the text into tokens

The tokenizer chops your message into pieces and swaps each piece for its id, just as in chapter 20. These are the real ids from GPT-2's dictionary, the one with 50,257 entries. The dot is a space.

All of that produced one token. Now comes the step that turns a predictor into a writer: the chosen token is stuck onto the end of the text, and the whole pipeline runs again on the slightly longer text to choose the token after it. And again. And again. The model writes by repeatedly reading everything so far, including its own words, and adding one more piece.

Honesty note: real systems save part of the work from earlier trips instead of redoing all of it, so each new token is cheaper than starting from nothing. The logic is still exactly this loop.

▸ try it

Watch a reply get written one token at a time

Illustration · hand-written numbers, not a live model

Press next token. Each press is one full trip through the network: you see its five most likely guesses, and the top one is added to the reply. Watch how sure the model is at different moments, and watch how the reply comes to an end.

The guesses and probabilities in this widget were written by hand to look like what a real model produces. They are not coming out of a network. Each step is also shown as a whole word to keep it readable; GPT-2's real tokenizer would write Pepperoni as two pieces, ·Pepper and oni. For a model that really does this live, with probabilities it learned by itself, go back to the name inventor in chapter 21.

you
tokens in the reply: 0 trips through the network: 0

Look at how the model's confidence moves around. After "Once" it is nearly certain that "upon" comes next. When it has to name the dragon, its best guess only gets 18%, because plenty of names would do. And nothing outside the model told it to stop. It stopped because, after the last period, the likeliest next token was the special end-of-text token from chapter 20.

Nothing is carried from one trip to the next except the text so far. There is no outline tucked away somewhere. And yet this loop produces essays, working code and step-by-step math. The reason is that being really good at "what comes next?", across everything people have written, takes a lot: grammar, facts, how stories go, how programs work, how an argument is built. A simple-sounding job, done well enough, turns out to need far more than it sounds like.

Honesty note: "no outline" does not mean no thinking ahead. Researchers who look inside these networks have found that, while choosing one token, a model can already be working toward a word it will need several tokens later, such as a rhyme. Chapter 35 has more about looking inside.

Sampling: why the same question gets different answers

Stage 4 ends with a probability for every token. So which one gets written? Not always the top one. The app rolls weighted dice, and each token's chance of being picked is its probability. This is called sampling, and it is why asking a chatbot the same thing twice gives you two different answers.

Three knobs control the dice. All of them work on the scores before the dice are rolled. (Engineers call these raw scores logits.)

  • Temperature. Every score is divided by the temperature before softmax. You met this knob in chapter 21. Below 1, the gaps between scores stretch, so the favorite wins more often. Above 1, the gaps shrink, so long shots get a real chance. At exactly 0 there are no dice at all: the top token wins every time.
  • Top-k. Only the k highest-scoring tokens are allowed. Everything else is cut, however the dice fall.
  • Top-p. Sort the tokens from most to least likely and walk down the list, keeping tokens until their probabilities add up to p (say 0.9). Cut the rest. Unlike top-k, the length of the list changes by itself: when the model is sure, two or three tokens are enough to reach 90%; when it is unsure, it takes many.

You turn the temperature up from 1 to 2. What happens to the bars?

Dividing every score by 2 halves the gaps between them, and smaller gaps mean more even probabilities. Try it below and watch the long shots start getting picked.

▸ try it

The sampling playground

Illustration · twelve hand-written scores

A model is partway through a sentence and has to choose the next token. Move the knobs and watch the bars reshape, then press sample 100 times to roll the dice for real. The formulas are the true ones. The twelve scores are made up for this demo: a real model would have a score for every token in its dictionary, and they would come out of the network.

The dragon's favorite food was .
1.00
all 12
off
tokenscorechance of being pickedpicked

chance right nowwhere the bar started (temperature 1, nothing cut)share of your samples so far

tokens still in the running: 12 of 12 samples: 0 different words picked: 0

What the knobs are doing right now

    What just happened

    With every knob at its starting position, pizza has a 44.5% chance and is about 100 times as likely as broccoli. Sample 100 times and pizza wins roughly 45 of them, though rarely exactly 45, and a second hundred gives a different tally. That wobble is the dice.

    • Temperature 2 drops pizza to 23.9% and leaves it only 10 times as likely as broccoli. Even homework, at 1.6%, turns up once or twice in most hundreds. The writing gets more surprising, and more likely to be nonsense.
    • Temperature 0.2 pushes pizza to 98.7%. Safe, and the same nearly every time.
    • Top-k 3 leaves pizza, cheese and noodles, and nothing else can ever be picked.
    • Top-p 0.9 keeps 6 tokens at temperature 1. Now change the temperature and leave top-p alone: at 0.5 it keeps only 3, and at 2 it keeps 9. The list grows and shrinks to match how sure the model is. Top-k can't do that.

    No setting is the correct one. Low temperature suits jobs with one right answer, like arithmetic or looking something up in a document. Higher settings suit brainstorming and stories, where a surprise is the point. The people who build a chatbot choose something in between.

    From text predictor to assistant: three stages of training

    Everything so far describes a model that continues text. That isn't yet a chatbot. Getting from one to the other takes three stages of training, and each stage starts from the weights the stage before left behind.

    Stage 1: pre-training. Next-token prediction on an enormous pile of text: web pages, books, computer code. Meta said its Llama 3 model was pre-trained on more than 15 trillion tokens. If you read for eight hours a day, every day, for eighty years, you'd get through that much in about three thousand lifetimes.

    What comes out is called a base model, and it is not an assistant. It is a document-continuer. Type "What's 2+2?" into a base model and it may carry on as if your question were the first line of a worksheet: "What's 3+3? What's 4+4?" It isn't being difficult. Continuing the document is the only thing it has ever been trained to do.

    Stage 2: example conversations. To turn a document-continuer into something that answers, you train it some more on a much smaller pile of one particular kind of text: conversations, written out in a fixed format, in which a user asks things and an assistant answers them well. Training an already-trained model further like this is called fine-tuning. Many of the examples are written by people hired to do it; others are drafted by a model and then checked. The task hasn't changed at all. It is still "predict the next token". But now the model has seen, over and over, that after a user's turn comes a helpful assistant's turn.

    That's the sneaky, important idea in this chapter: a chat is still a document. The model is continuing a transcript in which one of the characters happens to be a helpful assistant.

    Stage 3: learning what a good answer is. Copying examples only gets you so far, because for most questions nobody has written the perfect answer in advance. People can do something easier, though: look at two answers and say which is better. So the model writes several answers to the same question and people compare them. A second network is trained to predict which answers people prefer, which turns it into an automatic judge. Then the chatbot practices against the judge, a huge number of times, and its weights are nudged toward the kind of answer that scores well. This is called RLHF: reinforcement learning from human feedback. (Reinforcement learning, which means learning from a score instead of from the right answer, gets its own chapter: chapter 30.)

    More recently, companies have also started writing down the principles they want a model to follow: be honest, be helpful, don't help people do harm, and what to do when those pull in different directions. Anthropic calls its document a constitution and OpenAI calls its one a model spec. Both are public, so anyone can read them. They get used in training too. For example, a model can do the judging, comparing two answers by asking which one follows the written principles better.

    StageTrained onWhat it adds
    1. Pre-trainingtrillions of tokens of ordinary textlanguage, facts, patterns: a base model that continues documents
    2. Example conversationswritten-out chats in a fixed formatthe habit of answering as an assistant
    3. Preferences and principlescomparisons between answers, and written principlesjudgment about which answers are good ones

    Honesty note: every company does these stages a little differently, the recipes keep changing, and not every detail is published. The three-stage shape is the common one. Newer models add more on top, such as practice at reasoning step by step, which chapter 35 covers.

    The context window: everything it can see

    Here is roughly what a model is looking at when it writes a reply in a chat app:

    <system>

    You are Sparky, a friendly homework helper for middle schoolers. Today is Tuesday, March 3. Don't just hand over the answer: ask a question that helps the student get there.

    <user>

    whats 7 x 8

    <assistant>

    Good one to know by heart! Do you remember what 7 x 7 is? Start from there.

    <user>

    49

    <assistant>

    ▌ the model writes from here, one token at a time

    A made-up example. The markers in angle brackets are special tokens that say whose turn it is; every company uses its own.

    That is all of it: one long list of tokens. For every token it writes, the model reads the whole list and predicts what comes next.

    The list has a maximum length, the context window from chapter 20. And here is the part that surprises people: the context window is the model's entire memory of your conversation. It "remembers" what you said five minutes ago for one reason only. Your words are still in the window, and it reads them again on every trip. If a chat grows longer than the window, the oldest part has to be dropped or squeezed into a summary, and after that the model can't see it. Start a new chat and the window is empty.

    Meanwhile the weights, where everything from training is stored, do not change at all while you talk. Chatting is all forward passes and no nudging. Chapter 5 said that a trained model is its numbers and nothing else. Those numbers are exactly the same after your conversation as they were before it.

    Some apps have a "memory" feature that seems to break this rule. It doesn't. The app saves a few notes about you as ordinary text and quietly pastes them into the window at the start of each new chat.

    The system prompt: instructions you don't see

    Look at the top of that window again. Before your first message there is a block of text you didn't write and usually can't see. This is the system prompt: instructions from whoever built the app. It sets the assistant's role ("you are a homework helper") and its rules ("don't just hand over the answer"). It often holds plain facts the model has no other way of knowing, too, such as today's date. A model has no clock. If it knows what day it is, somebody wrote the date into the window.

    This is how the very same model can be a patient tutor in one app, a customer-service agent in another and a pirate in a third. Its weights are identical in all three. Only the text at the top of the window is different.

    Three things people get wrong

    • "It's learning from me as we chat." It is not. The weights are frozen while you talk. It can use what you said earlier because that text is still in the window, and for no other reason. (A company may later use saved chats as training text for a future model, depending on its rules and your settings. That would be a separate training run, months away, and not something that happens while you type.)
    • "It looks things up." Not unless it has been given a tool for that. By default, every word comes out of the weights: patterns absorbed from training text that ends at some cutoff date. When a chatbot does have a search tool, the app runs the search, pastes the results into the window, and the model reads them like any other text. Chapter 35 says more about tools.
    • "It sounds sure, so it must be right." Sounding sure is a writing style, and the model learned it from text along with every other style. A made-up fact comes out just as smoothly as a true one, because fluent text is what the training rewarded. Chapter 25 is about exactly this, and about how to check.

    A chatbot is a next-token predictor run in a loop, rolling weighted dice at every step, playing a role that its training and a hidden system prompt gave it. Everything it knows at this moment is either frozen in its weights or sitting in its context window.

    Challenge

    In the sampling playground, find two different ways to make sure broccoli can never be picked while cheese still can: one using only top-k, and one using only top-p. Then answer this without a widget: if a chatbot ran at temperature 0, it would give the same answer to the same question every time. What would you gain from that, and what would you lose?

    You ask a chatbot the same question in two fresh chats and get two different answers. Why?

    The weights are the same and nothing was learned in between. The model produces probabilities, and the token that gets written is drawn at random from them, so two runs can part ways at any step.

    An hour into a very long chat, the chatbot seems to have forgotten something you told it at the start. What is the most likely reason?

    The model can only see what is in the window. Once the conversation is longer than that, the oldest text is dropped or summarized. Its weights never changed in the first place.

    A chatbot tells you something about your town in a very confident voice. What does the confident voice tell you?

    The model was trained to produce fluent, likely text, and confident-sounding sentences are fluent and likely. Unless it has a search tool it didn't look anything up, and either way the tone is not evidence.