Neuro 13
0 / 37 chapters finished

Chapter 24: A model that decides instead of writes

Train a decision model on 10,000 real headlines.

The chatbot in chapter 23 writes. But a huge amount of software doesn't want an essay. It wants a decision: is this email spam? Which team should get this support ticket? How urgent is this message, from 0 to 1? Asking a chatbot those questions is like asking a novelist to tick a box. You get a paragraph back, and then your program has to dig the answer out of it.

In 2026 a new kind of model showed up built for exactly this job. The companies making them call them System One models. The name is borrowed from the psychologist Daniel Kahneman, who described two ways people think: System 1 is the fast gut reaction ("that's a dog"), System 2 is the slow, careful working-out ("what's 17 × 24?"). The first two well-known ones are Jev, from a company called TypeSafe AI (September 2026), and Clef, from Cloudflare (October 2026).

You don't chat with a System One model. You hand it some text plus a list of typed questions, and it hands back numbers. There are three kinds of question:

Question typeExampleWhat comes back
yes / noIs this spam?{"spam": 0.88}
choiceWhich topic: world, sports, business or scitech?{"choice": "sports", "confidence": 0.97}
scoreHow urgent is this, 0 to 1?{"urgency": 0.31}

Compare that with how chapter 23 described a chatbot:

  • No writing loop. A chatbot runs its whole network once per token, over and over, until the story is finished. A decision model runs once and it's done. That's most of why it is so much faster and cheaper.
  • It can't ramble. Every possible answer is listed in the question. The model can be wrong, but it can't wander off and invent something that wasn't on the list.
  • The doubt is part of the answer. You don't get "spam". You get "88% spam", and your program decides what to do with an 88.
  • It can't do anything else. No stories, no code, no explanations. It is a box-ticker, and proud of it.

Jev vs. Clef

JevClef
Readstext onlytext and pictures
Can you download the weights?no — you rent it over the internetyes — the weights are public, so you can run it on your own computers
Sizesecret27 billion weights, made by re-training an existing chatbot-style model
Speed and pricefaster and cheaperslower and pricier

Honesty note: these models are brand new, and most of what's known comes from the companies themselves and from early reviewers, not from independent scientists. Treat the details as "reported", not "proven".

Let's build one

Here's the good news: you already know every part. A decision model is chapter 10's snack picker, fed by chapter 20's lookup tables. We'll train one on 10,000 real news headlines stored in a plain file called headlines.csv — one news story per row (its headline plus the first sentence or two), each labelled with one of four topics (world, sports, business, scitech). The headlines come from a famous research collection called AG News, gathered in 2004.

The whole pipeline, start to finish:

  1. Chop the headline into words and look each one up in a dictionary to get its id number. The dictionary is built from the training headlines: any word seen at least 3 times gets an id.
  2. Each id grabs its row of the embedding table — 16 learned numbers per word.
  3. Average the rows into a single list of 16 numbers. That's the headline's "fingerprint".
  4. One layer turns 16 numbers into 4 scores, one per topic. Softmax turns the scores into probabilities. Done.

So is a token "just a dictionary word turned into a number"? In this model, yes, exactly — and you're about to see why real models don't do it that way. Our dictionary has about 8,750 words. Type a word that isn't in it and the model simply can't see it. That's the problem chapter 20's word-pieces solve: with pieces, nothing is ever missing.

The other thing to hold on to: the id number itself means nothing. Word 3,365 isn't "bigger" than word 483. The id is only a row number. All the meaning lives in the 16 numbers in that row — and those start out random and get nudged into place by the same training you've used since chapter 4.

Before you train it: the model will study 8,000 headlines, fifteen times over. Afterwards we test it on 2,000 headlines it has never seen. How will it do on those?

A bit worse. Some of what it picks up is true of news in general, and some is only true of those particular 8,000 headlines. The second kind is no help on new ones. Watch the gap open between the two lines: you met it in chapter 16.

▸ try it

Train a decision model on 10,000 headlines

Press the button. Your browser downloads the headlines (about 2 MB), builds the dictionary, and trains the model right here: 8,000 headlines to learn from, 2,000 held back for an honest test. Nothing is pre-baked.

epoch: 0 / 15 train accuracy: ? test accuracy: ? dictionary: ? words stored numbers: ?

Blue = accuracy on the headlines it trains on. Orange = accuracy on the 2,000 it has never seen. The gap between the lines is overfitting: the part it memorized instead of learned.

Ask it something

choice: which topic?
yes/no: is it world news?
yes/no: is it sports?
yes/no: is it business?
yes/no: is it scitech?

The model's entire reply:

(train the model first)

Peek inside: which words did it file together?

Nobody told the model which words are related. Type a word, and these are the words whose 16-number rows ended up most like its own.

(train the model first)

Should you believe its confidence?

We split the 2,000 test headlines into four groups by how sure the model said it was, then checked how often each group was actually right. If "90% sure" really means right about 9 times in 10, the model is calibrated.

(train the model first)

You're the boss: set the "sure enough" line

Real programs don't act on every answer. They pick a line: above it, the computer decides alone; below it, a human takes a look.

70%
computer decides: ? and gets right: ? sent to a human: ?

What just happened

Watch the two lines on the chart again. At the start the model is barely better than 25% — pure guessing, one in four. Then the embedding rows start to carry meaning and both lines take off. By the end it gets roughly 82 out of 100 unseen headlines right, using about 140,000 weights. Clef has 27 billion. Same idea, about two hundred thousand times bigger.

Three things worth poking at:

  • The grey pills. Words the dictionary has never seen are invisible to the model. Try a headline full of modern words — TikTok, ChatGPT, Fortnite. The training headlines are from 2004, so the model has never heard of any of them. A model only knows the world its data showed it.
  • "My dog ate my homework." None of the four topics fits. Sometimes the model shrugs and gives a low confidence, which is the right thing to do. Sometimes it's confidently wrong. Press "train again from scratch" a few times and watch the answer change — each run starts from different random weights. The lesson: the calibration bars only hold for headlines like the ones it was tested on. Show a model something unlike anything in its data and its confidence stops meaning much.
  • The word neighbors. Try software, president, profit, coach. The only thing the model was ever asked to do was sort headlines into four piles, yet words that push toward the same pile drifted together. This is chapter 20's promise about embeddings, made real: similar jobs, similar rows. It's rough around the edges, too, because 10,000 headlines is not much to learn a language from.
  • The "sure enough" slider. Slide it up: the computer handles fewer headlines but makes fewer mistakes. Slide it down: it handles more and gets more wrong. There's no correct setting. A spam filter can afford mistakes; a hospital can't. Choosing the line is a human's job.

What our toy can't do

Our model shares the shape of a System One answer — typed, one pass, with a probability — but it's a long way from Jev or Clef:

  • Its questions are baked in. Ours can only ever answer "which of these four topics?" because that's what it was trained on. The real ones read your question fresh each time, so you can ask about spam today and urgency tomorrow without re-training.
  • It ignores word order. We average the word rows, so "dog bites man" and "man bites dog" look identical to it. Real models keep the order, using the attention from chapter 22.
  • It only knows four boxes. Everything in the universe has to be world, sports, business or scitech. It will file your homework excuse under one of them.

None of that changes the big idea. A chatbot and a decision model are the same kind of machine pointed at different jobs: one predicts the next token, again and again; the other predicts the answer, once.

▸ run real python

Run the decision model

This is neuro10.py: the same model as the widget above, in about 150 lines of numpy. It reads headlines.csv, trains, checks its own calibration, makes some decisions, and finishes by listing which words ended up with similar embedding rows — nobody told it that football belongs near league.

neuro10.py Python loads on first run
(click "run" to execute)

A chatbot and a decision model are the same kind of machine. One predicts the next token, over and over. The other predicts the answer, once, and tells you how sure it is. What you do with that doubt is a human decision.

Challenge

Change LR = 1.0 to LR = 4.0 and run it again. Training accuracy climbs faster — but look at the test accuracy and at the "how honest is its confidence?" table. What happened to the model's honesty? Then try EMBED = 2: how well can it do with only two numbers per word?

Why is a decision model so much faster and cheaper than a chatbot at ticking a box?

A chatbot's cost is the loop: one full pass through the network for every token it writes. A decision model makes one pass and stops.

The model says it is 95% sure a headline is about sports. If the model is well calibrated, what does that mean?

Calibration is a promise about the whole pile, not about one headline. Any single 95% answer can still be one of the five it gets wrong.

You type a headline about a video game released last year, and several words show up as grey "not in dictionary" pills. What does the model do with those words?

A word with no id has no row in the embedding table, so it contributes nothing. This model's world ends in 2004, and it has no idea that it does.