Chapter 10: Picking from a list
From yes/no to choosing one of many.
So far every neuron has answered yes or no. But plenty of questions have more than two answers. Take "which snack should I grab?" Here is the house rule, as a table:
| hungry? | after school? | weekend? | snack |
|---|---|---|---|
| 0 | 0 | 0 | fruit |
| 1 | 0 | 0 | fruit |
| 0 | 1 | 0 | chips |
| 1 | 1 | 0 | chips |
| 0 | 0 | 1 | cookie |
| 1 | 0 | 1 | cookie |
| 0 | 1 | 1 | cookie |
| 1 | 1 | 1 | cookie |
Study the table for a moment. One of the three inputs makes no difference at all to the snack. Which one?
Hungry. Compare the rows in pairs: every time only "hungry" changes, the snack stays the same. Nobody will tell the network this. It has to work out for itself that one of its inputs is a decoy.
One output neuron per answer
Three possible answers means three output neurons: one for fruit, one for chips, one for cookie. Each possible answer is called a class, and each output neuron produces a score for its class. A higher score means "I think it's this one."
Scores can be any numbers, positive or negative, which makes them awkward to read. So the last step is a function called softmax, which turns a list of scores into probabilities that always add up to 1:
softmax_i = e^(z_i) / (e^(z_1) + e^(z_2) + e^(z_3))
Here z₁, z₂, z₃ are the three scores, and e is a famous fixed number, about 2.718. e^z means "e raised to the power z": e^0 = 1, e^2 ≈ 7.4, and e to a negative power is a small positive fraction. Two things matter here: e^z is always positive, and it grows fast as z grows.
So the top of the fraction makes every score positive and stretches the gaps between them. The bottom divides by the total, which is what makes the answers add up to 1.
softmax([2.0, 1.0, 0.1]) → [0.66, 0.24, 0.10]
Read that as "66% fruit, 24% chips, 10% cookie." The network's pick is whichever is highest.
Keeping score: how surprised was the model?
To train this network we need a single number that says how wrong it was, so training has something to shrink. That number is called the loss. The single neuron in chapter 4 used the simplest one, squared error: take truth − prediction, square it, and average over the examples.
A list of probabilities gets a better score. First write the right answer as a list too: [1,0,0] for fruit, [0,1,0] for chips, [0,0,1] for cookie. Exactly one slot is 1 and the rest are 0. That is called one-hot encoding.
Now look at just one number: the probability the model gave to the right answer. If it said 90% for the snack that turned out to be correct, it is barely surprised. If it said 10%, it is shocked. The cross-entropy loss turns that surprise into a number:
loss = −log(probability the model gave to the right answer)
(log here is the natural logarithm, the ln button on a calculator. It is the undo button for e^.) You never need to work it out by hand. What matters is its shape:
| probability given to the right answer | loss | in words |
|---|---|---|
| 0.90 | 0.11 | barely surprised |
| 0.50 | 0.69 | coin-flip surprised |
| 0.10 | 2.30 | shocked |
A probability of exactly 1 costs 0. As the probability heads toward 0 the loss climbs without limit: 0.01 costs 4.61. So being confidently wrong is punished much harder than being unsure. The network's loss is the average over all its training examples.
The scores are [2.0, 1.0, 0.1]. You add 1 to every score, making them [3.0, 2.0, 1.1]. What happens to the probabilities?
Nothing changes. Adding 1 to a score multiplies its e^z by e, and when every term on the top and bottom of the fraction is multiplied by the same number, it cancels. Softmax only cares about the gaps between scores. Check it with the sliders below.
Softmax by hand
Drag the three scores and watch softmax turn them into probabilities. Then pick which snack was actually right, and read off how surprised the model is.
Keeping score
Suppose the right answer was:
Three things worth trying:
- Make all three scores equal. You get 0.33 each, whether the scores are all −5 or all +5. Equal scores mean "no idea."
- Slide every score up by the same amount. The bars do not move, just as the prediction box said.
- Drag the right answer's score down to −5 and the others up to +5. The loss shoots past 10. Now drag the right answer's score up instead: the loss sinks toward 0 but can never go below it.
A trained snack picker
Training this network is the same loop as before: run the examples through, measure the loss, nudge every weight in the direction that lowers it, repeat.
The trained snack picker
This network (3 inputs, 6 hidden neurons, 3 outputs) was trained in your browser when the page loaded, starting from random weights. Toggle the inputs and watch the scores and probabilities change. Then toggle hungry on its own and see how little happens.
What just happened
After 5,000 rounds of training the right snack gets a probability above 0.99 on every row, so the bars look almost like a plain yes or no. The scores underneath tell the fuller story: the winning score sits several points above the others, and softmax stretches that gap into near-certainty. The exact scores change each time you reload the page, because training starts from different random weights, but the picks do not.
And the decoy? Toggling "hungry" barely moves anything. The network was never told that input was useless. All eight rows demand the same snack whether hungry is on or off, so training pushed the network until that input stopped changing its answer.
This pattern (one output neuron per class, then softmax, trained with cross-entropy) is how a network reads handwritten digits (10 classes; you will train one in chapter 27) and how a chatbot picks its next word-piece (tens of thousands of classes or more; see chapter 23). Same idea, more output neurons.
Run the real multi-class snack picker
This is neuro4.py: the same network as the widget, in numpy. Watch the loss it prints. It starts near 1.0 (a model that knew nothing and said one-third for each snack would score −log(1/3) ≈ 1.10) and falls to about 0.0002. Then it prints the probabilities for all eight rows.
To choose between several answers, give each one its own output neuron, let softmax turn the scores into probabilities, and train by punishing the model for how surprised it was by the right answer.
Add a fourth snack, smoothie, on paper. How many output neurons does the network need now, and what does the one-hot label for smoothie look like? Then find the line in neuro4.py that computes the loss and match each piece of it to the formula above.
A network's three output scores are all exactly the same. What does softmax give?
Softmax only looks at the gaps between scores. No gaps means no preference, so the probability is shared equally, whatever the scores are.
On one example the model gave the right answer a probability of 0.9. On another it gave the right answer 0.1. How do the two losses compare?
0.11 against 2.30, about twenty times bigger. Cross-entropy punishes confident mistakes far more than it rewards confident hits, which pushes the model hardest exactly where it is most wrong.
The classes are [fruit, chips, cookie]. What is the one-hot label for chips?
One slot per class, with a single 1 in the right class's slot. A plain "2" would wrongly suggest chips is somehow "twice" fruit.