Neuro 13
0 / 37 chapters finished

Chapter 12: Meal guesser

A network that has to combine clues.

The snack picker had three classes and three tidy yes/no inputs. Real classifiers usually have more classes and messier inputs. So here is the snack picker's big sibling: a small network that guesses which meal you are describing from menu-style clues.

Turning words into inputs

A neural network wants numbers, not words. In chapter 10 you used one-hot encoding for the labels. This time it goes on the inputs too. A word choice like course = dessert becomes three yes/no inputs:

course_main = 0
course_snack = 0
course_dessert = 1

Same for temperature, taste, diet, and base. The network never receives the word "pastry." It receives base_pastry = 1 with the other four base inputs set to 0. Five word choices become 14 inputs in all.

mealcoursetemperaturetastedietbase
apple piedesserthotsweetvegetarianpastry
sausage rollsnackhotsavorymeatpastry
cottage piemainhotsavorymeatpotato
baked potatomainhotsavoryvegetarianpotato
spaghettimainhotsavorymeatpasta
mac & cheesemainhotsavoryvegetarianpasta
ice creamdessertcoldsweetvegetariandairy
fruit saladdessertcoldsweetvegetarianfruit

Look closely at the table: it is full of near-misses. Cottage pie and baked potato match on everything except diet. So do spaghetti and mac & cheese (the spaghetti is bolognese; that is where the meat is). Ice cream and fruit salad differ only on base. And apple pie and sausage roll share a pastry crust while disagreeing about almost everything else. No single clue gives the answer away. The network must combine them.

Once the network is trained, you will describe a meal as main, hot, savory, pasta and leave the diet unknown. What will it say?

The two pasta dishes share it. Softmax always gives an answer, and the clues you did give fit both pasta dishes and rule out the rest. Try it below.

▸ try it

Train a meal guesser

Press "train 1,000" and check that train accuracy reaches 1.000 (if it stalls lower, press it again or reset: every run starts from different random weights). Then set the traits and watch the probability bars.

The experiment from the prediction box: set course = main, temperature = hot, taste = savory, base = pasta, and leave diet unknown. Then pick a diet and watch one meal take over.

Traits

Network shape

epoch: 0 train accuracy: ?

Prediction

How the clues become numbers

What training saved: the model itself, live

This is all of the trained model: 192 numbers for the current network shape. Press the train buttons and watch them get rewritten; freshly changed numbers light up. The grey // notes are not part of the model. They are labels so you can find your way around the grid.

What just happened

With the diet missing, most of the vote goes to the two pasta dishes. How they split it changes from run to run: sometimes close to even, sometimes more like 30/70, and now and then a third meal sneaks in a share. That is worth noticing. The network was only ever trained on complete meals, so nothing in training pinned down what it should say about half a meal. It was left to improvise. Fill in the diet and you are back on ground it was trained on, and one bar takes over.

Inside, the hidden neurons have become detectors for combinations of clues. One might respond to cold + sweet, another to pastry, another to main + pasta. Nobody assigned those jobs. Training nudged the weights until useful patterns appeared, and a different run will hand the jobs out differently. Working out what the hidden neurons in real models have learned is a research field of its own, called interpretability.

Press "show learned numbers" in the widget if you have not yet. This is the grown-up sibling of the dark "model" panels you have been watching since chapter 4. When someone says "I downloaded the weights," a much longer list like this one is what they got.

How many hidden neurons? That number is chosen by the builder before training starts. Training changes the weights and biases; it never adds neurons. A setting like this, picked by a person rather than learned, is called a hyperparameter. Try 2: the network usually gets stuck below 1.000, because two hidden neurons are too few to tell eight meals apart. Then try 8 or 12, which almost always get there. Bigger is not automatically better, though. A network with lots of spare room can simply memorize its training examples instead of finding the pattern behind them. That is called overfitting, and chapter 16 lets you watch it happen.
▸ run real python

Run the meal guesser

This is neuro6.py: the same network in numpy, with 10 hidden neurons. It prints its loss and accuracy as it trains, then its guess for each of the eight meals, then its guess for one mixed-up meal that is not in the table: a cold, sweet, meaty pastry snack. It leans toward sausage roll, at about 80%. There is no right answer for that one. The model is showing you which training meal the nonsense meal resembles most.

neuro6.py Python loads on first run
(click "run" to execute)

Words go into a network as one-hot inputs, and no single input has to carry the answer. The hidden layer learns which combinations of clues matter, and the output probabilities show how sure the network is.

Challenge

Add lasagna on paper. Write out its five traits: they come out identical to spaghetti's row. What can a network do when two different answers have exactly the same clues? What new trait would you add to the table to fix it?

Why does "base" get five separate yes/no inputs instead of one input numbered 1 to 5?

If pastry were 1 and fruit were 5, the network would be told that fruit is "five times pastry" and that potato sits between pastry and pasta. One-hot inputs say only "it is this one, and not the others."

You describe a hot, savory pasta main and leave the diet blank. The network splits its vote between spaghetti and mac & cheese. What does that tell you?

A split vote is the honest answer when the evidence fits more than one meal. A model that was 100% sure here would be worse, not better.

Who decides how many hidden neurons the meal guesser has?

The number of hidden neurons is a hyperparameter: a choice made by the builder. Training only adjusts the weights and biases inside the shape it was given.