Neuro 13
0 / 37 chapters finished
Part 1One neuron11 min read

Chapter 4: Training the neuron

Show it examples, nudge the weights, repeat.

Let's teach our one neuron to decide "can I have screen time?"

Rule: screen time is allowed if you practiced piano OR you finished your homework. Here are the examples we'll feed it:

piano done?homework done?screen time allowed?
0 (no)0 (no)0 (no)
1 (yes)0 (no)1 (yes)
0 (no)1 (yes)1 (yes)
1 (yes)1 (yes)1 (yes)

The training loop is the same idea every time:

  1. Start with random weights (the neuron has no clue)
  2. Run all 4 examples through the neuron and see what it predicts
  3. For each one, work out how wrong it was
  4. Nudge every weight a tiny bit in whichever direction would have made the answer slightly more right
  5. Repeat thousands of times

Watch one nudge happen, with real numbers

"Nudge every weight in the right direction" sounds like magic until you watch a single nudge happen. So let's do one, slowly, by hand.

Say the neuron just started, with random numbers: weight 1 (piano) = −0.5, weight 2 (homework) = 0.2, bias = −0.1. We show it the example piano = 1, homework = 0, right answer = YES:

total      = (−0.5 × 1) + (0.2 × 0) + (−0.1)  =  −0.6
prediction = sigmoid(−0.6)                    ≈   0.35
truth      = 1
error      = 1 − 0.35                         =  +0.65   (too low!)

The prediction is too low, so we want the total to come out bigger next time. Go through the three numbers one at a time:

  • Weight 1 (piano): piano was ON (1), so this weight took part in the total. Prediction too low → nudge it UP. (It was −0.5, dragging the answer down, which is exactly the wrong job for it. The nudge starts fixing that.)
  • Weight 2 (homework): homework was OFF (0), so this weight got multiplied by zero. It had no effect at all on this prediction, so it gets no nudge. You can't blame a weight that wasn't involved.
  • Bias: the bias is always involved (it gets added every single time, like an input that's stuck at 1). Prediction too low → nudge it UP.

And if the prediction had been too high? Same logic, flipped: every weight whose input was on gets nudged DOWN. The recipe for one weight is:

nudge = small step × error × that weight's input

The error sets the direction (too low → up, too high → down). The input decides who takes part (an input of 0 wipes its weight's nudge out). And the small step keeps each change tiny, so no single example can yank the neuron around.

Honesty note: the fast trainer further down this page and the book's Python scripts multiply in one more factor, the slope of the sigmoid curve at the current output. Slopes are what chapter 13 and chapter 14 are about. The direction logic is exactly what you just read.

In the widget below, the same neuron starts with the same three numbers. Suppose you pick the example piano = 0, homework = 0 → NO. The neuron says 0.48, which already counts as a NO. What will one nudge on that example do?

The truth is 0 and the neuron said 0.48, so it is still 0.48 too high. Being on the right side of 0.5 is not the same as having no error. Both inputs were 0, so both weights sit this one out, and only the bias gets nudged: from −0.10 to −0.27. Pick that example and check.

▸ try it

Do one nudge yourself

Pick a training example, read the error, then press "nudge the weights" and watch each number move, or refuse to move, for the reasons above. Then press "nudge once on each example" a dozen times or more, which is closer to what real training does. Keep your eye on the one NO example in the list at the bottom. The dark panel is the model itself, the actual numbers the computer stores, being rewritten live.

piano=0, homework=0 → NO
piano=1, homework=0 → YES
piano=0, homework=1 → YES
piano=1, homework=1 → YES

What the neuron says right now

The nudge this example asks for

The model itself, as the computer stores it

This is the entire model: three numbers. Every nudge rewrites it in place, and freshly changed numbers light up. When someone says "I downloaded a model," they mean a file holding a much, much longer list like this.


  
nudges so far: 0 w1: -0.50 w2: 0.20 bias: -0.10

What just happened

Did the NO example surprise you? Press "start over" and then count your presses of "nudge once on each example." The NO example begins on the correct side, at 0.475. The first rounds of nudges push it the wrong way, up to 0.635 by the fourth press, before it turns around. It is not back on the correct side until the 16th press, and that is the first moment all four examples have a ✓.

Three of the four examples want a YES, and every one of them pushes the bias up. Only the NO example pushes it down, and at first it loses. But the two weights are growing too. Once they are big enough to produce a YES without help, the YES examples stop leaning on the bias, their pushes fade, and the NO example drags the bias back below zero. Training is a tug-of-war between the examples, and the numbers settle where the pulls balance.

The trainer below plays the same tug-of-war faster. It adds up the nudges from all four examples, applies them together, and repeats.

The trainer starts from random weights, different every time you press "reset weights." If you reset and train 1,000 steps a few times over, how will the final predictions compare?

For this rule every start gets pulled to the same place. The random start is forgotten, and what remains is what the examples taught. Try it.

▸ try it

Train the neuron yourself

"Step once" runs all four examples through the neuron and applies their nudges. One pass through all the examples is called an epoch. "Train 1,000 steps" does a thousand epochs in a blink. Watch the four predictions crawl toward the right answers, then reset and do it again.

The model being rewritten

Training never touches the four examples. It only rewrites these three stored numbers, over and over, until they encode the rule.


  
epoch: 0 w1: ? w2: ? bias: ?

What just happened

After 1,000 steps the predictions read about 0.08, 0.95, 0.95 and 1.00, whatever the random start. (We tried 2,000 different starts. Every one had all four examples right within 100 steps, and after 1,000 steps they agreed to two decimal places.) They never reach exactly 0 or 1, because sigmoid only gets there with an infinitely large total. Keep training and they keep inching closer, with smaller and smaller nudges.

Nobody told the neuron "if piano OR homework is done, then yes." It got there by being slightly less wrong, over and over. Everything from here on is this same loop, made bigger.

Do not expect every network to forget its starting point like this one. In chapter 8, different random starts will find different solutions. And if you would like to watch this loop animated on a bigger network, try the second video in 3Blue1Brown's neural networks series.

Training is a loop: predict, measure the error, nudge every number that took part a little in the direction that shrinks the error, repeat. Nobody types the rule in. It ends up stored in the weights and the bias.

▸ try it

Draw the line yourself

Here is another way to see what those three numbers do. Each example is a dot, marked Y or N. The dashed line is the neuron's decision boundary, the place where its answer flips from NO to YES: the blue side says YES and the pink side says NO. Drag the sliders to move the line. A dot turns green when it lands on the correct side.

Try to get all 4 correct. If you get stuck, weight 1 = 1.0, weight 2 = 1.0, bias = -0.5 is one good answer, and there are many others. Chapter 6 explains why the boundary is always a straight line.

▸ run real python

Run the real script, in your browser

This is neuro1.py, the first of the book's Python scripts: the same neuron, the same four examples and the same training loop, in real Python with real numpy. (Build your own explains how to download all the scripts and run them on your own computer.) Edit anything you like, then press run.

It trains for 10,000 epochs, then prints the three numbers it ended up with (weights of 8.67 and 8.67, and a bias of −4.10) and its four predictions: 0.016, 0.990, 0.990 and 1.000. That is closer to 0 and 1 than the widget got, because it trained for ten times as long.

Heads up: the first time you press "run" on a page, your browser downloads Python itself (about 8 megabytes), so it needs an internet connection and a little patience. After that, every run on the page starts right away.

neuro1.py Python loads on first run
(click "run" to execute)
Challenge

Change y so that screen time is allowed only when piano and homework are both done. Before you run it, predict which of the three stored numbers will have to change the most. Then run it and compare with the numbers the OR rule gave you.

Now delete the line bias += adjustments.sum(), so the bias stays stuck at 0, and run both rules again. What happens to the first row? Which rule can the neuron no longer learn, and why?

The neuron's prediction was too high on an example where piano was 1 and homework was 0. Which numbers get nudged?

Too high means push the total down. Only the numbers that took part get moved: weight 1 (its input was on) and the bias (always on). Weight 2 was multiplied by 0, so it had nothing to do with the mistake.

Why is each nudge kept small?

Every example pulls the same few numbers in its own direction. Small steps let the pulls balance out. Small steps are slower, not faster, which is why training takes thousands of rounds.

After training, where is the rule "piano OR homework" stored?

There is no rule written anywhere and the examples are not consulted when the neuron answers. Three numbers, tuned by training, are the whole rule.