Neuro 13
0 / 37 chapters finished

Chapter 9: Checkpoint: one neuron

What to remember, a mixed quiz, and your first build-it project.

Eight chapters ago, "neuron" was just a word. Since then you have dragged one around by hand, trained one, watched one fail, and fixed it with a team of them. Before the networks get bigger, this is the place to check that the foundations are solid.

Eight things to remember

  1. A network only does arithmetic, so everything it handles (pictures, text, sound) has to be turned into numbers first.
  2. A neuron multiplies each input by a weight, adds a bias, and squishes the total with sigmoid into a number between 0 and 1.
  3. A weight says how much an input matters, a negative weight argues against, and the bias is the neuron's lean before any input arrives.
  4. Training is a loop: predict, measure the error, and nudge every number that took part a little in the direction that shrinks the error.
  5. Training changes only the weights and biases, while the examples and the shape of the network are chosen by a human.
  6. One neuron's decision boundary is a single straight line, so it can learn only linearly separable rules: OR and AND yes, XOR never.
  7. Hidden neurons each draw their own line and an output neuron combines them, which lets a network make shapes that no single line can.
  8. Nobody tells hidden neurons what to mean, and backpropagation is how they get trained anyway: it passes the blame for each mistake backward through the network.

Mixed quiz

Six questions from all over Part 1. If one of them trips you up, the "why" under it tells you which idea to revisit.

A neuron has weight 1 = 2, weight 2 = −3 and bias = 0.5. Both inputs are 1. What does it say?

Inputs being on is not enough: what counts is the total after the weights have had their say. A total below 0 comes out of sigmoid below 0.5, which means NO. This is the arithmetic from chapter 3.

On example A the neuron's error is 0.6. On example B it is 0.05. The same inputs are switched on in both. Which example moves the weights more?

The recipe from chapter 4 is nudge = small step × error × input. A big mistake pulls hard, and an example that is nearly right barely pulls at all. That is why training slows down as it succeeds.

A friend says: "My network got smarter during training because it grew new connections, like a brain." What really changed?

The shape of a network is fixed by a human before training starts. Training only rewrites weights and biases. See chapter 5.

New rule: YES only when input 1 is on and input 2 is off. Can a single neuron learn it?

The test is never how complicated the rule sounds. It is whether one straight line can split the YES dots from the NO dots, as in chapter 6. A negative weight on input 2 does the job here.

A single neuron trained on XOR ends up answering 0.5 to everything. What does that tell you?

This is the wall from chapter 7. More steps or a new start change nothing, because the problem is the shape of the model, not the effort.

Two friends train the same hidden-layer network on XOR. Both networks get every row right, but their weights look nothing alike. What happened?

The weights matter completely. They are the whole model. But nobody assigns the hidden neurons their jobs, so two runs can share the work out differently and both be right. See chapter 8.

Build it: teach a neuron your own rule

Time to run the whole of Part 1 yourself, on a rule nobody has tried before, because you are about to make it up.

  1. Invent a house rule with two yes/no inputs and a yes/no answer. Three to get you started:
    • Sleepover: you can go if your room is clean AND your homework is done.
    • Bike: you may ride to school unless it is raining AND dark.
    • Teams: you and a friend each pick a shirt, 0 for red or 1 for blue. You are on the same team if the shirts match.
  2. Write its truth table: four rows, one for each pair of inputs.
  3. Draw the four dots on paper and predict: can one straight line split the YES dots from the NO dots?
  4. Run the experiment and find out.

Here are the three starter rules as tables. Their usual names are AND, NAND ("not and") and SAME.

input 1input 2Sleepover (AND)Bike (NAND)Teams (SAME)
00011
10010
01010
11101

Before you run anything: which of the three will one neuron fail to learn?

Locked in. Now run all three below and see whether you were right.

▸ run real python

Teach a neuron your own rule

This is my_rule.py: one neuron with two weights and a bias, and a four-row table called ANSWERS near the top. It starts as the sleepover rule. Press run and read the output from top to bottom: the error shrinking, the three stored numbers, the four predictions, and a map of the neuron's answer across the whole graph paper.

Then change the four answers to the bike rule (1, 1, 1, 0) and run it again. Then the teams rule (1, 0, 0, 1). Then your own. Change RULE, INPUT_1 and INPUT_2 to match, so the printout speaks your language.

my_rule.py Python loads on first run
(click "run" to execute)

What you should have seen

Sleepover is learned. The script ends with weight 1 = 6.52, weight 2 = 6.52 and bias = −9.87. One input alone adds 6.52, which cannot beat a bias of −9.87. Both together add 13.04, which can. On the map, the 9s are squeezed into the top-right corner and a straight edge cuts them off from the 0s.

Bike is learned too, and look at the numbers: −6.52, −6.52 and 9.87. They are the sleepover numbers with every sign flipped. Flip the signs and every YES becomes a NO, which is all the "N" in NAND means.

Teams gets stuck. The error stops at 0.2500, all three stored numbers end at zero, every row says "shrug," and the map is a wall of 5s. You have met this wall before. Swap YES and NO in the secret door from chapter 7 and you get the teams rule: its YES dots sit on opposite corners too, and no straight line can split them.

Here is the surprising part. With two yes/no inputs there are only 16 possible rules, because each of the four rows is either YES or NO and 2 × 2 × 2 × 2 = 16. We ran this script on every one. A single neuron learns 14 of them. The only two it cannot learn are XOR and SAME. If the rule you invented got stuck, it was one of those two in disguise.

Challenge

Invent a rule where one of the inputs makes no difference to the answer. Before you run it, predict what that input's weight will be at the end. Then find a rule whose two weights end up with opposite signs.

Next: more of everything

Everything in Part 1 had two inputs, one yes-or-no answer, and four examples you could hold in your head. Part 2 lets go of all three. In chapter 10 a network picks from a list instead of answering yes or no. Then come deeper networks and real, messy data, and in chapter 14 you will step through the blame-passing that makes hidden neurons trainable, one number at a time. The neuron itself does not change. There are just going to be a lot more of them.