Neuro 13
0 / 37 chapters finished

Chapter 7: When one neuron isn't enough

Four dots that no single neuron can separate, and the research slump they came to stand for.

Let's try a harder rule. Imagine a secret door with two buttons:

The door opens if EITHER the left button OR the right button is pressed. But if both buttons are pressed at the same time, the door stays shut.

This is the "exactly one" rule, the XOR from chapter 6. Here's the data:

left button?right button?door opens?
000 (no buttons)
101 (left only)
011 (right only)
110 (both buttons)

And here it is on graph paper:

right button=1 =0 left=0 left=1 YES NO NO YES

A single neuron draws one straight line. Can you draw one line that puts both YESes on one side and both NOs on the other? Try it on paper. You can't: the rule is not linearly separable. So what will a neuron do when you train it on this anyway?

You are about to train one neuron on the secret door for 1,000 steps. Where will its four predictions end up?

Most people guess three out of four. Press the button below and see what really happens.

▸ try it

Watch a single neuron fail

Same trainer as in chapter 4, new data. Press "train 1,000 steps" and watch three things: the four predictions, the three stored numbers in the dark panel, and "avg error" (the errors on the four examples, each squared, then averaged). Then press it again. And again.

The model that can't win

Same three stored numbers as the screen-time model. No values of these three numbers can solve XOR.


  
epoch: 0 avg error: ?

What just happened

Every prediction landed on 0.500, all three stored numbers sank to zero, and the average error stopped at 0.2500. More training changes nothing. Press "reset weights" and try a different random start: same result, every time.

Why a shrug, and not three out of four? Training doesn't count ticks. It shrinks the average error. A line that gets three corners clearly right has to get the fourth clearly wrong, and being badly wrong once costs as much as being half-wrong four times. Meanwhile, at 0.5 the pulls cancel exactly: the two YES examples tug every number up, the two NO examples tug them down just as hard, and nothing moves. So the neuron settles on answering 0.5 to everything. (When a prediction reads 0.500, the widget shows a ? where the ✓ or ✗ would be, because a shrug is on neither side.)

One neuron draws one line, so no amount of training will ever teach it XOR. When the shape of a model cannot fit the problem, more training is not the fix. A different shape is.

▸ try it

Try to draw the line yourself

Same line-drawing widget as in chapter 4, with the secret-door labels. Drag the sliders. The YES dots are on opposite corners now. The best you can do is 3 out of 4. Try as long as you want.

The most famous small puzzle in AI

In 1958 a researcher named Frank Rosenblatt unveiled the perceptron: a machine that learned from examples by adjusting its weights, much like the neuron you have been training. The excitement was enormous. The New York Times reported that the U.S. Navy, which paid for it, expected a machine that would one day "walk, talk, see, write, reproduce itself and be conscious of its existence."

In 1969 two MIT researchers, Marvin Minsky and Seymour Papert, published a book called Perceptrons. With careful math, it proved that a perceptron with a single layer of adjustable weights has hard limits, however long you train it. XOR, the puzzle you just played with, is the simplest example.

The story is often told as "the book killed neural networks." The truth is messier. Money and attention were already drifting toward other kinds of AI, and historians still argue about how much the book mattered. But for roughly the next fifteen years, very few researchers worked on neural networks. A stretch like that, when the excitement and the funding drain away, is known as an AI winter.

Here's the twist. Minsky and Papert never proved that bigger networks were hopeless. It was already known that a network with a middle layer could compute XOR. What nobody had was a good way to train the middle layer. The method that does it is called backpropagation. It was worked out more than once in the 1970s, by Seppo Linnainmaa and Paul Werbos among others, and mostly ignored. Then in 1986 David Rumelhart, Geoffrey Hinton and Ronald Williams published a short paper showing that it lets a middle layer learn useful things by itself, and neural networks came back to life. It trains the networks in this book and every large AI system you have heard of. You will step through it by hand in chapter 14.

▸ run real python

Watch a real single-layer perceptron give up

This is neuro2.py: the script from chapter 4, the same neuron with its two weights and its bias, with the secret-door data swapped in. Every 2,000 rounds it prints its mean squared error, or MSE: square each example's error, then average them. It is the script's single score for "how wrong am I overall?", and the same number the widget above calls "avg error." Training is supposed to shrink it. Here it drops to 0.2500 and stays there. Every prediction ends at 0.500, and the weights and the bias end at zero.

neuro2.py Python loads on first run
(click "run" to execute)
Challenge

Try to beat XOR with one line anyway. Change the sliders in the line-drawing widget, then explain why your best answer still puts at least one corner on the wrong side.

A single neuron cannot learn the secret-door rule. What would actually fix that?

The limit is the shape, not the effort. One neuron can only draw one line, and no single line splits XOR, wherever it starts and however long it trains.

After training on XOR, the neuron answered 0.5 to everything instead of getting three out of four right. Why?

Nothing resets and nothing is confused. Training follows the error downhill, and for this puzzle the bottom of the hill is a shrug.

What did Minsky and Papert's 1969 book actually prove?

The proof was about single-layer perceptrons. Networks with a middle layer were not ruled out. They were waiting for a training method, which arrived as backpropagation.