Neuro 13
0 / 37 chapters finished

Chapter 17: Stacking hidden layers

Eight neurons, one layer or two: the spiral showdown.

Every network you have trained so far has had exactly one hidden layer: input → hidden → output. Real networks stack more: input → hidden → hidden → output, and the big ones go dozens of layers deep. The number of layers is called the network's depth. The wiring rule does not change at all. The outputs of hidden layer 1 simply become the inputs of hidden layer 2. Every neuron in layer 2 does the same multiply-add-squish from chapter 3; it just works on layer 1's answers instead of on the raw input.

Why would that help? The network in chapter 8 built its answer out of straight lines, because each hidden neuron looks at the raw inputs, and one neuron's question is always a line. A second hidden layer builds shapes out of those shapes: layer 1 draws lines, and layer 2 bends and combines them into curves and pockets.

Here is a puzzle that makes depth visible: two spirals, tangled around each other. Every dot is a training example. Its position on the graph paper is the input (two numbers, x and y), and its color is the right answer. This is the dots-on-graph-paper picture from chapter 6, all grown up. If XOR was the puzzle one neuron could not solve, the spiral is XOR's final boss.

You have 8 hidden neurons to spend on the spiral. You can put all 8 in one layer, or split them into two layers of 4. Which arrangement gets further?

Two layers of 4, at least from the starting weights this page uses. Eight straight lines added together are not enough to follow the curl. Four lines fed into four more neurons are. Train both below and compare.

▸ try it

Same neurons, one layer or two

Pick a network and press "train 2,000" twice. The background shows the network's current opinion about every point on the map; the dots are the training examples it has to match. Watch the boundary bend as training runs, and write down the accuracy after 2,000 and 4,000 epochs. Then try the next network. (Training here uses Adam, the method from chapter 13.)

1 layer × 8
2 layers × 4 + 4
1 layer × 16
2 layers × 8 + 8
epochs: 0 accuracy: ? stored numbers: ? starting weights: the book's

What each hidden neuron is looking for

Each little map shows where on the graph paper one hidden neuron gets excited (blue) or pushes back (red). Layer 1's maps are always stripes: straight-line questions. Layer 2's maps are bent: shapes built out of layer 1's stripes.

The model, as stored

What layer 2 actually does

Each network begins from the same fixed set of "random" starting weights every time you select it, so your numbers should match these.

networkstored numbersafter 2,000 epochsafter 4,000 epochs
1 layer × 83375%79%
2 layers × 4 + 43784%97%
1 layer × 166597%98%
2 layers × 8 + 810598%99%

Look at the first two rows. Both networks have 8 hidden neurons and nearly the same number of stored numbers. The one-layer network stalls, and more training barely helps: it is still at 83% after 8,000 epochs. The two-layer network finds the spiral.

The little neuron maps show why. In the one-layer network every map is a stripe, and the output can only add eight stripes together. In the two-layer network, layer 1 makes four stripes and layer 2 turns them into four bent shapes. Bent shapes are what a spiral is made of, and only a second layer can make them.

The honest scoreboard

One result in that table should surprise you: a single layer of 16 neurons solves the spiral too. That is a famous result called the universal approximation theorem at work: one hidden layer can fit almost any shape, as long as it is wide enough.

And one run proves little, because the starting weights matter. Press "new random start" and train again, and the same network can end up somewhere different. So we ran each network from 60 random starts, for 8,000 epochs each:

  • 1 layer × 8 never reached 95%. Its best run got to 93%.
  • 2 layers × 4 + 4 reached 95% in 29 of its 60 starts.
  • 1 layer × 16 reached 95% every single time.
  • 2 layers × 8 + 8 reached 95% in 49 of its 60 starts. The other 11 got stuck near a coin flip and stayed there.

Two things are true at once. Stacking lets the same eight neurons do something they could not do side by side. And deep networks are touchier: they are the only ones here that ever got properly stuck. A good share of modern deep learning (ReLU from chapter 11, and the tricks you will meet in chapter 18) exists to make deep networks train reliably.

So why is every serious network deep?

Because depth lets a network reuse what earlier layers built, and for some shapes that is enormously cheaper. The spiral is too small to show it. Here is the cleanest example there is.

Take one input, a number between 0 and 1, and a layer of just two ReLU neurons. With the right weights, that pair turns a rising line into a tent: up, then back down. It has folded the number line in half, like folding a strip of paper. Now feed the tent into a second, identical pair. It folds everything it is given, so out come two tents. A third pair folds those. Every layer reuses all the folds made before it.

Each new layer adds just two neurons. What happens to the number of straight pieces in the zigzag every time you add a layer?

It doubles. The new layer folds every piece the earlier layers made, not just one of them. That is what "reuse" means here.

▸ try it

Fold the line

Slide to add layers and watch the zigzag. Left to right is the input, from 0 to 1; up is the network's output. These weights were set by hand, not learned: in every layer, neuron A fires on its input, neuron B fires on its input minus 0.5, and the pair hands on 2 × A − 4 × B.

1
neurons in this network: 2 straight pieces it draws: 2 neurons one layer would need: 2

Seven layers is 14 neurons, drawing 128 straight pieces. A single hidden layer of ReLU neurons can only put one bend in the line per neuron, so it needs about one neuron for every piece: 128 of them. Keep going and the gap gets absurd. Twenty layers is 40 neurons. One layer would need more than a million. This is a proven result, not a lucky run (Telgarsky, 2016).

Honesty note. The folding weights were chosen by a person. The widget shows what depth can express with very few neurons, and training does not always find answers this neat. But it is the real reason for depth. When a problem has structure that can be reused (edges into parts and parts into objects, or letters into words and words into meaning), each layer builds on everything below it, while a shallow network has to pay for every piece separately.

Two footnotes for the curious. The hidden activation in the spiral widget is tanh, sigmoid's cousin that outputs −1 to +1 instead of 0 to 1 (that is why the little maps have both blue and red). And the spiral is the classic demo from TensorFlow Playground, where you can stack up to six hidden layers. This book has its own version in chapter 33.

Each layer builds on the one before it, so a deep network can reuse what it has already made: lines, then shapes made of lines, then shapes made of shapes. A wide enough single layer can often do the same job, but for layered problems it pays for every piece separately, and the bill grows out of reach.

Challenge

Check our survey. Pick "2 layers × 4 + 4," press "new random start," train 8,000 epochs (four presses), and note whether it passed 95%. Do that ten times. Then do the same for "1 layer × 8." Last, keep pressing "new random start" on "2 layers × 8 + 8" until you find a start that gets stuck near 50%. What do its layer-2 neuron maps look like compared with a run that worked?

In a network with two hidden layers, what do the neurons in the second hidden layer receive as inputs?

Each layer feeds the next. Layer 2 never sees x and y directly; it sees what layer 1 made of them, which is what lets it build shapes out of shapes.

Eight neurons in one layer stalled on the spiral. The same eight neurons as two layers of four solved it. What is the best explanation?

Same data, same training, same neuron count. The difference is the arrangement. And "always" is too strong: with 16 neurons, the one-layer network solved the spiral as well.

In the folding widget, why does a deep network need so few neurons compared with a single layer?

The neurons are identical, and nothing in that widget was trained at all. The saving comes purely from reuse: every new layer doubles the work already done, and a single layer has nothing to reuse.