Chapter 33: The playground
Pick the data, the layers and the knobs. Break it. Fix it.
Every network in this book so far came with its settings already chosen. Somebody picked the data, the number of layers and the learning rate, and you pressed train. In this chapter nobody picks anything for you.
Below is a lab bench with every part from Parts 1 and 2 on it: dots on graph paper, neurons, hidden layers, activations, a learning rate, a test set. You choose the dots. You build the network. You press play and find out whether your guess was right.
Nothing in the lab is pre-trained. Each time you press play, a network with random weights starts learning in your browser, using the backpropagation you stepped through in chapter 14.
One color code runs through the whole lab: orange means a positive number and blue means a negative one. An orange dot is an example whose right answer is "orange". The network answers "orange" wherever its output neuron's weighted sum comes out positive, and "blue" wherever it comes out negative.
The lab opens with a circle of orange dots inside a ring of blue ones, and a network with one hidden layer of 4 neurons. Once it has learned to separate them, what shape will its orange region be?
Press play and look. Each hidden neuron can only draw one straight line (that was the lesson of chapter 6), so four neurons can fence the circle in with at most four sides. The tanh squish rounds the corners, but the shape never becomes a true circle.
Build it, break it, fix it
Work from top to bottom: choose some dots, shape a network, then press play. Change anything you like while it runs.
1 · The dots
Click or drag on the big map to drop dots. Every fourth dot is kept back as a test dot.
2 · The network
3 · What it thinks
Training knobs
You can move the sliders while it trains. In the diagram, squares are neurons and lines are weights: orange for positive, blue for negative, thicker for bigger.
Over time training test
Reading the lab
- The big map is the graph paper from chapter 6. Each dot is one example: its position (two numbers, x and y) is the input, and its color is the right answer. The background shows what the network would answer at every point on the paper.
- The diagram is the network itself. Each square is one neuron, drawn as a tiny copy of the map that shows where that neuron's output is positive or negative. Squares in the first hidden layer are always split by a straight line. Squares further along are bent, because they are built out of the squares before them. Point at a square to enlarge it, and click it to pin it to the big map.
- The two scores. The training score is the share of the training dots the network gets right. The test score uses 200 more dots from the same pattern that it never trains on. Tick "show the test dots" to see them.
- The buttons. One step is one epoch: one pass through all the training dots. Reset goes back to epoch 0 with the same starting weights. New start rolls new random ones.
- The knobs. The learning rule is the recipe for nudging the weights: plain gradient descent, or Adam, the version with a memory from chapter 13. The learning rate is the step size. Batch size is how many dots the network looks at before each nudge. The shape of the network, meaning how many layers it has and how many neurons are in each, is called its architecture, and the lab shows how many stored numbers that shape needs.
Honesty note, in two parts. First, the noise slider gives some training dots the wrong color and draws a black ring around each one. The test dots always keep their true color, so the test score tells you how well the network knows the real pattern. That is the same choice chapter 16 made, and real test sets are not so tidy. Second, the lab opens on plain mini-batch gradient descent, with no extras, so that every knob does exactly what its chapter said it does. That makes it slower and fussier than the Adam that chapter 17 and chapter 32 use. Adam is one click away, and mission 6 compares the two.
Lab notebook
Ten missions, each tied to a chapter you have read. Every one has a button that sets the lab up for you and a box to tick when you have cracked it. Your ticks are remembered on this device: 0 of 10 done.
Under each mission is a note of what we found when we ran it. Try it yourself before you open the note. Our numbers come from the exact setup behind each button, and that setup always starts from the same random weights, so if you press play without changing anything you should see exactly our numbers. That takes some care. Browsers disagree in the last digit of functions like tanh, and training magnifies tiny differences into big ones, so the lab does that arithmetic itself, the same way everywhere. Press new start and your numbers will differ. How much that matters is the lesson of mission 8.
-
The lonely neuron
Take away every hidden layer and you are left with the single straight-line neuron from chapter 6. Which of the five sets of dots can it solve, and what do those have that the others lack?
What we found
Only the two blobs: 100% within 2 epochs. On the two moons its test score stalled at 92%. After 300 epochs it was at 43% on the ring, 49% on the XOR corners and 57% on the spirals. Its line keeps swinging about on those three, but it never does reliably better than a coin flip. The blobs are the only set where one straight line can separate the colors. That is what linearly separable means. With no hidden layers the lab draws the neuron's line as a dashed line, so you can watch it swing into place.
-
XOR on a budget
Chapter 8 solved a four-dot XOR with a small hidden layer. This XOR has 200 dots. Starting from one hidden neuron and adding one at a time, what is the smallest hidden layer that gets the test score to 95%?
What we found
Three. One neuron scored 67% on the test. Two drew a diagonal stripe and stopped at about 86%: over ten different random starts, two neurons never got past 91%. Three reached 96% by epoch 300, and got to 95% or better in 9 starts out of 10. Five did it every time. A stripe was enough for four dots sitting on the corners. It is not enough when the dots fill the whole corner regions, because the far ends of each region stick out of the stripe.
-
Three sides
Fence in the orange circle with exactly three hidden neurons, then take one away and train again. Why is three the smallest number that works? The line-drawing rule from chapter 6 is all you need.
What we found
With three neurons the network scored 100% on its training dots by epoch 68, and the orange region is a triangle with rounded corners. With two it stalled at 86% on training and 82% on the test. Each hidden neuron draws one straight line, and you cannot close a fence with fewer than three straight sides. Two lines leave a gap, however you turn them.
-
Cram, then cure
This is chapter 16 with every control unlocked: the biggest network, only 60 training dots, and 9 of them the wrong color (ringed in black). On fast, run it to epoch 1,500 and watch the two lines on the chart, then cure it at least three different ways.
What we found
The test score climbed to 98% by epoch 92 and never went higher. Then the network began bending its boundary around the ringed dots. By epoch 1,500 its training score was 97% and its test score was down to 85%. Four cures, each tried alone from the same setup and scored on the test at epoch 1,500: more data (400 training dots: 94%), a smaller network (one hidden layer of 8: 93%), weight decay, which chapter 16 named but did not build (set "discourage big weights" to 0.01: 97%, and you can watch the lines in the diagram stay thin), and stopping early (pause at the peak instead of running on: 98%). Push the weight decay slider to 0.03 and you learn something else: the score drops to 50%, because every weight is being squashed toward zero faster than training can grow it.
-
Too hot, too cold
Chapter 13 warned that a learning rate can be too big or too small. On the spirals, find one that makes training fall apart, one that makes it crawl, and the fastest one that still works. Switch the chart to loss to see the damage.
What we found
With this setup, 0.1 reached 95% on the training dots at epoch 712, and 0.3 got there sooner, at epoch 417. At 1 the network touched 95% at epoch 328 and then lost it again, swinging between 49% and 96%. At 3 and at 10 it never learned: at 10 the loss, which starts near 0.7, leapt as high as 36. Going the other way, 0.01 had reached only 58% after 2,000 epochs, and at 0.001 the loss crept from 0.671 to 0.669 in 2,000 epochs. That is still learning, just far too slowly to be any use.
For a real explosion, press "the exploding one" and then play: ReLU, four layers of 16, a batch size of 1 and a learning rate of 10. In ten tries out of ten the weights shot past a billion inside the first epoch, and the lab stops with a warning. Tanh and sigmoid never did this in any run we made. Their outputs cannot go beyond 1, and that keeps every nudge small. A ReLU output has no ceiling, so big weights make big outputs, big outputs make bigger nudges, and bigger nudges make bigger weights.
-
Plain against Adam
Chapter 13 gave gradient descent a memory and called the result Adam, and chapter 32 reaches for it by default. Solve the spirals with the plain rule, then switch the learning rule to Adam and solve them again: how many epochs does each one need?
What we found
From these setups the plain rule reached 95% on the training dots at epoch 712 and Adam at epoch 501. Over 12 random starts both solved the spirals every time, and Adam was first every time: about 400 epochs (the middle of its 12 results) against about 900. Look at the learning rate, though. The lab moves it to 0.01 when you pick Adam, because the two rules measure their steps differently. A plain step is the learning rate times the slope. An Adam step is about the size of the learning rate itself, however gentle the slope is. Give Adam the plain rule's 0.1 and it thrashes: in six tries of 4,000 epochs it never reached 95%.
-
Squisher showdown
Same spirals, same two layers of 8 neurons, three activations from chapter 11. Which gets the training score to 95% first: tanh, ReLU or sigmoid? Look closely at the edges of the shapes each one draws.
What we found
ReLU got there at epoch 461 and tanh at epoch 712. Sigmoid was still at 60% after 2,000 epochs. Over eight random starts of 1,500 epochs each, tanh and ReLU got there all 8 times and sigmoid never did. ReLU's shapes have straight edges and sharp corners. A ReLU neuron's map is perfectly flat on one side of a straight fold and a steady slope on the other, so everything built from such maps is made of flat pieces. Tanh draws smooth curves. Sigmoid is not broken, only slow: its curve is flatter, so each nudge is smaller. With the learning rate turned up to 1 and 3,000 epochs to work with, it solved the spirals in 4 starts out of 6.
-
Wide against deep
One hidden layer of 16 neurons has 65 stored numbers, and so do two hidden layers of 8 and 4. Chapter 17 found that neither shape simply wins, so run your own race on the spirals, from several random starts and with each learning rule. Which reaches 95% on the training dots first, and how often does each get there at all?
What we found
From these two buttons, which use the plain rule, the two-layer network reached 95% at epoch 1,061 and the wide one at epoch 2,772. Then we ran 12 random starts of 4,000 epochs under each learning rule. With the plain rule both networks solved the spirals all 12 times, and the two-layer one was first every time: about 1,300 epochs (the middle of its 12 results) against about 2,650. With Adam the two-layer network was first in 10 starts and the wide one in 2. Adam's runs were less dependable as well: inside 4,000 epochs the wide network fell short twice and the two-layer one once, though all three got there when we let them run on to 8,000.
So in this lab, at this size, two narrow layers usually learn the spirals in fewer epochs, and one wide layer gets there too. That is one small experiment on one set of dots. Chapter 17 ran a bigger one, with 60 starts, and found the wide network the more dependable of the two.
-
One at a time, or all at once
The mini-batches of chapter 13 are the batch size slider, and this setup nudges the weights after every single dot. Count the epochs until the ring closes, then press reset, slide batch size to "all" and count again.
What we found
With a batch size of 1 the training score passed 95% at epoch 4. With 16 it took 47 epochs, and with all 200 dots in one batch it took 615. An epoch is one pass through the dots either way, but with a batch of 1 that pass contains 200 nudges, and with one big batch it contains exactly one. The price of small batches shows on the chart: each nudge is based on very few dots, so the line wobbles more.
-
The unlearnable
Draw a set of dots that even the biggest network (four layers of 16) cannot get above 60% on the test. What is your drawing missing that the spirals have? Think of the memorizing in chapter 16.
What we found
Confetti works: orange and blue dots scattered with no pattern at all. With 480 dots the network scored between 70% and 81% on the dots it trained on and between 49% and 53% on the held-back ones, which is a coin flip. With only 80 dots it memorized 90% or more of its training dots and still did no better than a coin flip on the test. The spirals are hard, but they follow a rule, so knowing some dots tells you about the others. Confetti has no rule. There is nothing to learn, only answers to memorize. A checkerboard is the in-between case: with 4 squares along each side the test score ended at about 90%, and with 8 it ended between 58% and 73%. (To keep this fair we did not draw by hand: a short program scattered our dots and colored them.)
How to experiment like a scientist
A lab like this makes it easy to click around until something works. That is fun, and it teaches you very little, because afterwards you cannot say why it worked. Four habits turn clicking into finding out.
- Change one thing at a time. If you add a layer and raise the learning rate together and the score improves, you do not know which change did it. Changing one thing and holding everything else still is called a controlled experiment. The reset button exists for this: it replays the same starting weights, so the one thing you changed is the only difference.
- Guess before you press play. Say it out loud or write it down: "two neurons will get about 90%". A guess that turns out wrong is the most useful thing that can happen to you in a lab, because it shows you exactly which idea in your head needs fixing.
- Write down what happened. The settings, the epoch, both scores. Memory rounds everything toward what you expected. The copy link button stores the whole setup in a web address, which makes it a good thing to paste into your notes.
- Repeat with a different random start before you believe it. A network begins from random weights, and a computer makes its "random" numbers from a starting number called a random seed. A different seed means a different start, and sometimes a different ending.
This book learned the fourth habit the hard way. An early draft of chapter 17 said that a network with one hidden layer stays stuck near a coin flip on the spirals, and that making the layer wider barely helps. That came from a single run, from a single random start. Run again from 60 different starts, with Adam as the learning rule, one hidden layer of 16 solved the spirals 60 times out of 60, and the chapter had to be rewritten.
Mission 8 has a smaller example. With Adam, the wide network won 2 of our 12 races and the two-layer one won 10. Watch only one of those two races and you would walk away believing the opposite of what the other ten say. One run is a story. Twelve runs are evidence.
A network's behavior comes from a handful of choices: the data, the shape, the activation, the learning rule and its rate. You understand a choice once you have changed it, guessed what would happen, and watched.
Build the smallest network you can (fewest stored numbers) that gets the test score on the two spirals to 95%. We managed it with 30 stored numbers, though it took Adam, nearly 4,000 epochs and several random starts. When you beat that, press "copy link to this setup" and send the link to a friend with a dare attached. The link rebuilds your exact dots, network and starting weights on their screen.
A network with no hidden layers trains on the two moons. Where does its test score end up?
One neuron draws one straight line. A straight line can cut most of one moon away from the other, but the tips of the moons hook around each other, and no straight line gets those. More epochs do not help, because the limit is the shape, not the training. You can check this with mission 1.
The biggest network is training on 60 noisy dots. Its training score is still climbing, but its test score peaked a while ago and is now falling. Which of these will not help?
A falling test score while the training score climbs means the network has moved on from learning the pattern to memorizing the wrong-colored dots. More epochs give it more time to memorize. More data, a smaller network, weight decay and stopping early all push the other way. Mission 4 lets you try each one.
You race a wide network against a deep one on the spirals, once each, and the wide one wins. What should you do before telling anyone that wide beats deep?
Each run begins from different random weights, and a single run can go either way. In our 12 races with Adam the two-layer network won 10 and the wide one won 2, so one race could easily have shown you either result. Changing the learning rate until you get the answer you wanted is not an experiment at all.
A lab like this was not our idea. It was made famous in 2016 by TensorFlow Playground, built by Daniel Smilkov and Shan Carter. Ours is written from scratch for this book, with the book's own colors, missions and held-back test dots, but if you have enjoyed this one, go and play with the original too.
When you have had enough of dots, chapter 34 steps back to show where everything you have just been playing with came from.