Chapter 16: Learning vs. memorizing
Watch a network ace the homework and fail the test.
There are two ways to get ready for a math test. The slow way: work through the practice sheet until you understand why each answer is what it is. The fast way, popular at 11 p.m. the night before: memorize the answer key. Question 1 is 42, question 2 is 7, question 3 is 19. Either way you score 100% on the practice sheet. The difference only shows when the real test asks different questions.
A neural network faces the same choice, and nothing in training pushes it toward the slow way. Training asks for one thing only: get the training examples right. Understand the rule or memorize the answers, the loss goes down either way. Chapter 15 gave the memorizing version a name, overfitting, and a way to catch it: hold back a test set that the network never trains on. In this chapter you'll watch overfitting happen, learn to spot it on a chart, and try three cures.
A secret rule and a sloppy answer key
Here's the experiment. Take a square of graph paper and draw a gently wavy line across the middle. That line is the secret rule: a dot above it is orange, a dot below it is blue. Scatter some dots on the paper, color each one by the rule, and hand them to a network as its training examples. A dot's position (two numbers, x and y) is the input, and its color is the right answer. This is the dots-on-graph-paper picture from chapter 6, except the boundary is now a curve, so the network gets hidden layers to bend with.
Now the twist that makes it realistic: 15 out of every 100 training dots have the wrong color. Real data is like this. Someone tired clicks the wrong button. A sensor glitches. A photo of a muffin gets filed under "chihuahua". Wrong answers hiding in the training data are called label noise, and nobody tells the network which labels are the wrong ones. To the network, a mislabeled dot looks exactly as trustworthy as any other. It's a practice sheet whose answer key has typos in it.
That gives us a number worth remembering: 85%. A network that learned the wavy line perfectly would disagree with every mislabeled dot, so it would score 85% on its training dots, not 100%. The only way to score much higher is to learn the typos too.
For the test we use 300 fresh dots that the network never trains on, colored by the true rule with no mistakes. So the test score answers one question: how well does this network know the real rule? A network that does well on examples it has never seen is said to generalize, and that is the whole point of training one.
Honesty note: in a real project the test set comes out of the same messy pile as the training set, so it has wrong labels too, and you can't clean them out because you don't know which ones they are. We gave ours a clean test so that its score means exactly "how much of the true rule does the network know". The shape of the chart you're about to see is the same either way.
The big network below trains for 1,000 epochs, long after it has found the rough shape of the rule. What will its score on the test dots do over that time?
Press train and watch the teal line. It peaks early. After that, more training makes the network worse at the real rule, even while its score on the training dots keeps going up.
Ace the homework, fail the test
Press train. The map shows the training dots, with the network's current opinion painted behind them: orange where it would answer orange, blue where it would answer blue, stronger color where it is more sure. The chart shows its two scores as training goes on. Watch what grows around the ringed dots, and what the test line does while that is happening.
The network is never shown the rings or the dashed line.
score on the 300 unseen test dots (the exam)
Test dots are drawn as diamonds . A big diamond with a black outline is one the network currently gets wrong.
What just happened
For the first few hundred epochs the network does exactly what you'd hope. It paints orange on top and blue underneath, with a boundary that settles close to the dashed line. At epoch 305 its test score peaks at 96%. Its score on the training dots at that moment is 83%, just under the 85% that a perfect learner would get. It is "wrong" about the ringed dots, and it is right to be.
But training doesn't know that. All the loss can see is 12 dots the network keeps getting wrong, and gradient descent keeps pushing to fix them. The big network has 1,185 stored numbers and only 80 dots to satisfy, so it has plenty of room to spare. It uses that room. It bends the boundary out of shape, pushing long fingers of orange deep into the blue region to reach the ringed dots down there, and blowing a bubble of blue around a ringed dot stranded up in the orange. Each one it captures pushes the training score up a notch. By epoch 900 it scores a perfect 100% on its training dots.
And the test? Down to 80%. At its best moment the network got 11 of the 300 test dots wrong. By the end it gets 59 wrong.
Those fingers and islands are what memorizing looks like. An island of blue in the middle of the orange isn't a rule. It's a sticky note that says "the answer near this exact spot is blue", written because of a single dot, and that dot was a typo. Switch on show the 300 test dots and look at the black-outlined diamonds: they sit inside the fingers and islands. Each one is an innocent dot that happened to land next to a typo.
This is the 11 p.m. student. A big memory, a short practice sheet and plenty of time: memorizing every answer, typos included, is the easiest road to a perfect homework score. A big network with only a few examples is in exactly that position.
Is this a fluke of one lucky set of dots? Press new dots and train again. We ran this setting on 30 different sets of dots. The big network's test score fell from its peak every single time: by 14 points on average, never by fewer than 8. It finished with between 93% and 100% on its training dots and between 74% and 87% on the test.
How to read the two lines
The widget's chart, with one line for the training score and one for the test score, is one of the most useful pictures in all of machine learning. Here is the run you just watched, with the two stages marked.
- Both lines climbing together: learning. Whatever the network is picking up also works on dots it has never seen, so it must be something about the rule.
- Training line climbing, test line falling: memorizing. The network is getting better at its homework in ways that don't carry over to anything else.
- The peak of the test line: the best moment. That's the best network this run ever produced. Everything after it made things worse.
The space between the two lines at the end, 100% against 80%, is the bill for memorizing. People call it the generalization gap.
One odd thing about our chart: early on, the test line sits above the training line. That's the label noise. On this homework 85% (the dotted line) is the best an honest learner can do, while the clean test goes all the way up to 100%. On real data the test has mistakes in it too, so the training line is normally on top the whole way, and what tells you memorizing has begun is the gap getting wider. You'll see exactly that in chapter 24, where you train a model on 10,000 real news headlines.
Three cures
Go back to the widget and try each of these. Changing a setting gives you a fresh, untrained network.
Cure 1: more data. Slide training dots up to 400 and train the same big network. Now there are 60 mislabeled dots, but they are scattered among 340 correct ones. The dots are packed closer together now, so an island around a typo would swallow its correctly colored neighbors too, and they outvote it. The training score stops at 85%, right where an honest learner should be, and the test score finishes at 96%. Try the stops in between and you can watch the gap close: with 20 dots the big network finishes at 67% on the test, with 80 dots at 80%, with 200 dots at 86%, with 400 dots at 96%. More data is the most reliable cure there is, which is one reason AI companies go to so much trouble to collect it.
Honesty note: more data makes memorizing much harder, not impossible. Look closely at the 400-dot map and you can find a faint dent or two where the network has started to try. When we let that same run carry on to 1,500 epochs, it had crept up to 86% on the training dots and slipped to 94% on the test.
Cure 2: a simpler model. Back to 80 dots, and pick the small network: 17 stored numbers instead of 1,185. That is enough to draw a smooth, gently bending boundary, and not enough to chase a dozen separate typos. It finishes with 83% on the training dots and 95% on the test. It never came close to acing the homework, and it beat the big network on the exam by about 15 points.
Cure 3: stop early. Big network, 80 dots, train to the end, then press rewind to the best moment. The fingers and islands vanish. The good network was there all along, at epoch 305, and all you had to do was stop. This is called early stopping, and it is exactly as simple as it sounds: keep a copy of the network from the moment the score on unseen data was at its best.
| What we trained | Training dots | Test dots |
|---|---|---|
| Big network, 80 dots, all 1,000 epochs | 100% | 80% |
| Cure 1: big network, 400 dots | 85% | 96% |
| Cure 2: small network, 80 dots | 83% | 95% |
| Cure 3: big network, 80 dots, stopped at epoch 305 | 83% | 96% |
Read that table twice. The row with the best homework score has the worst exam score. A high training score was never the goal.
How dependable are the cures? Across the same 30 sets of dots, 400 dots beat 80 dots on the test all 30 times (96% on average against 81%). The small network beat the big one 28 times out of 30, but it is the shakier cure: its test score ranged from 74% to 96%, because a network that small sometimes can't quite follow the wave either. A model can be too simple as well as too big.
The golden rule: never peek at the test
There's a catch hiding in cure 3. How did the rewind button know that epoch 305 was the best moment? It looked at the test scores.
That sounds harmless. We never trained on the test dots. But think about what happened: we checked the network at 200 different moments during training and kept the version that did best on those particular 300 dots. Some of that was skill and some was luck, a version that just happened to suit those dots. So the 96% we reported is a little too rosy. Give that network 300 brand-new dots and it would probably score slightly lower.
Now imagine doing a lot of choosing that way. Try five network sizes and keep the one with the best test score. Try ten learning rates. Try different stopping points. Each choice bends the model a little more toward one particular set of test dots. No weights ever touched them, but you did. The memorizing just moved from the network into the person training it.
So the golden rule of machine learning is: the test set stays unseen until the very end. If the score is supposed to tell you how the model will do on data nobody has seen, then nobody can have used that data for anything. Not the network, and not you.
Which leaves a puzzle. You still need some unseen data to decide when to stop and which size to use. The answer is a third pile:
| Pile | What it's for | At school |
|---|---|---|
| Training set | The network learns from it | The homework |
| Validation set | You make choices with it: when to stop, how big a network, which learning rate | The practice exam |
| Test set | Used once, after every choice is made, to get an honest score | The real exam, in a sealed envelope |
A validation set is a slice of your data held back from training, just like the test set, but one you're allowed to look at as often as you like. A common split is 80% training, 10% validation, 10% test. In real work the rewind button would watch the validation score, and the test set would stay in its envelope until the last day.
Other ways to discourage memorizing
Two more tools, named here so you recognize them later. Weight decay shrinks every weight a tiny bit on every step, so the network can only keep the large weights that sharp little islands need if they really earn their keep. Dropout switches off a random handful of neurons on each training step, so the network can't count on any single neuron to remember a single example. We won't build either one in this chapter, though the lab in chapter 33 has a weight-decay slider you can try. You'll find both in the training recipe of a great many real models.
So why do giant networks work at all?
If 1,185 stored numbers are enough to memorize 80 dots, a chatbot with hundreds of billions of weights ought to be the worst crammer in history. Why isn't it?
Mostly because of cure 1. Those models are trained on enormous amounts of data: hundreds of billions of words and more. It is the 400-dot end of the slider, pushed about as far as anyone can afford to push it. With that much to get right, learning the patterns is a cheaper way to lower the loss than memorizing each sentence.
Honesty note: that isn't the whole story. Big models do memorize some of what they read, and a chatbot can sometimes recite a famous passage word for word. And researchers have found that very large networks overfit less than the simple picture in this chapter predicts, even when they have room to memorize everything. Why that happens is an open research question. What nobody disputes is the golden rule: you only find out what a model has learned by testing it on data it has never seen.
A network's score on its training data tells you how well it remembers. Only its score on data it has never seen tells you what it learned. When the first keeps climbing while the second falls, it has stopped learning and started memorizing.
The same experiment in numpy
This is overfit.py. It makes its own dots (not the same ones as the widget, so the numbers differ a little) and trains three networks: big with 80 dots, big with 400 dots, small with 80 dots. Read the three tables as they scroll past, then compare the summary at the bottom with the table above.
When we ran it, the big network with 80 dots finished at 100% on its training dots and 79% on the test, after peaking at 95% back at epoch 110. With 400 dots it finished at 85% and 97%. The small network finished at 85% and 94%.
Two things under the hood, for the curious. The hidden neurons in this chapter squish with tanh, a cousin of sigmoid that outputs −1 to +1 instead of 0 to 1. And the gradient descent uses momentum: each step keeps 90% of the step before it, like a ball that keeps rolling, which gets through the slow stretches of training faster. Momentum is one of the "smarter step" rules that chapter 13 mentioned. It is not what causes the memorizing. We switched it off and ran the widget's default setting again: the same network on the same dots still peaked and then memorized, only far more slowly. Its training score passed 97% at epoch 11,730 instead of epoch 775.
Find the breaking point. With the big network, slide the dots up 20 at a time. How many does it take before the network can no longer reach 100% on its training dots in 1,000 epochs? Then try the reverse: give the small network only 20 dots. Can it memorize now? What does that tell you about what "big" and "small" really mean?
Two networks are trained on the same data. Network A: its training accuracy climbs to 91% and its test accuracy climbs to 89%, and then both level off. Network B: its training accuracy climbs to 99.8%, while its test accuracy reaches 88% early on and then slides down to 74%. Which one is overfitting?
B is getting better at its homework and worse at the exam at the same time, which is the signature of memorizing. A's two lines move together and stay close. A small steady gap like that is normal.
You know that about 1 in 10 labels in your training data is wrong. Your network scores 100% on that training data. What does that tell you?
The real rule disagrees with every wrong label, so a network that learned only the rule would top out around 90%. Scoring 100% means it reproduces the mistakes as well, and the only way to do that is to memorize them.
You train 20 networks of different sizes, pick the one with the highest test score, and report that score as "how good my model is". What went wrong?
Picking the winner by test score is peeking. Part of why that network won is luck with those particular test examples. The choice should have been made on a validation set, with the test set saved for one final, honest measurement.