Chapter 13: How learning actually works
Rolling downhill on the loss landscape.
Ever since chapter 4 this book has said that training "nudges the weights in the right direction," and you even did a nudge by hand. This chapter pins down the full rule. It is short, it is the same for a three-weight neuron and a chatbot, and every other chapter in the book leans on it.
The loss landscape
Training needs a score to improve. That score is the loss: one number that says how wrong the network is on its training examples. You have met two kinds. A single yes/no neuron can use squared error, (truth − prediction)², averaged over the examples. A network that picks from a list uses cross-entropy, −log(probability it gave the right answer), from chapter 10. Either way: 0 is perfect, and bigger is worse.
Now imagine every weight in the network is a knob. For each setting of all the knobs, the network has some loss. If you could draw the loss for every possible setting you would get a landscape: hills where the network is badly wrong, valleys where it does well. Training is a search for a low place.
A network with two weights has a landscape you could build out of clay. The largest language models have hundreds of billions of weights, and nobody can picture a landscape with that many directions. Luckily the rule for getting downhill does not care how many directions there are.
The gradient is just the slope
Stand anywhere on a hillside and you can feel which way is downhill and how steep it is. The gradient is that information for the loss landscape: for every weight, it says how much the loss would rise or fall if you turned that one knob up a tiny bit. Working out all those slopes at once is the job of backpropagation, which gets its own chapter next (chapter 14).
Once you have the gradient, the rule for learning is one line:
new_weight = old_weight − (learning_rate × gradient)
The minus sign is what makes it descent: if turning a knob up would raise the loss, turn it down, and the other way around. The learning rate sets how far to turn. Do that for every weight, and you have taken one step of gradient descent.
You have already used this rule. The recipe from chapter 4 was nudge = small step × error × input. The "small step" is the learning rate, and "error × input" is nearly the whole gradient for a single neuron. That chapter's honesty note mentioned one more factor, and now you can see where it comes from: it is the sigmoid's slope from chapter 11. The prediction passes through the sigmoid, so the gradient picks up the sigmoid's slope on the way back. (Squaring the error also adds a factor of 2, which everyone folds into the learning rate.)
In the widget below, the learning rate sets how big each step is. What do you expect to happen if you make it much too big?
It bounces higher and higher. A step that is too long lands further up the far side than where it started. The slope is steeper up there, so the next step is longer still. Find the spot on the slider where this starts.
Roll the ball downhill
The curve is the loss. Left to right is the value of one weight. The red ball is where the network is now, and each step moves it by learning rate × slope. The faint dots show where it has been. Start by pressing "step downhill" a few times and watching the gaps between the dots.
What just happened
The steps got shorter as the ball came down, even though the learning rate never changed. That is because each step is learning rate × slope, and the slope flattens out near the bottom. Gradient descent slows itself down as it arrives.
Now put the ball back and try the slider on the smooth bowl:
- Too small (0.01): after 300 steps the ball still has not reached the bottom. It is learning, just painfully slowly.
- About right (0.15): it settles in 44 steps.
- Big (about 2): every step jumps clean over the bottom to the other side. It zigzags in and still gets there.
- Too big (3 and up): each bounce lands higher than the last, the ball leaves the map, and the loss keeps growing without limit. This is what people mean when they say training "blew up."
The learning rate is a hyperparameter: you choose it, the network does not. The exact numbers above belong to this one bowl. Real networks need their own, usually far smaller, anywhere from 0.1 down to 0.00001. A common trick is a schedule: start with bigger steps to cover ground, then shrink them to settle in.
Getting stuck: local minima
Switch the widget to "bumpy valley" (momentum off, learning rate back near 0.15) and look at the shape before you roll. Where will the ball end up?
In the small dip. At the bottom of the dip the slope is zero, so the step is zero, and plain gradient descent has no way of knowing that something better lies just over the bump.
A dip like that is called a local minimum: lower than everything right next to it, but not the lowest point on the map. For decades people worried that real networks would get trapped in them.
It turns out to matter much less than this drawing suggests. Our ball has one weight, so it can only go left or right. A real network has millions of weights, which means millions of directions to move in, and a spot is only a true trap if every one of them slopes upward at once. That is rare. In practice big networks nearly always find somewhere low enough to work well, even though nobody can prove it is the lowest point. Exactly why this goes so smoothly is still not fully understood.
Gradient descent with a memory: momentum and Adam
Plain gradient descent has no memory. Each step looks only at the slope under its feet. That is why it crawled on gentle slopes and parked in the first dip.
Momentum gives it a memory by making it behave like a heavy ball. The ball keeps a velocity. At each step, most of the old velocity carries over, and the slope adds a new push:
velocity = 0.9 × velocity − (learning_rate × gradient) new_weight = old_weight + velocity
When the slope keeps pointing the same way, the pushes pile up and the ball gathers speed: up to ten times the plain step. When the slope keeps flipping, the pushes cancel out.
Turn on "momentum" in the widget and roll the bumpy valley again. This time the ball carries enough speed to coast through the small dip, and it settles in the deepest valley (88 steps at a learning rate of 0.15). Then try momentum on the smooth bowl. The ball sails past the bottom and sloshes back and forth before it settles: 94 steps, where plain descent took 44. Momentum is a trade, not a free win.
Adam is the method most networks are trained with today. It uses momentum, and adds a second memory: each weight also keeps track of how big its own recent gradients have been, and its step gets divided by that typical size. Weights with huge, jumpy gradients take cautious steps. Weights with tiny gradients take bolder ones. In effect every weight gets its own learning rate, tuned automatically, while you still choose one overall rate. The PyTorch scripts that come with this book use Adam.
Mini-batches: noisy descent
So far a "step" has meant: run every training example through the network, average the gradients, then move. With 60,000 examples that is a great deal of work for a single step.
What everyone does instead is shuffle the examples, take a small handful (say 32), work out the gradient from just those, step, and move on to the next handful. Each handful is a mini-batch, and the method is called stochastic gradient descent, or SGD. ("Stochastic" means "involving chance.")
Each step is now only roughly right, because 32 examples are a small sample of the whole set. Picture the ball wobbling its way downhill instead of rolling smoothly. But you get nearly two thousand wobbly steps for the price of one perfect one, which is a far better deal. There is also evidence that the wobble helps, by shaking the ball out of narrow dips and into wide, flat valleys that hold up better on new data.
That is the whole of gradient descent: find the slope, step against it, repeat. The idea is older than computers. The French mathematician Augustin-Louis Cauchy described it in 1847.
Learning is rolling downhill on the loss. Find the slope for every weight, move each one a small step against its slope, and repeat. The learning rate sets the step size, and getting it wrong in either direction is the most common way for training to fail.
On the bumpy valley with momentum off, raise the learning rate a little at a time, putting the ball back and rolling it each time. Find the smallest learning rate that gets the ball over the small dip without any momentum at all. Why might that be a reason to start training with big steps and shrink them later?
What does the gradient tell you?
The gradient is only local information: the slope right where you are standing. It points downhill from here. It does not know where the bottom is or how far away.
You start training a network and the loss jumps around wildly, then climbs to enormous numbers. What is the most likely cause?
Steps that are too long overshoot, land somewhere steeper, and overshoot by even more. A learning rate that is too small does the opposite: the loss falls, but very slowly.
What does momentum add to plain gradient descent?
Momentum uses exactly the same gradient. It just does not forget the previous steps. Nothing guarantees the lowest point, and on the smooth bowl you saw momentum take longer.