Chapter 32: What the professionals use
Build your own mini-PyTorch in 60 lines, then read the real thing.
Count the backward passes you have met so far. d_o and d_h for the secret door in chapter 8. A different pair for the snack picker in chapter 10. Nine separate blames, worked out one multiplication at a time, in chapter 14. Another set for the headline model in chapter 24, with a special line just for its embedding table. Every time the network changed shape, somebody had to work out the blame formulas again by hand. One slip, and you get a network that runs without complaint and quietly learns nothing.
People who build networks for a living don't do this. They use a framework: a toolkit whose main trick is working out the blame automatically, for any network you can write down. The one most AI researchers use is called PyTorch.
A framework can look like magic. It isn't, and the best way to be sure is to build the heart of one yourself. That takes 60 lines of Python. Then you'll read real PyTorch and find you can recognize every line.
What if every number kept notes?
Think back to how blame moved in chapter 14. At every step the recipe was the same: take the blame that arrived, multiply it by a slope, pass it on. The recipe never changed. Only the shape of the network decided which numbers got multiplied by which.
So here is the idea. While the computer does the forward pass, make every number jot down two things: which numbers it was made from, and how sensitive it is to each of them. With those notes, one small function can walk any calculation backwards and hand out the blame. Nobody has to work out a formula ever again.
The official name is automatic differentiation ("differentiation" is the math word for finding slopes). The web of notes is called a computation graph.
Build it, piece by piece
The whole engine is one Python class called Value. You can run it further down. First, here it is in five pieces.
class Value:
def __init__(self, data):
self.data = data
self.grad = 0.0
self.parents = ()
self.slopes = ()
self.op = ""
1. A number with a memory. A Value wraps one number, data. The other four lines are its notebook. parents will hold the Values it was calculated from, slopes one slope for each parent, and op the name of the calculation.
grad is where the blame goes. PyTorch calls blame "grad", short for gradient, so we use its spelling. It starts at 0 because nobody has been blamed yet.
def __add__(self, other):
if not isinstance(other, Value):
other = Value(other)
out = Value(self.data + other.data)
out.parents = (self, other)
out.slopes = (1.0, 1.0)
out.op = "+"
return out
2. Adding. Python runs __add__ whenever it sees a + b and a is a Value. It makes a new Value holding the sum, then writes the notes: my parents are a and b.
Both slopes are 1. Push a up by 0.001 and the sum goes up by exactly 0.001. The first two lines let you add a plain number, as in a + 3, by wrapping the 3 first.
def __mul__(self, other):
if not isinstance(other, Value):
other = Value(other)
out = Value(self.data * other.data)
out.parents = (self, other)
out.slopes = (other.data, self.data)
out.op = "*"
return out
__radd__ = __add__
__rmul__ = __mul__
3. Multiplying. The same again for a * b, with one twist in the slopes. Say a is 2 and b is −3. Push a up by 0.001 and the product moves by −3 × 0.001. So the slope for a is b's value, and the slope for b is a's value. Each one's slope is the other one, just like the wires in chapter 14.
The last two lines make 3 * a work as well as a * 3.
def sigmoid(self):
s = 1 / (1 + math.exp(-self.data))
out = Value(s)
out.parents = (self,)
out.slopes = (s * (1 - s),)
out.op = "sigmoid"
return out
4. Squishing. Sigmoid, the squish from chapter 3, has one parent. Its slope is the one you dragged along the curve in chapter 11: output × (1 − output), written here as s * (1 - s). That is 0.25 in the middle of the curve and nearly 0 out at the flat ends.
def history(self):
"""Me and my ancestors, oldest first."""
order, seen = [], set()
def visit(value):
if value not in seen:
seen.add(value)
for parent in value.parents:
visit(parent)
order.append(value)
visit(self)
return order
def backward(self):
self.grad = 1.0
for item in reversed(self.history()):
notes = zip(item.parents, item.slopes)
for parent, slope in notes:
parent.grad += slope * item.grad
5. Reading the notes backwards. history() follows the parent notes all the way back and lists every Value that went into this one. A Value only joins the list after all of its parents have, so parents always come first.
backward() is the whole trick. The final answer gets blame 1: move the answer by a tiny bit and the answer moves by exactly that bit. Then it goes through the history backwards, and every Value pays each parent that parent's slope times its own blame.
Multiplying slopes along a chain of notes is the chain rule from chapter 14. And look at the +=. It is there for a reason you are about to see.
That is all of it, apart from two lines at the end that let print() show a Value neatly. There is no formula in there for any particular network. There are three tiny slope rules, for adding, multiplying and squishing, and one loop that reads notes.
In f = (a * b + c) * a, the Value a is used twice. When the blame comes back, what should a end up with?
Added. Pushing a changes f along two wires at once, and the two effects pile up. That is what the += in backward() is for. Watch it happen on the last step of the warm-up below.
Watch the blame flow through the notes
Every box is a Value: its number in blue, its blame in orange. Every wire is a parent note. Press step. The forward pass fills in the numbers from left to right and tags each wire with its slope. Then the backward pass sends the blame from right to left. Read the line under the buttons each time: it shows that step's arithmetic. Start with the warm-up, then switch to the neuron.
The small tag on a wire is its note: the slope. A circled × or + is the calculation that made the box, and the little S-curve means squish. The sliders work at any time, including after the backward pass has finished.
Check a blame by wiggling
Use the blame: one training step
What just happened
The forward pass did two jobs at once. It worked out the numbers, and it left a note on every wire. The backward pass never looked at the formula (a × b + c) × a. It only read notes.
On the last step of the warm-up, a was paid twice. It got 4 along the wire into f, then −6 along the wire into d, and 4 + −6 = −2. That −2 is a real prediction about the world: push a up a little and f goes down about twice as far. The wiggle button checks it. Pushing a from 2 to 2.001 moves f by −0.002003, which is −2.003 for every 1 of push.
Why −2.003 and not −2 exactly? Because 0.001 is small, not endlessly small. A wiggle is an estimate. The backward pass gives the exact slope.
Now the neuron. That graph is the single neuron from chapter 4, with its loss attached. When the backward pass finishes, w and b each hold a blame, and a blame says which way to nudge. Every press of nudge w and b is one complete round of training: forward, backward, nudge. With the starting numbers, one press takes the loss from 0.25 to 0.0496, and twelve presses take it to 0.0047.
Honesty note: chapter 14 used half of the miss squared as its loss, which keeps arithmetic done by hand tidy. An engine doesn't mind untidy, so here the loss is just the miss times itself. Every blame comes out twice as big, and every nudge still points the same way.
Run your engine
Here is the real thing: the 60-line engine, then two tests of it.
Part 1 builds the same two graphs as the widget, prints the notes the engine kept, and checks every blame against a wiggle. This wiggle is a millionth, much smaller than the widget's.
Part 2 is the payoff. It builds the secret-door network from chapter 8 out of Values: 2 inputs, 4 hidden neurons, 1 output, 17 numbers to learn. Then it trains it.
Before you run it, scroll to the training loop at the bottom of the script. How many lines of blame formulas, like d_h = ..., does a 17-number network need now?
None. The loop says loss.backward() and that is the entire backward pass. You could change the network to 2-8-8-1 and that line would not change.
Run the engine
This is autograd.py. In Part 1, compare the two columns. In Part 2, watch the loss fall and read the four answers at the end. It uses nothing but plain Python: no numpy this time.
In Part 1 the two columns agree to five decimal places: backward() says −2.000000 for a, and the wiggle says −2.000003. A smaller wiggle gives a closer estimate. Change WIGGLE to 0.001, the size chapter 14 used, and the same line reads −2.003000.
In Part 2 the loss starts at 1.0414 and ends at 0.0029 after 500 steps. The four answers come out as 0.025, 0.968, 0.979 and 0.028: the secret door, solved. On each step the engine made 182 Values, wrote a note on every one, and read them all back.
The script always starts from the same random weights, so you will see those same numbers. Change random.seed(13) to another number for a different start. We tried 100 different seeds. All 100 solved the puzzle, with a final loss between 0.0022 and 0.0077.
Remember why chapter 14 gave up on wiggling as a way to train: it needs a whole forward pass for every single number. The backward pass gets every blame in one sweep. With 17 numbers that is a nice saving. With billions, it is the only thing that makes training possible.
Honesty note: this engine is a teaching toy. It keeps a separate note for every single number, in plain Python, which is far too slow for a network of any real size. And the design is not our invention. The math is called reverse-mode automatic differentiation, and the Finnish mathematician Seppo Linnainmaa worked it out in 1970. Tiny teaching engines like this one were made popular by Andrej Karpathy's micrograd. Ours was written from scratch for this book and keeps its notes in a different way, but the idea is the same.
A framework is not magic. Its main trick is the one you just built: keep notes during the forward pass, then read them backwards to get the blame for every number, in any network you can write down.
Now read the real thing
PyTorch is that engine, grown up. Three things are different.
- It works on whole grids of numbers at once. A PyTorch tensor is a grid of numbers: a list, a table, or a stack of tables. A tensor keeps notes the way a Value does, but one note covers the whole grid. "Multiply this grid by that one" is a single note, where our engine would write thousands.
- It is fast. The grid math is done by carefully tuned code, and on a graphics chip if you have one.
- It knows hundreds of calculations, each with its slope rule built in. Ours knows three.
Here is the neuron from the widget again, written in real PyTorch:
import torch x = torch.tensor(2.0, requires_grad=True) w = torch.tensor(0.5, requires_grad=True) b = torch.tensor(-1.0, requires_grad=True) prediction = torch.sigmoid(w * x + b) miss = prediction - 1.0 loss = miss * miss loss.backward() print(x.grad, w.grad, b.grad)
On our laptop that prints tensor(-0.1250) tensor(-0.5000) tensor(-0.2500). Those are the same three blames your engine found. requires_grad=True means "keep notes for this one". You can even peek at a note: print(loss.grad_fn) shows <MulBackward0 ...>, which is PyTorch's way of saying "this was made by a multiply".
A PyTorch training loop has the same moves as yours, with new names. Here they are side by side:
| The move | In your engine | In PyTorch |
|---|---|---|
| A number that keeps notes | Value(2.0) | torch.tensor(2.0, requires_grad=True) |
| Forward pass | predict(left, right) | model(x) |
| Score the mistake | miss * miss, added up | loss_fn(scores, labels) |
| Clear the old blame | p.grad = 0.0 | optimizer.zero_grad() |
| Pass the blame back | loss.backward() | loss.backward() |
| Nudge every number | p.data -= LEARNING_RATE * p.grad | optimizer.step() |
The optimizer is PyTorch's name for the part that does the nudging. The numbers it nudges, every weight and every bias, are called parameters. That is why the list in autograd.py has that name.
Why clear the old blame?
Because of the +=. New blame lands on top of whatever was already there, and PyTorch works the same way. Left alone, the blame from every earlier step would still be sitting in grad when the next step arrives.
Go back to autograd.py. Delete the two lines for p in parameters: and p.grad = 0.0 just above loss.backward(), and change STEPS = 500 to STEPS = 1000. What will the run do?
Try it. At step 100 the loss already reads 0.0004, far ahead of the normal run's 0.1054. But every step is now shoving the weights with the pile of all the blame so far, and the pile never empties. Long after the answers are right, the weights keep growing. In a normal run the biggest weight ends up near 10. Here one of them passes 500, and a little after step 650 sigmoid has to work out a number too big for Python. The script stops with OverflowError: math range error. Leaving out optimizer.zero_grad() is a classic PyTorch bug for the same reason. Press "reset code" to get the lines back.
Your iris network, in PyTorch
In chapter 15 you trained on 150 real flowers with every gradient written out by hand. That was neuro7.py. Here is the same network on the same flowers with the same train and test split, in PyTorch.
PyTorch cannot run inside this page. It is a large program built to talk directly to your computer's processor and graphics chip. On our laptop the installed folder is 339 MB. So the two scripts in the rest of this chapter are for reading here and running on your own computer. You can get every script in the book in one zip file.
Iris in PyTorch
This is neuro8.py. Read it top to bottom and tick off the moves from the table: the forward pass, the loss, then zero_grad, backward, step. To run it, unzip the book's scripts, open a terminal in the neuro13 folder that comes out, and type these two lines. On Windows, type python in place of python3.
pip install torch scikit-learnpython3 neuro8.pyWhat it printed on our laptop. This is a recording of a real run, not a live one:
131 numbers to learn epoch 0: train accuracy 0.825 test accuracy 0.700 epoch 20: train accuracy 0.967 test accuracy 0.967 epoch 40: train accuracy 0.975 test accuracy 0.967 epoch 60: train accuracy 0.983 test accuracy 1.000 epoch 80: train accuracy 0.983 test accuracy 0.967 epoch 100: train accuracy 0.983 test accuracy 0.967 epoch 120: train accuracy 0.983 test accuracy 0.967 epoch 140: train accuracy 0.992 test accuracy 0.967 epoch 160: train accuracy 0.983 test accuracy 0.967 epoch 180: train accuracy 0.983 test accuracy 0.967 final train accuracy: 0.983 final test accuracy: 0.967
A test accuracy of 0.967 is 29 of the 30 test flowers. Your hand-written neuro7.py scores 29 of 30 as well. Same data, same shape of network, same result, and not one gradient formula in the file.
We ran it with 20 different random starts. Eighteen runs got 29 of 30, and two got all 30. With only 30 test flowers, one flower is worth more than three points, so don't read much into the difference.
Reading a real script, chunk by chunk
The second script, neuro9.py, reads handwritten digits like the network your browser trained in chapter 27. This time it uses all of MNIST: 60,000 pictures to learn from and 10,000 held back for the test. Here it is in six chunks. Each one is something you have already done by hand.
device = (
"cuda" if torch.cuda.is_available()
else "mps" if torch.backends.mps.is_available()
else "cpu"
)
Where to do the math. cuda is a graphics chip made by Nvidia, mps is the graphics chip in an Apple computer, and cpu is the ordinary processor every computer has. Chapter 18 explained why graphics chips suit this work. Later, .to(device) moves the model and each batch of pictures onto whichever one was found.
prepare = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,)),
])
train_set = datasets.MNIST(root=DATA, train=True, download=True, transform=prepare)
test_set = datasets.MNIST(root=DATA, train=False, download=True, transform=prepare)
train_batches = DataLoader(train_set, batch_size=BATCH, shuffle=True)
test_batches = DataLoader(test_set, batch_size=512)
The data. Three habits from chapter 15, one line each. Normalize is feature scaling: subtract the average pixel, 0.1307, and divide by the spread, 0.3081. train=True and train=False are the train and test split, which MNIST comes with ready-made. DataLoader shuffles the pictures and deals them out in mini-batches of 128.
model = nn.Sequential(
nn.Flatten(),
nn.Linear(784, 128),
nn.ReLU(),
nn.Linear(128, 64),
nn.ReLU(),
nn.Linear(64, 10),
).to(device)
The network. nn.Flatten unrolls each 28 × 28 picture into a list of 784 numbers, as in chapter 27. nn.Linear(784, 128) is a whole layer of 128 neurons: it is x @ w + b, and it makes its own weights and biases and gives them random starting values. nn.ReLU() is np.maximum(0, x) from chapter 11. Two hidden layers make it a deep network, as in chapter 17, and the ten outputs are one score for each digit, as in chapter 10. In all there are 109,386 numbers to learn.
loss_fn = nn.CrossEntropyLoss() optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
The loss and the nudger. CrossEntropyLoss is the softmax and the "how surprised was the model?" score from chapter 10, done in one go. That is why the network above has no softmax layer of its own. model.parameters() hands the optimizer every number it is allowed to change. Adam is a smarter nudger than plain gradient descent. There is more about it below.
for pictures, labels in train_batches:
pictures, labels = pictures.to(device), labels.to(device)
scores = model(pictures)
loss = loss_fn(scores, labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()
The loop. The last five lines now read like a sentence. Run the batch forward. Score the mistake. Clear the old blame. Pass the new blame back through the notes. Nudge every number. Put it next to the loop at the bottom of autograd.py: it is the same loop.
def accuracy(batches):
right = 0
with torch.no_grad():
for pictures, labels in batches:
pictures, labels = pictures.to(device), labels.to(device)
right += (model(pictures).argmax(dim=1) == labels).sum().item()
return right / len(batches.dataset)
The honest test. torch.no_grad() means "don't keep notes". Testing only needs the forward pass, and notes cost time and memory. argmax picks the digit with the highest score, and the function counts how often that pick was right.
Handwritten digits in PyTorch
This is neuro9.py, whole. The first run downloads MNIST, about 12 MB, into a folder called data next to the folder the script is in.
pip install torch torchvisionpython3 neuro9.pyWhat it printed on our laptop. This is a recording of a real run, not a live one:
doing the math on: mps 60000 training pictures, 10000 test pictures 109,386 numbers to learn epoch 1: loss 0.3212 train accuracy 0.9542 test accuracy 0.9502 epoch 2: loss 0.1291 train accuracy 0.9730 test accuracy 0.9655 epoch 3: loss 0.0898 train accuracy 0.9790 test accuracy 0.9707 epoch 4: loss 0.0704 train accuracy 0.9844 test accuracy 0.9719 epoch 5: loss 0.0551 train accuracy 0.9871 test accuracy 0.9763 the first ten test pictures: really a 7, network says 7 ok really a 2, network says 2 ok really a 1, network says 1 ok really a 0, network says 0 ok really a 4, network says 4 ok really a 1, network says 1 ok really a 4, network says 4 ok really a 9, network says 9 ok really a 5, network says 5 ok really a 9, network says 9 ok
After five trips through the 60,000 training pictures, the network read 9,763 of the 10,000 test pictures correctly. Your number will be close to ours but probably not identical: four runs from four different random starts scored between 0.9731 and 0.9764.
Compare the two accuracy columns. After the first epoch they were 0.0040 apart. After the fifth, training accuracy is 0.9871 and test accuracy is 0.9763, which is 0.0108 apart. That widening gap is the memorizing from chapter 16.
Now think about what loss.backward() did in there. This network has 109,386 numbers. Finding their blame by wiggling would take 109,386 extra forward passes for every single nudge. The backward pass found all of it in one sweep, 469 times per epoch.
What else a framework gives you
Automatic blame is the main trick. A framework also hands you a lot of other things you would otherwise build yourself.
- Graphics chips. One line,
.to(device), moves the math onto a GPU, where thousands of small sums happen at the same time. You never write a line of code for the chip itself. It pays off for big networks. For the small one inneuro9.py, the graphics chip in our laptop was no faster than its ordinary processor, and a run took a few minutes either way. - Ready-made layers.
nn.Linearis a layer of ordinary neurons.nn.Conv2dis the sliding filters from chapter 28.nn.Embeddingis the lookup table from chapter 20. Each one knows its own shapes and its own slope rules. - Optimizers. Plain gradient descent nudges every number by learning rate × blame. Adam, the usual choice, keeps a running average of each number's recent blame and steps along that, which smooths out the zigzags. It also takes bigger steps for numbers whose blame has been small and smaller steps for numbers whose blame has been large. In PyTorch that is one line.
- Data loading. Shuffling, cutting into mini-batches and preparing each picture is one
DataLoader.
PyTorch is not the only framework. The other two you are most likely to hear about are JAX and TensorFlow, both from Google. The spelling differs. The main trick is the same one.
Teach your engine something new. It cannot subtract yet: prediction - 1 is an error, which is why the script has to write prediction + (-TARGET). Add a __sub__ method to Value, modeled on __add__. The only real decision is the two slopes: if b goes up a little, which way does a - b move? Then prove it works. Change warm_up in Part 1 to use a subtraction and check that the two columns still agree.
Harder: add __pow__ so that miss ** 2 works. It has one parent, and the slope of "a squared" is 2 × a. Then rewrite the loss in total_loss as (predict(left, right) - target) ** 2 and run Part 2 again. The loss should fall exactly as it did before.
And one to break: set LEARNING_RATE to 0.1, then to 30.0. One crawls and one slams into a wall. Read the four answers at the end of each run to see which is which.
During the forward pass, what does the engine write down for each new Value?
The number plus its notes: parents and slopes. The blame cannot be written yet, because it depends on everything that happens later in the calculation. That is why there has to be a backward pass at all.
In a PyTorch training loop, what does loss.backward() do?
It only works out the blame. Nothing changes until optimizer.step() does the nudging. They are two separate moves, just as loss.backward() and p.data -= ... are two separate lines in your engine.
You invent a brand-new shape of network that nobody has built before. What do you have to work out by hand before a framework can train it?
Nothing extra. Each small calculation already carries its own slope rule, and the backward pass only reads notes. That is the reason frameworks exist, and it is why people can try a new idea for a network in an afternoon.