Neuro 13
0 / 37 chapters finished

Chapter 18: Scaling this up

What changes between 9 weights and 9 billion.

The networks you have trained so far hold somewhere between 3 and a few hundred numbers. The ones in the news hold billions. Weights and biases together are called a model's parameters, and the parameter count is the usual way to say how big a model is:

NetworkYearParameters
Screen-time neuron (chapter 4)3
XOR network (chapter 8)17
Meal guesser (chapter 12)192
Digit reader (chapter 27)about 51,000
LeNet-5, an early digit reader used on bank checks1998about 60,000
ResNet-50, an image classifier2015about 25.6 million
GPT-3, the language model that led to ChatGPT2020175 billion
Llama 3.1 405B, a language model anyone can download2024405 billion
The newest chatbotsnowusually not published

From the meal guesser's 192 parameters to GPT-3's 175 billion is a jump of nearly a billion times. What do you think had to change inside to make that work?

The same math. That is the most surprising fact in this chapter, so the first section is about it.

What stays the same

The math. The forward pass of a 175-billion-parameter model is still: multiply inputs by weights, add a bias, bend with an activation, pass it on. The loss is still cross-entropy. Backpropagation still finds the gradient, and gradient descent still steps against it. The loop inside neuro4.py is, in outline, the loop that trains a chatbot.

Honesty note. The wiring of a large language model is more elaborate than layers stacked in a row. The most important addition is called attention, and it gets its own chapter (chapter 22). But it is built from the same parts and trained in exactly the same way.

What changes

More layers. Your networks have one or two hidden layers. ResNet-50 has 50 layers, GPT-3 has 96, and Llama 3.1 405B has 126. You saw why depth helps in chapter 17: each layer reuses what the layers below it built. In an image model the early layers respond to edges, the middle ones to textures and parts, and the late ones to whole objects. That stacking of simple pieces into complicated ones is the "deep" in deep learning.

More data. Iris has 150 rows. MNIST has 60,000 training images. ImageNet, the dataset ResNet-50 was trained on, has about 1.28 million labelled training images. GPT-3 read 300 billion tokens during training (a token is a word or a piece of one), which is very roughly a couple of million books' worth. Llama 3.1 was trained on more than 15 trillion. Bigger models need more data to be worth their size, and researchers have measured how the two should grow together. Those measurements are called scaling laws (Kaplan and others, 2020; Hoffmann and others, 2022).

More computing. Every one of those parameters gets used on every token and updated at every training step. That takes special hardware.

Why graphics chips?

Think about what one layer has to do. Every neuron multiplies each of its inputs by a weight and adds the results up. A layer with 1,000 neurons and 1,000 inputs is a million multiplications. The weights of a layer form a grid of numbers called a matrix, and running the layer is one giant matrix multiplication.

Here is the useful part: none of those sums depends on any of the others. Neuron 1's total can be worked out without waiting for neuron 2's. The job splits into thousands of small, separate sums that could all be done at the same moment, if only you had thousands of little calculators.

That is exactly what a GPU (graphics processing unit) is. It was designed for video games, where the color of every pixel on the screen has to be worked out many times a second, and each pixel is a small sum that does not depend on its neighbors. A normal processor has a handful of powerful cores that work through tasks one after another. A GPU has thousands of simple ones working side by side. In 2012 a network called AlexNet, trained on two gaming graphics cards, won the main ImageNet competition by a wide margin, and the field has run on GPUs ever since.

What does this cost? Most companies keep it private, but Meta published figures for Llama 3.1 405B. It was trained on up to 16,000 of Nvidia's H100 chips, each drawing about 700 watts, roughly a microwave oven running flat out. The total came to 30.84 million GPU-hours. A single chip would have needed about 3,500 years. Going by those figures, the chips alone used about 21.6 million kilowatt-hours of electricity, which is what 2,000 average American homes use in a year. That is one training run of one model, and the chips cost far more than the electricity: each is reported to sell for tens of thousands of dollars.

What gets harder

Keeping the gradient alive. Remember the signal that shrank to a trillionth in chapter 11? A hundred layers of anything will misbehave unless you are careful. Modern networks use ReLU-family activations, carefully chosen starting weights, and a trick called residual connections: shortcuts that let the signal skip past a layer, so it always has a clear path backward.

Overfitting. A model with billions of parameters has plenty of room to memorize. The cures are the ones from chapter 16, applied on a giant scale, starting with the simplest: much more data.

Engineering. A model that does not fit on one chip has to be split across thousands, and with that many machines something breaks every day. Saving progress, restarting, and watching loss curves for trouble is a large part of the real work.

Why bigger keeps working

Here is the fact at the center of modern AI: so far, when people have made these models bigger and trained them on more data, they have kept getting better, and fairly predictably. Many researchers expected clever hand-built designs to win instead. Rich Sutton's short essay "The Bitter Lesson" is about why they keep losing to simple methods given more computing power. Nobody knows how long the trend will last. It has not stopped yet.

Next: feeding a network words

Everything in this part of the book has fed networks tidy numbers: yes/no switches, measurements in centimeters. The giant models in the table above were trained on text. But a network can only multiply and add, and you cannot multiply a weight by a word. Turning text into numbers is where the next part begins, in chapter 20, right after a checkpoint to make sure this part has stuck.

A giant model is not a different kind of machine. It is the same multiply, add, bend and nudge, with more layers, billions of parameters, oceans of data, and thousands of chips doing the sums side by side.

Challenge

Count the parameters in a tiny network with 3 inputs, 5 hidden neurons, and 2 outputs. Count the weights first, then remember that every hidden and output neuron also has a bias. Then work out how long it would take to type out all 175 billion of GPT-3's parameters at one per second, without sleeping. (You should get 32, and about 5,500 years.)

What is different about the basic math inside a model with billions of parameters, compared with the meal guesser?

The parts and the training rule are the same. What changes is how many there are, how they are wired, and how much data and computing they need.

Why are graphics chips so good at running neural networks?

A GPU's cores are individually simple. Its strength is having thousands of them, which suits a job made of thousands of separate multiply-and-add sums. That is true whether the network handles pictures, text, or sound.

A team makes its model ten times bigger but keeps training it on the same small dataset. Which problem are they most likely to run into?

More parameters means more room to memorize. That is why bigger models are trained on more data: the two have to grow together.