Chapter 2: The big picture
What a neural network is, before any detail.
A neural network is a very large pile of tiny math equations. Together they turn an input (like a picture, or a sentence) into an output (like "this is a cat" or the next word in a story).
Each tiny equation has some adjustable numbers called weights. At the start the weights are random and the network is terrible. You show it lots of examples where you already know the right answer. Each time it gets one wrong, you nudge every weight a tiny bit in the direction that would have made it slightly less wrong. (How does it know which direction is "less wrong"? In chapter 4 you'll do one of these nudges yourself, with real numbers, and chapter 13 and chapter 14 explain how the direction is worked out for millions of weights at once.)
Do this millions or billions of times and the weights end up encoding a surprising amount of knowledge about the world.
That's the main idea. The rest of the book shows how far that idea can stretch.
Is it a brain?
The name is borrowed from the brain. Your brain is a web of cells called neurons that pass tiny signals to each other, and each one "fires" when the signals arriving at it add up to enough. The math version you're about to meet was loosely inspired by that. Loosely is the key word.
A real neuron is a living cell, far more complicated than a line of arithmetic. Your brain has about 86 billion of them, each wired to thousands of others, and they talk in brief electrical spikes whose timing matters. The math version just passes one number along. And nobody has found the brain using the nudging method in this book; how brains learn is still an open question. What's inside ChatGPT is not a simulated brain. It's arithmetic that borrowed a name.
Everything becomes numbers
Arithmetic only works on numbers, so before a network can do anything with a picture, a sentence or a sound, somebody has to turn it into numbers. The rest of the book leans on this, so here are the three you'll meet most.
A picture is a grid of numbers. Zoom far enough into any photo and it turns into tiny squares called pixels. Each pixel is stored as a number that says how bright it is.
That smiley is 64 numbers. A color photo keeps three numbers per pixel (how much red, green and blue), so a 12-megapixel phone photo is 36 million numbers.
Text is a list of id numbers. Your computer already stores every letter as a number: c is 99, a is 97 and t is 116, so "cat" is 99, 97, 116. Language models use a different list, where whole chunks of words get one id each. You'll build one of those lists in chapter 20.
Sound is a list of samples. A microphone measures how hard the air is pushing on it, thousands of times a second. Each measurement is one number, called a sample. One second of CD-quality music is 44,100 of them.
So when this book says a network "looks at a photo" or "reads a sentence," it means a list of numbers went in. What comes out is numbers too: 0.93 might mean "93% sure this is a cat."
Where this book is going
- One neuron (you are here). The tiny machine everything else is built from: how it decides, how it learns, and the small puzzle it can't solve until it gets help in chapter 8.
- How networks learn. More answers, more layers and real data, starting at chapter 10. You'll step through the one algorithm that trains every network (chapter 14) and watch a network memorize its homework instead of learning (chapter 16).
- Machines that read and write. Turn text into numbers (chapter 20), train a model that invents names (chapter 21), and see what a chatbot is really doing (chapter 23).
- Seeing, creating, playing. A network that reads your handwriting (chapter 27), one that turns noise into pictures (chapter 29), and one that learns by trial and error (chapter 30).
- Your turn. The tools professionals use (chapter 32), a playground for breaking things (chapter 33), and projects of your own (chapter 36).
Parts 1 to 4 each end with a checkpoint: what to remember, a mixed quiz, and something to build.
If you like watching things move while you read, the 3Blue1Brown neural networks series on YouTube animates many of the ideas in this book. It's a great companion to what you're reading now.
A neural network is a huge pile of simple arithmetic with adjustable numbers called weights. It starts out random and gets good by being nudged, over and over, toward answers we already know.
Before moving on, name three problems that could be turned into "numbers go in, a prediction comes out." Try one from school, one from a game, and one from real life. For each, what would the numbers going in be?
A brand-new network that has not been trained yet is asked "is this a cat?" What does it do?
An untrained network still does its arithmetic and still produces an answer. The answer is just meaningless, because the weights are random numbers. Training is what makes them useful.
During training, the network gets an example wrong. What happens next?
There is no list of saved answers and nobody writes rules. Learning is nothing more than many small nudges to the weights.
Someone says: "The network looked at my photo." What actually went into the network?
A network only does arithmetic, so everything it handles has to be numbers first: pixels for pictures, id numbers for text, samples for sound.