Chapter 34: The story so far
Eighty-three years on one line, and you have built a small version of almost all of it.
You have now built a small version of nearly everything in modern AI: a neuron, a network that learns, a language model, attention, a digit reader, a picture maker, a game player, and the engine that trains them all. This chapter puts the whole story on one line, so you can see where each thing you built sits, and how long the world waited for it.
Chatbots arrived in most people's lives in 2022, which makes neural networks feel like a new invention. They aren't.
How old is the idea of an artificial neuron: a little unit that adds up its inputs and fires if the total is big enough?
The first one was described in 1943. Step through the line below and watch how long the idea had to wait.
Fifteen moments, 1943 to 2026
The line runs from the first neuron on paper to the models of 2026. Press later to step through it, or tap anywhere on the line to jump. Each card says where in this book you met the small version. The dots are colored by the part of the book that covers them, and they sit at their true distance apart, so watch how the spacing changes.
-
1943
A neuron made of math
Warren McCulloch and Walter Pitts publish the first mathematical model of a neuron: a unit that adds up its inputs and fires if the total reaches a threshold. Nothing in it learns. A person has to choose the wiring.
Your small version: chapter 3, where you set the weights by hand too.
-
1958
The perceptron: a neuron that learns
Frank Rosenblatt unveils a machine that adjusts its own weights from examples. A newspaper reports that the U.S. Navy expects it to walk, talk and see one day.
Your small version: chapter 4, where a neuron found its own weights.
-
1969
The book of limits
Marvin Minsky and Seymour Papert's book Perceptrons proves that a single layer of adjustable weights can never learn a puzzle like XOR. Money and interest were already moving elsewhere, and neural networks go quiet for about fifteen years.
Your small version: chapter 7, where you watched one neuron fail at exactly that puzzle.
-
1980
The neocognitron
Kunihiko Fukushima builds a network of small pattern detectors, reused across the whole picture and stacked in layers, inspired by the seeing part of a cat's brain. It has no good way to learn its filters yet.
Your small version: the sliding filters and pooling of chapter 28.
-
1986
Backpropagation teaches the hidden layer
David Rumelhart, Geoffrey Hinton and Ronald Williams show that passing the blame backwards lets hidden neurons find useful jobs for themselves. The method had been worked out before, more than once, and mostly ignored.
Your small version: the hidden neurons of chapter 8, and the blame you passed back by hand in chapter 14.
-
1989–1998
Networks that read handwriting
Yann LeCun and his colleagues train a convolutional network with backpropagation to read ZIP codes on real mail. By 1998 a system built around a network of that kind is reading several million bank checks a day, and the same group has put together the MNIST digits.
Your small version: the digit reader you trained and drew for in chapter 27.
-
2012
AlexNet
Two graduate students, Alex Krizhevsky and Ilya Sutskever, and their professor Geoffrey Hinton train a convolutional network with about 60 million weights on two gaming graphics cards. In a contest to label photos it gets 15.3% wrong. The best other entry gets 26.2% wrong. Many people date the current boom in AI to that result.
Your small version: the convolutional network of chapter 28. Chapter 18 explains the graphics cards.
-
2013–2016
Learning games from the score alone
DeepMind's deep Q-network learns Atari games from nothing but the screen and the score. In March 2016 the company's Go program, AlphaGo, beats Lee Sedol, one of the best players in the world, by four games to one.
Your small version: the creature that learned a maze from rewards in chapter 30.
-
2017
The transformer
Eight researchers, most of them at Google, publish "Attention Is All You Need". It describes a network built almost entirely out of attention. Every big chatbot today is a descendant.
Your small version: the locker room that learned where to look in chapter 22.
-
2020
Pictures from noise
Jonathan Ho, Ajay Jain and Pieter Abbeel show that a network trained to guess the noise in a ruined picture can make new pictures that rival the best of any method at the time. The core idea dates from 2015.
Your small version: the dots that un-ruined themselves into a picture in chapter 29.
-
2020–2021
The shapes of proteins
Proteins are the tiny machines inside every living cell, and a protein's shape decides what it does. In November 2020 DeepMind's AlphaFold 2 predicts those shapes in a blind test far more accurately than anything before it. By mid-2022 its makers have published predicted shapes for around 200 million proteins, free for any scientist to use.
No small version in this book. Its main ingredient is one you know, though: the attention of chapter 22.
-
2022
Out of the lab
Image generators arrive within months of each other: DALL·E 2 in April, Stable Diffusion in August. On November 30, ChatGPT opens to the public. Two months later an estimated 100 million people are using it.
Your small version: the name inventor of chapter 21, which runs the same loop that chapter 23 follows through a chatbot.
-
2024
Models that think before answering
In September 2024 OpenAI releases o1-preview, a model trained to write out a long chain of working before it gives an answer. Other labs follow within months.
No small version yet. Chapter 35 explains it, and it is made from the loop in chapter 23.
-
2024
Two Nobel prizes
In October the physics prize goes to John Hopfield and Geoffrey Hinton "for foundational discoveries and inventions that enable machine learning with artificial neural networks". Half of the chemistry prize goes to Demis Hassabis and John Jumper for AlphaFold 2. The other half goes to David Baker for designing new proteins by computer.
Nothing to build here. Look back along the line, though: it is the same Hinton as in 1986 and 2012.
-
2026
Models that decide instead of write
The first System One models appear: Jev in September and Clef in October. You hand one some text and a typed question. It answers in a single pass, with a probability attached, and writes nothing.
Your small version: the headline sorter you trained in chapter 24.
What the line shows
The idea is old. The neuron you played with in chapter 3 is recognizably the one from 1943, and the training loop from chapter 4 would not have surprised Rosenblatt.
The spacing is lopsided. The first 69 years of the line, from 1943 to 2012, hold seven of the fifteen moments. The 14 years since then hold the other eight. What changed around 2012 was mostly not the ideas. The neocognitron was 32 years old, and the famous backpropagation paper was 26. What arrived was enough data and enough computing power to make the old ideas work, which is the story chapter 18 told.
You have been here before. For 12 of the 15 moments, this book had you build, train or take apart a small version. The three you have not built are the three that are hardest to shrink: a protein-shape predictor, a prize, and a reasoning model. Chapter 35 takes on the last of those.
Honesty note: a timeline is somebody's choice of what to include, and this one leaves out far more than it keeps. Most of these steps were also the work of many more people than the two or three named on each card.
Almost every idea in modern AI is older than it looks. What arrived late was the data and the computing power to make the old ideas work, and you have now built a small version of nearly all of them.
Pick the moment on the line that surprised you most and find out one thing about it that is not on the card: who the people were, what machine they used, or what happened next. Then write a card for 2030. Put today's date on it and keep it somewhere you will find it again.
Convolutional networks were being trained to read handwriting in 1989. Why did it take until 2012 for one to win a big contest on real photos?
Backpropagation and stacked layers were both in use by 1989. What AlexNet had that the earlier networks lacked was more than a million labelled photos and graphics cards fast enough to train on them.
The 1969 book showed that a single layer of adjustable weights can never learn XOR. What did it take to get past that limit?
More training can't fix a shape that doesn't fit the problem: that was chapter 7. A hidden layer can, and backpropagation is the method that made hidden layers learn.
A friend says, "AI was invented a few years ago, when chatbots came out." What is the most accurate reply?
The neuron is from 1943, backpropagation became famous in 1986, and the transformer inside every big chatbot is from 2017. The chatbots of 2022 were those ideas, made enormous and opened to the public.
The newest dots on the line are still being drawn. Chapter 35 goes there next: models that think before they answer, models that use tools, what researchers find when they look inside, and the questions nobody has answered yet.