Chapter 28: How networks see
Slide a tiny filter over a picture and watch edges light up.
At the end of chapter 27 you found your digit reader's blind spot. Slide a digit three pixels sideways and it could no longer read most of them. You can see at a glance that nothing important changed. Why can't the network?
This chapter builds the network that chapter asked for: one that knows pixels have neighbors, and that can learn what the end of a stroke looks like once and then spot it anywhere. It is one of the most successful ideas in all of machine learning, and it is how computers learned to see. The trick is to stop treating a picture as a long list of unrelated numbers and to look at it the way you do: one small patch at a time, hunting for the same shapes wherever they turn up.
Why a bag of pixels fails
A quick reminder of what went wrong. The fully connected network from chapter 27 flattens the 28 × 28 picture into one long row of 784 numbers, and every hidden neuron has its own separate weight for each of them. Nothing tells the network that two pixels sat side by side. The pixels might as well have been poured out of a bag, which is why it read scrambled pictures just as well as normal ones.
That explains the sliding digit too. When a 7 moves three pixels to the right, its ink lands on different pixels, and those pixels have different weights. Whatever the network learned about a stroke at pixel 300 is no help with the same stroke at pixel 303. It would have to learn every digit again at every position, and nobody showed it digits at every position.
A picture is not a bag. Which pixels are neighbors is the whole point: a line is a run of inky neighbors, and a corner is two lines meeting. We want a network with that fact built in.
A filter is a tiny pattern detector
Forget the 784 weights per neuron. Take a tiny grid of weights instead: 3 × 3, just nine numbers. That little grid is called a filter. (You will also see it called a kernel, which means the same thing.)
Lay the filter on the top-left corner of the picture so that it covers nine pixels. Multiply each pixel by the weight sitting on top of it, and add up the nine products. That is a weighted sum, the same move the neuron made in chapter 3, only with nine inputs. Write the answer down.
Now slide the filter one pixel to the right and do it again. And again. At the end of the row, drop down one pixel and start the next row. Keep going until the filter has visited every spot it can. The answers, laid out in the same arrangement as the spots they came from, make a new grid of numbers called a feature map. The whole slide-multiply-add routine is called a convolution.
What do the answers mean? Here is a filter with −1s down its left side, 0s in the middle and +1s down the right, sitting on two different patches of a picture. As in chapter 27, each pixel says how much ink is there, already divided by 255: 0 is blank paper and 1 is solid ink.
On the first patch the inky pixels line up under the +1s, the blank pixels under the −1s add nothing, and the sum is big. On the second patch every pixel is inky, so the −1s exactly cancel the +1s. This filter does not care how much ink there is. It responds to one thing only: paper on the left, ink on the right. It is a detector for one kind of edge.
That is the whole trick. A filter gives a big number wherever the patch underneath looks like the filter's own pattern, and a small or negative one where it doesn't. Slide it across a picture and the feature map lights up exactly where that pattern is.
The widget below has a picture called "block": a solid square of ink on blank paper. You slide that same paper-to-ink filter over it. Where will the output map be anything other than zero?
Only the left and right sides. Inside the square everything is equally inky, so the filter cancels to 0. Along the top and bottom the ink changes as you go down, not across, and this filter only compares left with right. Pick "block" and "vertical edges" below and check. Then switch to "horizontal edges" and watch the other two sides appear.
Slide the filter
On the left is a picture: 28 × 28 numbers, from 0 (blank paper) to 1 (solid ink). The nine boxes in the middle are the filter. The map on the right is what comes out. Point at any cell of the output map: the orange frame shows the nine pixels that made it, and the panel underneath shows the arithmetic. Press slide the filter to watch it travel. Then try the other pictures and filters, or type your own numbers into the nine boxes.
Output colors: positive zero negative, stretched so the biggest number in the map gets the strongest color.
The framed output number (): pixel × weight, nine times
What just happened
- Vertical edges. On the 4, the map is blue wherever paper turns to ink as you go right (the left side of each stroke) and red where ink turns back to paper (the right side). Same edge, opposite direction, opposite sign. Everything flat, blank or inky, comes out as exactly 0.
- Horizontal edges. The same filter turned on its side. Now the tops and bottoms of strokes light up and the upright strokes nearly vanish.
- Blur. All nine weights are one ninth (the boxes show 0.11), so every output is the average of nine neighbors.
- Sharpen and outline. A big positive weight in the middle with negative weights around it compares each pixel with its neighbors. In "outline" the nine weights add up to 0, so anything flat gives 0 and only the places where something changes survive.
- Do nothing. A single 1 in the middle copies the picture. A filter is only arithmetic. The interesting part is which nine numbers you put in it.
Filters like these are older than neural networks. The blur and sharpen buttons in a photo editor do exactly this arithmetic, with numbers a person chose.
Did you notice the output map is 26 × 26 and not 28 × 28? The 3 × 3 filter is not allowed to hang off the edge of the picture, so it only fits in 26 places across and 26 down: 676 stops in all. Many networks first add a border of zeros around the picture, called padding, so that the map comes out the same size as the picture. The network later in this chapter does that.
Honesty note: a mathematician would call what we just did cross-correlation. A true convolution flips the filter upside down and back to front before sliding it. In a network the filter's numbers are learned, so flipped or not makes no difference, and everyone in machine learning says "convolution" anyway.
The same weights, everywhere
Here is something you did not do in that widget: set different weights for different parts of the picture. The same nine numbers were used at all 676 stops. That is called weight sharing, and it does two things at once.
Position stops mattering. An edge detector that works in the top-left corner works in the bottom-right corner too, because it is literally the same detector. If the digit slides sideways, the lit-up spots in the feature map slide along with it. Nothing has to be learned twice.
The number of weights collapses. A filter has the same number of weights however big the picture is. A plain layer does not:
| a plain layer of 64 neurons | one 5 × 5 filter | |
|---|---|---|
| a 28 × 28 gray digit | 50,176 weights | 25 weights |
| a 12-megapixel color photo | 2,304,000,000 weights | 75 weights |
The big number is not a typo. A 12-megapixel phone photo is 4,000 × 3,000 pixels, and each pixel has a red, a green and a blue number: 36,000,000 inputs. Give each of just 64 plain neurons a weight for every one of them and you need 2,304,000,000 weights in a single layer. A 5 × 5 filter on the same photo needs 75: 25 for each of the three colors. (Each neuron and each filter also has one bias, which we left out of the table.)
Sharing weights is a bet, though. It bets that neighboring pixels belong together, and that a pattern means the same thing wherever it shows up.
Remember the scramble experiment from chapter 27? Shuffle the pixels of every picture the same way and the plain network reads them as well as ever. We ran it again with a network built from filters, the one you are about to meet. On normal digits it beats the plain network. What happens when it trains on scrambled ones?
It gets worse. On normal test digits it scored 98.5%. Trained and tested on scrambled ones, 93.6%. The plain network, trained the same way on the same digits, scored 97.5% and 97.4%: it did not notice. Shuffle a picture and neighbors are not neighbors any more, so there is nothing left for a small filter to find. The bet only pays when neighbors matter. In real pictures they do.
From one filter to a layer
One filter finds one pattern. A convolutional layer uses several filters at once, and each one produces its own feature map. A layer with 8 filters turns one picture into a stack of 8 maps: one showing where the upright edges are, one for some other pattern, and so on.
Two small extras, both old friends. Each filter has a bias that is added to every sum. And every number then goes through ReLU from chapter 11, which turns negatives into 0. So in a feature map, 0 means "my pattern is not here" and a big number means "it is right here".
Pooling: shrink the map, keep the news
A feature map records where a pattern was found, exact to the pixel. That is more detail than is good for us: a 7 is still a 7 if its corner sits one pixel higher. Pooling throws some of the "where" away on purpose. Cut the map into 2 × 2 blocks and keep only the biggest number in each block. This kind is called max pooling.
Two things come of it. The map shrinks to half the width and half the height, so the next layer has a quarter as many numbers to deal with, and each of its filters reaches across twice as much of the original picture. And the map becomes a little tolerant: if a pattern moves by one pixel, the biggest number in its block is often still the same number, so the pooled map often does not change at all.
Pooling has no weights. There is nothing in it to learn. It is a fixed rule.
Stack them up
Now do it all again. A second convolutional layer slides its filters over the pooled maps instead of over the picture. If the first layer made 8 maps, each second-layer filter is 5 × 5 and 8 deep: it reads the same patch of all eight maps at once, so it can respond to combinations, like "an upright edge here, with a flat top just above it".
It also sees more. In the network below, a first-layer filter looks at a 5 × 5 patch of the picture. Because of the pooling in between, a second-layer filter is fed by a patch 14 pixels wide: half the width of the whole picture.
A network built this way, from layers of filters with pooling in between, is called a convolutional network. The standard story of what its layers do goes like this. Early layers find edges. The next find corners and curves made of edges. Later ones find parts made of those, like an eye or a wheel, and the last ones find whole things. Small and simple at the bottom, big and abstract at the top.
Honesty note: that tidy ladder is a cartoon. The edge detectors in the first layer are real, and they turn up in almost every network trained on pictures. Higher up it gets messier. People who look inside big image networks find units that respond to textures such as fur or brick more than to shapes, units that respond to several unrelated things, and plenty that nobody can put a name to. The direction of the story is right. The neatness is not.
Nobody designs the filters
Here is the part that matters most. In the first widget you picked the nine numbers, or used a preset that a person worked out long ago. In a convolutional network nobody picks. Every filter starts out as random numbers. Training nudges them with backpropagation from chapter 14, exactly like any other weight. (The one twist: a filter is used at every position, so its nudge is the sum of the nudges from all of them.) The network has to discover which patterns are worth detecting, with nothing to go on but "get more digits right".
The network in the next widget is built like this:
| step | what comes out | weights |
|---|---|---|
| 8 filters, each 5 × 5, then ReLU | 8 feature maps, 28 × 28 | 208 |
| max pooling, 2 × 2 | 8 maps, 14 × 14 | 0 |
| 32 filters, each 5 × 5 and 8 deep, then ReLU | 32 feature maps, 14 × 14 | 6,432 |
| keep the strongest number in each map | 32 numbers | 0 |
| one plain layer, then softmax | 10 probabilities, one per digit | 330 |
| total | 6,970 |
The fourth step is pooling taken to the limit: instead of the biggest number in each 2 × 2 block, keep the single biggest number in each whole map. It turns every map into one answer: "how strongly did my pattern show up, anywhere at all?" The plain network of chapter 27, 784 → 64 → 10, has 50,890 weights. This one has about a seventh as many.
The 8 first-layer filters began as random numbers and were trained only to get digits right. Nobody mentioned edges. What do you expect them to look like now?
Look at the top row of the widget below. Most of the filters are blue on one side and red on the other: edge detectors, tilted at different angles, like rougher versions of the presets you tried. A couple are mostly blue and simply fire on ink. Nobody put those patterns there. Spotting edges turned out to be useful for telling digits apart, so training found them. Train again from different random numbers and you get a different set, in a different order.
Filters nobody designed
Draw a digit in the box with your mouse or finger, or load a test digit the network has never trained on. Your drawing gets the same tidy-up as in chapter 27: the line is redrawn at the thickness of the pens in MNIST, the box around the ink is shrunk to fit 20 × 20, and the digit is slid until its center of mass sits in the middle of a 28 × 28 picture. Then watch each stage of the network respond. Read down a column: a filter, the map it makes from your digit, and the pooled map.
loading the trained network…
what the network gets
positive weight zero negative weight. These never change, whatever you draw.
The small orange frame marks the strongest number in each map. Those 32 numbers, and nothing else, go on to the last layer, which turns them into the ten bars above.
What just happened
- The filters are the learned part. They stay the same whatever you draw, because nothing on this page is training.
- The feature maps are your digit seen through each filter. A filter that is blue on top and red underneath lights up along the bottom edges of your strokes, and a tilted one lights up on slanted strokes. Draw a 1 and then a 0 and watch which maps care. A few maps are faintly blue all over: those filters have a positive bias, so they are slightly "on" even over blank paper.
- The pooled maps are the same pictures at half the width and half the height. Chunkier, and still easy to recognize: not much was lost.
- The second layer looks different. Most of its 32 maps are nearly blank with a few lit-up spots, and which maps light up changes from digit to digit. Each second-layer filter is waiting for one particular combination of first-layer patterns, and most combinations are not in your drawing.
One thing you can't do here is fool it by drawing in a corner, because the tidying step moves your digit back to the middle before the network sees it. To test what started this chapter, the digit has to be moved after tidying.
Back to the blind spot
Time to settle what started this chapter. Here are two networks side by side. One is plain: the same shape as the one you trained in chapter 27, 784 → 64 → 10, but taught with 55,000 digits instead of 8,000, which is why it starts from a higher score than yours did. The other is the convolutional network you just looked inside. Both learned from the same 55,000 centered digits, with the same recipe. Neither was ever shown a digit that had been pushed off-center.
We push every one of the 10,000 test digits 4 pixels to the right and ask both networks to read them. The plain network drops from 97.5% right to 33.2%. What do you expect from the convolutional network, which scores 98.5% on centered digits?
It scores 98.4%. Its filters find the same strokes wherever they are, and its last pooling step only asks how strongly each pattern showed up, not where. Try it yourself below.
The shift test
Draw a digit or load a test digit, and check that both networks read it. Then drag the slider to push the tidied digit across the picture, one pixel at a time. Whatever slides off the edge is gone. Watch the two answers.
loading the trained networks…
Plain network
784 → 64 → 10 · 50,890 weights · a separate weight for every pixel
Convolutional network
filters → pooling → filters → strongest → 10 · 6,970 weights · shared
Both networks were trained ahead of time by the script below. The line under each card is not about your drawing: it is that network's score on all 10,000 test digits pushed the same way.
The measured truth
One drawing proves little, so here is the whole test set: all 10,000 test digits, pushed to the right, read by both networks.
| digits pushed right by | plain network | convolutional network |
|---|---|---|
| 0 pixels | 97.5% | 98.5% |
| 2 pixels | 78.0% | 98.5% |
| 4 pixels | 33.2% | 98.4% |
| 6 pixels | 13.6% | 97.5% |
Same digits to learn from, same training recipe, a seventh of the weights, and a completely different result. At 6 pixels the plain network is close to guessing (one digit in ten would be 10%). The difference is not what the networks were shown. It is how they are wired.
Follow a shifted digit through the convolutional network and you can see why. The filters find the same strokes in their new places, because the same weights run everywhere. The feature maps slide over by the same amount. And the last pooling step keeps only the strongest number in each map, wherever it happens to be. So the 32 numbers that reach the final layer hardly move, and neither does the answer.
The fine print
This result is easy to oversell, so here are three things it does not mean.
Up and down are harder. Switch the direction in the widget. Handwritten digits are taller than they are wide, so they have less empty margin above and below than at the sides. Pushed 4 pixels down, the convolutional network reads 91.9% of the test digits, and at 6 pixels 80.7%. (The plain network: 15.4% and 4.3%.) Two things go wrong when a digit is pressed against the frame. Past a certain point its ink slides off the picture, and no wiring can read ink that is not there. And even before that, the edge of the frame gets in the way: filters can only be placed inside it, so the detectors that would sit just beyond the end of a stroke, looking back at it, have nowhere to stand.
Sharing weights is not the whole trick. We trained a third network with the same two layers of filters but a different ending. Instead of keeping only the strongest number in each map, its last layer has a separate weight for every position in every map, which is how LeNet-5 and many other classic convolutional networks finish. On centered digits it beat ours: 99.2%. Pushed 4 pixels right it scored 81.5%, and at 6 pixels 37.3%. That is better than the plain network and far worse than ours. Shared filters make the feature maps slide along with the digit. Whether the network then stops caring where things are is decided by its last steps. Networks that end the classic way are usually also shown shifted, stretched and turned copies of their training pictures, a trick called data augmentation.
Sliding is the only thing it gets for free. Turn a digit on its side, or make it twice as big, and the same filters no longer match. Tolerance to turning and resizing is not built into the wiring. It has to be learned from examples.
Where this came from
The idea of stacking shared detectors with pooling in between goes back to 1980, when Kunihiko Fukushima in Japan built a network he called the neocognitron. He was inspired by what biologists had found in the seeing part of a cat's brain: cells that respond to an edge at one particular angle in one small patch of what the eye sees. What his network lacked was a good way to learn its filters.
In 1989 Yann LeCun and his colleagues at Bell Labs supplied it. They trained a convolutional network with backpropagation to read handwritten ZIP codes from real United States mail. By 1998 the design had grown into LeNet-5, with about 60,000 weights, and a system built around a network of that kind was reading several million bank checks a day. The MNIST digits you have been using were put together by the same group to test networks like these.
Then, for more than a decade, most people working on computer vision preferred other methods. Convolutional networks could read small gray digits, but real photos seemed out of reach. What changed was the amount of data and computing power. In 2012 a network called AlexNet, built by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, entered ImageNet, a yearly contest to sort photos into 1,000 categories, with 1.2 million photos to learn from. AlexNet had about 60 million weights and trained for five to six days on two graphics cards made for video games. A guess counted as right if the true label was among a network's top five. The best of the other entries got 26.2% of the test photos wrong. AlexNet got 15.3% wrong. Within a couple of years nearly every entry was a convolutional network, and many people date the current boom in AI to that result.
Where convolutional networks are used
Anywhere a computer has to make sense of something laid out on a grid, where neighbors matter:
- a phone camera finding faces to focus on, and a photo app that lets you search your pictures for "dog";
- reading: addresses on envelopes, license plates, the text in a photo of a sign;
- medicine: flagging possible signs of disease in X-rays, scans and photos of skin or eyes, for a doctor to check;
- the cameras in cars that pick out lanes, other cars and people;
- sound: turn a recording into a picture of pitch against time, then slide filters over that;
- board games: AlphaGo, the program that beat a world champion at Go in 2016, read the board with convolutional layers.
They are no longer the only way to see. Many of today's image models instead cut the picture into small square patches, treat each patch like a token, and run them through a transformer using the attention from chapter 22. Convolutional networks are still everywhere, especially where a model has to be small and fast.
A filter is a tiny pattern detector with a handful of weights, and the same filter is reused at every position of the picture. Stack layers of filters with pooling in between and a network can find small patterns, then patterns made of patterns, wherever they are. Nobody designs the filters: training finds them.
The script that trained both networks
This is convnet.py, the script that produced the weights your browser has been running, and every measurement in this chapter. It uses PyTorch (you will meet it properly in chapter 32), which is too big to load inside this page, so read it here and run it on your own computer. Things to look for: the two class blocks that describe the networks, the shifted function, and the fact that train never calls it.
pip install torch numpypython3 convnet.pyTwo to try. With the first widget: invent a filter for slanted lines. It should light up strongly along the "slash" picture and give 0 along the straight sides of the "block" (its corners will still show). Hint: make the three weights in every row add up to 0, and the three in every column too. Why does that silence straight edges? With Python: teach the plain network about shifting the brute-force way. In convnet.py, add import random at the top, and inside train replace net(x[batch]) with net(shifted(x[batch], random.randint(-4, 4), random.randint(-4, 4))). That is data augmentation. Run it and read the plain column of the shift test: what did it gain, what did it give up on centered digits, and what happens at 6 pixels, further than anything it was shown?
A filter produces a big positive number at one spot in a picture. What does that tell you?
A filter measures how well a patch matches its pattern. Ink alone does not do it: the edge filter gave exactly 0 on a patch that was solid ink.
One 5 × 5 filter is used on a 28 × 28 gray picture, and then on a gray picture 1,000 pixels wide and 1,000 tall. How many weights does it need?
The filter makes far more stops on the big picture, but it uses the same 25 weights at every stop. That is weight sharing, and it is why these networks can handle large pictures at all.
Our convolutional network still read digits pushed 4 pixels sideways, though it never trained on a shifted digit. Why?
Shared filters make the feature maps slide along with the digit, and keeping only the strongest number in each map throws the position away. The 2 × 2 pooling helps a little, but on its own it only forgives a shift of about a pixel.