Neuro 13
0 / 37 chapters finished

Chapter 27: Read my handwriting

Train a real digit reader, then draw for it.

Somewhere right now a machine is reading handwriting. One is sorting envelopes by the ZIP code scrawled on the front. Another is reading the amount on a paper check. This was one of the first real jobs neural networks ever held: in the 1990s, networks built by Yann LeCun's team were reading the handwritten numbers on millions of bank checks.

In this chapter you'll build one. You'll train a network on thousands of real handwritten digits, here in your browser, and then you'll write digits for it yourself and watch it read them. After that you'll go looking for its blind spot. It has a big one.

A picture is a grid of numbers

First problem: a neuron eats numbers, and a picture isn't a number. Or is it?

Zoom far enough into any picture on a screen and it turns into little squares. Each square is a pixel, and to a computer a pixel is just a number saying how bright that square is. (A color picture uses three numbers per pixel: how much red, green and blue.) Our pictures are tiny and have no color: 28 pixels wide and 28 tall, with one number per pixel saying how much ink is there. 0 means blank paper. 255 means solid ink. The numbers in between are the gray smudge at the edge of a pen stroke.

Why stop at 255? Each pixel is stored in one byte of memory, and a byte can hold exactly 256 different values: 0 up to 255.

▸ try it

Look inside a picture

This is a real handwritten digit. Move your mouse over it (or tap it). The magnifier on the right shows the actual numbers stored for the pixels under your pointer. Find the middle of a pen stroke, then its fuzzy edge, then blank paper.

the picture: 28 × 28 pixels
the numbers under your pointer
row –, column – ink: – of 255 the network is fed: – input number – of 784

The same picture, unrolled

A network doesn't take a grid. It takes a list. So the 28 rows are laid end to end, top row first: 784 numbers in one long line. The orange mark shows where the pixel under your pointer ends up.

That unrolling has a name: we flatten the picture, which just means laying a grid out as one long list. The network you're about to train has 784 inputs, because 28 × 28 = 784: one input for every pixel. Each pixel's number is divided by 255 on the way in, so every input lands between 0 and 1. (That's the feature scaling from chapter 15.)

Notice what flattening throws away. In the grid, a pixel sits right on top of the pixel below it. In the list those two are 28 places apart, and nothing tells the network they were ever neighbors. Hold that thought. It comes back at the end of the chapter.

10,000 real digits

The digits come from MNIST, the most famous dataset in machine learning: 70,000 handwritten digits, written by American high-school students and Census Bureau workers, and put together by Yann LeCun, Corinna Cortes and Christopher Burges. For nearly thirty years, new ideas for reading images have been tried out on MNIST first.

We'll use 10,000 of them, 1,000 of each digit: 8,000 to train on and 2,000 held back for the honest test from chapter 15.

MNIST is by Yann LeCun, Corinna Cortes and Christopher J.C. Burges, and is shared under the Creative Commons Attribution-ShareAlike 3.0 license. Our sample is in digits.bin.

Train it

You already know every part of this network:

  • 784 inputs, one per pixel.
  • 64 hidden neurons, each using ReLU from chapter 11. Every one of them is wired to all 784 inputs.
  • 10 output neurons, one per digit, with the softmax from chapter 10 turning their ten scores into ten probabilities that add up to 1.
784 inputs one per pixel ⋮ ⋮ 64 hidden neurons (ReLU) 049 ⋮ ⋮ 10 outputs, one per digit

Count the weights. Each hidden neuron has 784 weights and a bias: 64 × 785 = 50,240. Each output neuron has 64 weights and a bias: 10 × 65 = 650. That's 50,890 numbers to learn, and they all start out random.

Training is the loop you've used since chapter 4: show the network a small batch of pictures (32 at a time), use backpropagation from chapter 14 to find out which way to nudge every weight, nudge, repeat. One trip through all 8,000 training digits is one epoch. We'll do 10.

Before any training, the weights are random. Out of 100 test digits, about how many will the untrained network get right?

About 10. There are ten possible answers and the test set has the same number of each digit, so even a network that guesses blindly is right about one time in ten. Remember that number: 10% is the score for knowing nothing at all.

▸ try it

Train a digit reader

Press the button. Your browser downloads the 10,000 digits (about 8 MB) and trains the network right here, starting from random weights. Watch the row of test digits underneath: the network has never trained on those, and its guesses turn from red to green as it learns.

epoch: 0 / 10 train accuracy: ? test accuracy: ? before training: ? weights: 50,890 time: ?

Blue = accuracy on the 8,000 digits it trains on. Orange = accuracy on the 2,000 it has never seen. The chart is zoomed in: its bottom edge is 80%, not zero.

Twenty test digits and what the network says they are

The ones it got wrong

When training finishes, the test digits it misread show up here, with what it said and how sure it was.

What just happened

Before training, the network scored about 10%: a blind guess, as predicted. One epoch later it was reading nearly 90 out of 100 digits it had never seen. By the end it gets 94 or 95 out of 100. Nobody told it what a 3 looks like. It worked that out from 8,000 examples, by nudging 50,890 numbers.

Your exact numbers will differ a little from run to run, because every run starts from different random weights. Press the button again and see.

Now look at the two lines on the chart. The blue line climbs to about 99%: the network gets nearly every training digit right. The orange line stalls about five points lower. That gap is the overfitting from chapter 16, on real data. Part of what the network learned is "what digits look like", and part is "what these particular 8,000 digits look like". Only the first part helps on the test.

Then spend a minute with the ones it got wrong. Some are fair: a 4 with a closed top really does look like a 9, and you might have to squint yourself. Others are easy for you and the network is still wrong, sometimes while claiming to be very sure: in our runs, between 18 and 39 of the mistakes came with more than 90% confidence. Both kinds matter. A test score of 95% does not mean "nearly perfect". It means that one digit in every twenty is wrong, and the network doesn't always know which one.

Draw for it

Those 2,000 test digits were written decades ago by strangers. The real test of a handwriting reader is your handwriting.

▸ try it

Write a digit

Draw one digit in the box with your mouse or finger. Make it any size, anywhere. The small picture is what the network actually receives, and the bars show how sure it is of each answer. Try all ten digits.

This uses the network from the widget above.

tidy-up on
tidy-up off
what the network actually sees
?
draw a digit

With the tidy-up on, it should read most of what you write. Now switch the tidy-up off and draw the same digits again: small, or up in a corner. Watch the small picture. The network is suddenly hopeless, and nothing about the network changed.

So what is the tidy-up? The people who made MNIST didn't just scan handwriting and save it. They put every single digit through the same routine, and with the tidy-up on, your drawing goes through it too:

  1. Find the box around the ink. That's the dashed orange box on your drawing.
  2. Resize it to fit a 20 × 20 square, without stretching it out of shape. A tall thin 1 stays tall and thin. (We also redraw your line at the thickness of the pens in MNIST.)
  3. Find the ink's center of mass: the point where the drawing would balance on a fingertip, marked with the orange cross. Slide the digit so that point sits in the exact middle of the 28 × 28 frame.
  4. Average. Each of the 784 pixels gets the average amount of ink in the patch of drawing it covers. That's where the soft gray edges come from.

With the tidy-up off, the network gets your drawing shrunk straight down to 28 × 28, wherever and however big you drew it. Every digit it ever trained on filled the middle of the frame. A small 7 in the corner is nothing like any 7 it has seen.

Getting data into the shape a model expects is called preprocessing, and it is a big part of real machine-learning work.

A model only works on inputs that look like the ones it was trained on. Feed it something else and it doesn't say "I don't know". It still picks a digit, and often sounds sure about it.

Honesty note: even with the tidy-up, this network will misread some of your digits. A mouse is not a pen, and you don't write like an American student in the 1990s. If you put a bar through your 7 or a flag and a foot on your 1, the network has seen very few like yours. That's the same lesson again, and it's why the headline model in chapter 24 had never heard of anything invented after 2004.

What is each neuron looking for?

Each hidden neuron has 784 weights, one for every pixel. That's the same count as the picture, so we can do something sneaky: lay one neuron's weights out as a 28 × 28 picture of their own and look at what the neuron cares about.

In the tiles below, blue marks a positive weight: ink on that pixel pushes the neuron toward firing. Red marks a negative weight: ink there pushes it toward staying quiet. White means the weight is close to zero, so the neuron ignores that pixel.

▸ try it

The 64 hidden neurons

One tile per hidden neuron. Press the button to retrain from scratch and watch the tiles form: they begin as random speckle and turn into smudgy shapes in the first second or two. Then click any tile to see which test digits excite that neuron most, and which answers it votes for.

Pick a neuron

Train the network, then click a tile.

You might have hoped for 64 neat little pictures of digits. You get swirls and smudges instead. Each neuron is a rough template that says "I like ink here and I don't like ink there". One might like a stroke across the top and hate ink in the middle, which makes it fire for 7s and stay quiet for 4s. No single neuron reads a digit. The ten output neurons listen to all 64 smudge detectors at once and add up their votes.

Four things worth spotting:

  • The edges stay faintly speckled. Nobody writes in the corners: 110 of the 784 pixels are blank in every one of the 8,000 training digits. A weight multiplied by 0 changes nothing, so it never gets any blame, so training never moves it. Those weights are still the small random numbers they started as.
  • A few tiles stay faint all over. In our runs, between 4 and 14 of the 64. Their weights have hardly moved from the random start. Click one: if it fires at all it fires weakly, and its votes are small. Training found the other neurons more useful and spent most of its nudges on them.
  • Now and then a neuron goes silent. In about half of our runs, one or two neurons ended up firing for no digit at all. ReLU sends out 0, so no blame flows back, so the neuron can't learn its way out. That is called a dead neuron. If you click a tile and the panel says it never fires, you've found one.
  • Train again and every tile is different. Different random start, different 64 detectors, about the same score. There isn't one right set of weights. There are many, and training finds one of them.

Break it

Now for the blind spot. Go back to the tiles for a second: every weight belongs to one exact pixel. A neuron that likes ink at row 9, column 14 has no opinion at all about row 9, column 17, unless it happened to learn that separately.

The network reads about 95% of the test digits. Suppose we slide every test digit 3 pixels to the right (about a tenth of the picture's width) and test it again. What happens?

Fewer than half. In our runs the score fell from about 95% to about 46%. (Sliding up or down is even worse: about 36%.) You would hardly notice that a digit had moved three pixels. To this network it has become a different picture.

▸ try it

Nudge a digit

Here is a test digit the network reads correctly. Use the arrows to slide it one pixel at a time. No tidy-up comes to the rescue here: the network gets exactly the picture you see. The map shows every nudge at once: green where the network still gets it right, red where it doesn't.

This uses the network from the training widget.

every nudge up to 6 pixels
(click a square to jump there)
?

For most digits the green patch is small. A typical digit keeps about 36 of the 169 squares green: a patch around six pixels across, somewhere near the middle. Step outside it and the network reads the digit as something else, and may tell you it's sure. Averaged over all 2,000 test digits and all four directions, in our runs a nudge of 1 pixel dropped the score from about 95% to about 89%. Two pixels: about 67%. Three: about 41%. Four: about 22%. Five or six pixels and it's back near 10%, the score for knowing nothing.

This is why MNIST's makers centered every digit so carefully, and why your drawings needed the tidy-up. The network never learned "a 7 is a bar on top with a slanted stroke under it". It learned "a 7 has ink on these particular pixels".

The scramble experiment

Here's a stranger way to see the same thing. Take every picture and shuffle its 784 pixels, like cutting it into 784 tiny squares and dealing them back in a different order. Use the same shuffle for every picture, training and test. A 3 is now a spray of dots. So is a 7. You can't tell them apart.

We scramble all 10,000 pictures the same way and train a brand-new network on the scrambled ones. How well will it read scrambled test digits?

About as well as before. Run it and see.

▸ try it

Scramble every picture

The top row shows six test digits. The bottom row is the same six after the shuffle. Press the button to train a fresh network that only ever sees scrambled pictures, and compare its test accuracy with the network you trained earlier.

Train the first network before running this.

epoch: 0 / 10 normal pictures, test: ? scrambled pictures, test: ?

Both lines are test accuracy. Blue = the network trained on normal pictures. Orange = the new network trained on scrambled ones.

Its neurons, unscrambled

After training, eight of the new network's neurons appear here twice: first as the network stores them, then with every pixel put back where it came from.

The two lines land almost on top of each other. We trained the normal network 70 times, and it finished between 93.7% and 95.2% on the test digits. We trained a scrambled one a dozen times, and it finished between 94.0% and 95.0%. Scrambling makes no difference you can measure. The network reads scrambled digits that no human could, and it isn't doing anything clever. It simply never knew the pixels had an order.

Think about what a layer like this actually is. Every hidden neuron is connected to every input, which is why it's called a fully connected layer. Input 301 and input 302 are just two wires. Nothing in the network says they sit side by side in the picture, so shuffling the wires costs it nothing. (The last panel shows it: put the pixels back and the scrambled network's neurons turn out to be the same kind of smudges as before.)

That is a real weakness, not a party trick. Everything that makes a picture a picture depends on which pixels are next to each other: edges, corners, loops, strokes. A network that ignores neighbors has to learn "a stroke at this spot" and "the same stroke two pixels over" as separate facts, 784 inputs at a time. It works on small, carefully centered digits. On a real photo, where a cat can be anywhere in the frame, it struggles badly.

To a computer, a picture is a grid of numbers. A plain network can learn to read handwriting from those numbers, but it treats them as a bag of 784 unrelated inputs. It has no idea which pixels are neighbors, so it only works when the input looks just like its training data.

What we want is a network that knows pixels have neighbors, and that can learn "this is what the end of a stroke looks like" once and then spot it anywhere in the picture. That's chapter 28, where the network gets eyes.

▸ run real python

Run the digit reader

This is digits.py: the same network in about 120 lines of numpy. It reads digits.bin, draws one digit using letters so you can see the grid of numbers, trains for 10 epochs printing the accuracy after each one, then slides the test digits sideways to measure the blind spot. In the browser the whole thing takes roughly half a minute the first time, including loading Python.

digits.py Python loads on first run
(click "run" to execute)
Challenge

Three things to try. First, in the script, change SCRAMBLE = False to SCRAMBLE = True and run it again: the digit it prints turns into confetti, so does the test accuracy drop? Second, set it back and try HIDDEN = 8. How well can just eight smudge detectors read digits? Third, no code at all: with the tidy-up on, try to draw a digit that any person would read easily but the network gets wrong. When you find one, work out what makes it different from the digits in the "twenty test digits" row.

A 28 × 28 picture goes into this chapter's network as…

One number per pixel: how much ink is there. The grid is flattened row by row into a list of 784, and each one goes to its own input.

Your drawings were read far better with the tidy-up on. Why?

The network only ever saw digits that filled the middle of the frame. The tidy-up doesn't improve your handwriting or peek at the answer. It makes your input look like the training data, which is the only kind of input the network can handle.

Scrambling the pixels the same way in every picture didn't hurt the network. What does that tell you?

A fully connected layer wires every input to every neuron and treats them all alike. It has no idea which pixels are neighbors, so a new order costs it nothing. That's also why sliding a digit a few pixels breaks it.