Chapter 36: Build your own
Three capstone projects, from an afternoon to a weekend.
You have read how it works. The way to find out whether you understand it is to make something that nobody set up for you. Here are three projects: one for an afternoon, done right on this page; one for a weekend, on your own computer; and one that is a real investigation, the kind you could enter in a science fair. Each ends with something to show another person.
Project 1 (an afternoon): teach it your own symbols
In chapter 27 a network learned ten digits from 8,000 examples that strangers wrote decades ago. This time there are no strangers. You invent the symbols, you draw every example, and you find out for yourself how many it takes.
Pick two to four symbols that look different from each other: a star, a lightning bolt, a spiral, your initials. Draw about a dozen of each. Then train a network on them and draw new ones for it to read.
You draw twelve stars and twelve bolts, all about the same size, all in the middle of the pad, and train on them. Then you draw a perfectly good star, small, in the top corner. What will the network make of it?
This network has no tidy-up step and no filters. It is the plain kind from chapter 27, which learns "ink on these particular pixels". A small star in the corner puts ink on pixels it has never seen inked, and softmax always picks something. Try it, and then try the fix in step 2.
Teach it your own symbols
Everything here happens in your browser. Your drawings are kept on this device so they are still here when you come back, and they are never sent anywhere.
1 Draw your examples
Click a symbol to choose it, type a name for it, then draw it on the pad and press keep. Draw each one a little differently, the way you naturally would.
a drawing the network trains on every fourth drawing is held back as a test, and never trained on. Click a drawing to delete it.
2 Train
The network is the plain kind from chapter 27: 784 inputs, one hidden layer, one output per symbol. It starts from random weights every time you press train.
"Slid copies" makes 8 extra copies of every training drawing, each pushed up to 4 pixels in a random direction. Making extra training examples by changing the ones you have is called data augmentation.
| trained on | pictures | its training drawings | held-back drawings | held-back drawings, slid 3 pixels |
|---|
Your drawings have changed since this network was trained. Train again to bring the scores up to date.
3 Draw for it
Draw one of your symbols. The small picture is the 28 × 28 grid of numbers the network actually receives. Then use the buttons to slide or shrink what you drew, and watch the bars.
What to look for
Your numbers depend entirely on what you drew, so there are none to quote here. These are the things worth catching.
- The first two columns flatter it. With a dozen drawings of each symbol, the network usually gets every training drawing right, and most or all of the held-back ones. Before you celebrate, read the last column. Those are the same held-back drawings, pushed three pixels up, down, left and right. A person would not notice the difference.
- It memorized where the ink was. That is chapter 16 and the blind spot from chapter 27 arriving together. A few dozen pictures are far too few for tens of thousands of weights to learn what a star is. They are enough to learn where your stars happened to be.
- The lever. Switch to "also train on slid copies" and train again. Now compare the two rows of the table, especially the last column. You did not draw anything new. You showed the network the same drawings in slightly different places, which told it that position is not what makes a star a star.
- What the lever does not fix. Press shrink a few times. Slid copies taught it about sliding and nothing about size. A model learns the kinds of variety it was shown, and no others.
- Somebody else's hand. Ask another person to draw your symbols in step 3 without seeing your examples. Then ask them to add a few of their own to step 1 and train again.
Honesty note: chapter 28 showed a better answer to the sliding problem than brute force: build "the same pattern can be anywhere" into the wiring. This project deliberately uses the plain network, so that you run into the problem yourself and feel what data augmentation buys.
To show someone: train it on symbols only you know the meaning of, and have them guess which is which from the bars alone. Then let them try to draw one it gets wrong.
Project 2 (a weekend): your data, your computer
The scripts in this book are short on purpose, so that you can take one and point it at something of your own. Pick one of these, or invent a third.
A name inventor for a list you collect. names.py from chapter 21 will learn from any text file with one word per line. Collect a few hundred words of one kind: towns in your country, star names, breeds of dog, characters you made up. Save them in a file, lowercase, using only the letters a to z. In the script, change names.txt to your file's name, and run it. Things to expect: a list of fewer than about 200 words gets memorized (you saw that with the 129 foods), and the more alike your words are, the more convincing the inventions.
A sorter for text you gather. neuro10.py from chapter 24 reads a file with two columns, a topic and a piece of text. Build your own: a few hundred lines of two to four kinds that you can label quickly. Lines from different books. Song titles by decade. Your own messages. Save it as a .csv file whose first line is topic,text, the same as in headlines.csv, and put double quotes around any text that has a comma in it. In the script, change headlines.csv to your file's name and change the list of topics at the top to yours. The script trains on the first four fifths of your rows and tests on the rest, so shuffle the rows first: if every line of one kind comes last, it never gets to train on that kind. Look at what it gets wrong on the held-back rows before you look at its score.
When it does not work, and at first it won't, these are the usual reasons:
| What you see | What it usually means |
|---|---|
| An error naming a file | The script is not being run from the folder the file is in, or the name is spelled differently. |
| Training score high, held-back score at the baseline | It memorized. You need more examples, or your labels do not follow from the text. |
| Both scores stuck at the baseline | It is not learning at all. Check that the labels are spelled the same way every time, then try more epochs. |
The loss jumps around or prints nan | The learning rate is too big for your data. Halve it and run again. |
| It works suspiciously well | Look for a shortcut, like the news-agency tags in chapter 25. Is there a give-away word in every example of one kind? |
To show someone: the three funniest things your name inventor wrote, or the one sentence that fools your sorter and your explanation of why.
Project 3 (as long as you like): a real investigation
Scientists who study neural networks do what you did in chapter 33: they change one thing, run it many times, and write down what happened. You can do a real one of these with the tools in this book. A good question is one where you honestly do not know the answer, and where the answer is a number you can measure.
Three to start from:
- How much does luck matter? In chapter 33, pick one setup and press "new start" twenty times, writing down the epoch at which it reaches 95% each time, or that it never did. How wide is the spread? Does a bigger network make the spread narrower or wider?
- How many examples is enough? In the symbol teacher above, train with 4, 8, 12, 16 and 20 drawings of each symbol, and record the last column each time. Is the improvement steady, or does it flatten out? Does the answer change with slid copies switched on?
- Does a creature that explores more end up better off? In chapter 30, run 500 episodes at each of five settings of the exploring slider and record the average points over the last 20. Then dry up the lava and do it again. Which setting would you choose if you did not know whether the maze was going to change?
Whatever you choose, write it up. A lab report can be half a page:
QUESTION One sentence. What do I want to find out?
PREDICTION What I think will happen, and why. Written BEFORE I run anything.
METHOD Exactly what I set, what I changed, and what I measured.
Enough detail that a friend could repeat it.
RESULTS A table of what happened. Every run, not just the good ones.
Include how many random starts I used.
CONCLUSION Was my prediction right? What is my best explanation?
NEXT The question this made me want to ask.
Two rules make the difference between an investigation and an afternoon of clicking. Change one thing at a time. And never trust a single run: the depth chapter of this book once taught something false because of exactly that, until somebody repeated the experiment from sixty random starts.
To show someone: your prediction and your results table, side by side. If they disagree, even better.
Run the code on your own computer
Every Python program in this book runs on the page, except the three that need PyTorch. Running them on your own computer is faster, lets you use your own files, and is the only way to do project 2.
- Get Python. Download it from python.org and install it. On Windows, tick "Add Python to PATH" in the installer. If the computer is not yours, ask first.
- Get the scripts. Download neuro13-code.zip and unzip it. You get a folder called
neuro13with every script and every dataset in it. - Open a terminal in that folder. On a Mac, open the Terminal app, type
cd(with a space), drag the folder onto the window and press return. On Windows, open the folder, click the address bar, typecmdand press Enter. - Install numpy, once:
python3 -m pip install numpy. On Windows, typepythonwherever these instructions saypython3. - Run something:
python3 neuro1.py
neuro7.py also needs scikit-learn (python3 -m pip install scikit-learn). The three PyTorch scripts need more, and chapter 32 explains how to install it.
Every script in the book
| Script | Chapter | What it does | Runs on the page? |
|---|---|---|---|
neuro1.py | Training the neuron | One neuron learns the screen-time rule | yes |
neuro2.py | When one neuron isn't enough | The same neuron fails to learn XOR | yes |
my_rule.py | Checkpoint: one neuron | Teach one neuron a rule you invent | yes |
neuro3.py | Hidden neurons fix it | A hidden layer solves XOR | yes |
neuro4.py | Picking from a list | The snack picker: three possible answers | yes |
neuro5.py | What does that "squish" actually do? | The snack picker again, with ReLU | yes |
neuro6.py | Meal guesser | The meal guesser, which has to combine clues | yes |
backprop.py | Backpropagation: passing the blame backwards | One round of backpropagation, every number printed | yes |
neuro7.py | Training on a real dataset | 150 real iris flowers, with a held-back test set | yes |
overfit.py | Learning vs. memorizing | A network memorizes noisy dots, and three cures | yes |
my_data.py | Checkpoint: learning | Train a network on a table you write | yes |
names.py | A language model you can train | A language model that invents names | yes |
attention.py | Attention: a fuzzy lookup | A network learns where to look | yes |
neuro10.py | A model that decides instead of writes | Sort 10,000 headlines, with honest confidence | yes |
my_messages.py | Checkpoint: language | Sort messages you write | yes |
digits.py | Read my handwriting | Read handwritten digits | yes |
convnet.py | How networks see | Trains the convolutional network the chapter uses | no: needs PyTorch |
diffusion.py | Pictures from noise | Turn static into shapes | yes (a quick version) |
maze.py | Learning by trial and error | A creature learns a maze from rewards | yes |
autograd.py | What the professionals use | A 60-line engine that finds every gradient for you | yes |
neuro8.py | What the professionals use | The iris network in PyTorch | no: needs PyTorch |
neuro9.py | What the professionals use | Handwritten digits in PyTorch, on all of MNIST | no: needs PyTorch |
Understanding something and being able to build it are different skills, and the second one only comes from building. Start smaller than you think you should, change one thing at a time, and show somebody what you made.
When you have picked a project, or finished one, chapter 37 has the best places to keep going.