Neuro 13
0 / 37 chapters finished
Part 5Your turn13 min read

Chapter 36: Build your own

Three capstone projects, from an afternoon to a weekend.

You have read how it works. The way to find out whether you understand it is to make something that nobody set up for you. Here are three projects: one for an afternoon, done right on this page; one for a weekend, on your own computer; and one that is a real investigation, the kind you could enter in a science fair. Each ends with something to show another person.

Project 1 (an afternoon): teach it your own symbols

In chapter 27 a network learned ten digits from 8,000 examples that strangers wrote decades ago. This time there are no strangers. You invent the symbols, you draw every example, and you find out for yourself how many it takes.

Pick two to four symbols that look different from each other: a star, a lightning bolt, a spiral, your initials. Draw about a dozen of each. Then train a network on them and draw new ones for it to read.

You draw twelve stars and twelve bolts, all about the same size, all in the middle of the pad, and train on them. Then you draw a perfectly good star, small, in the top corner. What will the network make of it?

This network has no tidy-up step and no filters. It is the plain kind from chapter 27, which learns "ink on these particular pixels". A small star in the corner puts ink on pixels it has never seen inked, and softmax always picks something. Try it, and then try the fix in step 2.

▸ build it

Teach it your own symbols

Everything here happens in your browser. Your drawings are kept on this device so they are still here when you come back, and they are never sent anywhere.

1 Draw your examples

Click a symbol to choose it, type a name for it, then draw it on the pad and press keep. Draw each one a little differently, the way you naturally would.

a drawing the network trains on every fourth drawing is held back as a test, and never trained on. Click a drawing to delete it.

2 Train

The network is the plain kind from chapter 27: 784 inputs, one hidden layer, one output per symbol. It starts from random weights every time you press train.

train on only my drawings
also train on slid copies

"Slid copies" makes 8 extra copies of every training drawing, each pushed up to 4 pixels in a random direction. Making extra training examples by changing the ones you have is called data augmentation.

epoch: 0 / 60 pictures trained on: ? weights: ?
trained onpicturesits training drawingsheld-back drawingsheld-back drawings, slid 3 pixels

3 Draw for it

Draw one of your symbols. The small picture is the 28 × 28 grid of numbers the network actually receives. Then use the buttons to slide or shrink what you drew, and watch the bars.

what the network sees
?
train the network first

What to look for

Your numbers depend entirely on what you drew, so there are none to quote here. These are the things worth catching.

  • The first two columns flatter it. With a dozen drawings of each symbol, the network usually gets every training drawing right, and most or all of the held-back ones. Before you celebrate, read the last column. Those are the same held-back drawings, pushed three pixels up, down, left and right. A person would not notice the difference.
  • It memorized where the ink was. That is chapter 16 and the blind spot from chapter 27 arriving together. A few dozen pictures are far too few for tens of thousands of weights to learn what a star is. They are enough to learn where your stars happened to be.
  • The lever. Switch to "also train on slid copies" and train again. Now compare the two rows of the table, especially the last column. You did not draw anything new. You showed the network the same drawings in slightly different places, which told it that position is not what makes a star a star.
  • What the lever does not fix. Press shrink a few times. Slid copies taught it about sliding and nothing about size. A model learns the kinds of variety it was shown, and no others.
  • Somebody else's hand. Ask another person to draw your symbols in step 3 without seeing your examples. Then ask them to add a few of their own to step 1 and train again.

Honesty note: chapter 28 showed a better answer to the sliding problem than brute force: build "the same pattern can be anywhere" into the wiring. This project deliberately uses the plain network, so that you run into the problem yourself and feel what data augmentation buys.

To show someone: train it on symbols only you know the meaning of, and have them guess which is which from the bars alone. Then let them try to draw one it gets wrong.

Project 2 (a weekend): your data, your computer

The scripts in this book are short on purpose, so that you can take one and point it at something of your own. Pick one of these, or invent a third.

A name inventor for a list you collect. names.py from chapter 21 will learn from any text file with one word per line. Collect a few hundred words of one kind: towns in your country, star names, breeds of dog, characters you made up. Save them in a file, lowercase, using only the letters a to z. In the script, change names.txt to your file's name, and run it. Things to expect: a list of fewer than about 200 words gets memorized (you saw that with the 129 foods), and the more alike your words are, the more convincing the inventions.

A sorter for text you gather. neuro10.py from chapter 24 reads a file with two columns, a topic and a piece of text. Build your own: a few hundred lines of two to four kinds that you can label quickly. Lines from different books. Song titles by decade. Your own messages. Save it as a .csv file whose first line is topic,text, the same as in headlines.csv, and put double quotes around any text that has a comma in it. In the script, change headlines.csv to your file's name and change the list of topics at the top to yours. The script trains on the first four fifths of your rows and tests on the rest, so shuffle the rows first: if every line of one kind comes last, it never gets to train on that kind. Look at what it gets wrong on the held-back rows before you look at its score.

When it does not work, and at first it won't, these are the usual reasons:

What you seeWhat it usually means
An error naming a fileThe script is not being run from the folder the file is in, or the name is spelled differently.
Training score high, held-back score at the baselineIt memorized. You need more examples, or your labels do not follow from the text.
Both scores stuck at the baselineIt is not learning at all. Check that the labels are spelled the same way every time, then try more epochs.
The loss jumps around or prints nanThe learning rate is too big for your data. Halve it and run again.
It works suspiciously wellLook for a shortcut, like the news-agency tags in chapter 25. Is there a give-away word in every example of one kind?

To show someone: the three funniest things your name inventor wrote, or the one sentence that fools your sorter and your explanation of why.

Project 3 (as long as you like): a real investigation

Scientists who study neural networks do what you did in chapter 33: they change one thing, run it many times, and write down what happened. You can do a real one of these with the tools in this book. A good question is one where you honestly do not know the answer, and where the answer is a number you can measure.

Three to start from:

  • How much does luck matter? In chapter 33, pick one setup and press "new start" twenty times, writing down the epoch at which it reaches 95% each time, or that it never did. How wide is the spread? Does a bigger network make the spread narrower or wider?
  • How many examples is enough? In the symbol teacher above, train with 4, 8, 12, 16 and 20 drawings of each symbol, and record the last column each time. Is the improvement steady, or does it flatten out? Does the answer change with slid copies switched on?
  • Does a creature that explores more end up better off? In chapter 30, run 500 episodes at each of five settings of the exploring slider and record the average points over the last 20. Then dry up the lava and do it again. Which setting would you choose if you did not know whether the maze was going to change?

Whatever you choose, write it up. A lab report can be half a page:

QUESTION     One sentence. What do I want to find out?
PREDICTION   What I think will happen, and why. Written BEFORE I run anything.
METHOD       Exactly what I set, what I changed, and what I measured.
             Enough detail that a friend could repeat it.
RESULTS      A table of what happened. Every run, not just the good ones.
             Include how many random starts I used.
CONCLUSION   Was my prediction right? What is my best explanation?
NEXT         The question this made me want to ask.

Two rules make the difference between an investigation and an afternoon of clicking. Change one thing at a time. And never trust a single run: the depth chapter of this book once taught something false because of exactly that, until somebody repeated the experiment from sixty random starts.

To show someone: your prediction and your results table, side by side. If they disagree, even better.

Run the code on your own computer

Every Python program in this book runs on the page, except the three that need PyTorch. Running them on your own computer is faster, lets you use your own files, and is the only way to do project 2.

  1. Get Python. Download it from python.org and install it. On Windows, tick "Add Python to PATH" in the installer. If the computer is not yours, ask first.
  2. Get the scripts. Download neuro13-code.zip and unzip it. You get a folder called neuro13 with every script and every dataset in it.
  3. Open a terminal in that folder. On a Mac, open the Terminal app, type cd (with a space), drag the folder onto the window and press return. On Windows, open the folder, click the address bar, type cmd and press Enter.
  4. Install numpy, once: python3 -m pip install numpy. On Windows, type python wherever these instructions say python3.
  5. Run something: python3 neuro1.py

neuro7.py also needs scikit-learn (python3 -m pip install scikit-learn). The three PyTorch scripts need more, and chapter 32 explains how to install it.

Every script in the book

ScriptChapterWhat it doesRuns on the page?
neuro1.pyTraining the neuronOne neuron learns the screen-time ruleyes
neuro2.pyWhen one neuron isn't enoughThe same neuron fails to learn XORyes
my_rule.pyCheckpoint: one neuronTeach one neuron a rule you inventyes
neuro3.pyHidden neurons fix itA hidden layer solves XORyes
neuro4.pyPicking from a listThe snack picker: three possible answersyes
neuro5.pyWhat does that "squish" actually do?The snack picker again, with ReLUyes
neuro6.pyMeal guesserThe meal guesser, which has to combine cluesyes
backprop.pyBackpropagation: passing the blame backwardsOne round of backpropagation, every number printedyes
neuro7.pyTraining on a real dataset150 real iris flowers, with a held-back test setyes
overfit.pyLearning vs. memorizingA network memorizes noisy dots, and three curesyes
my_data.pyCheckpoint: learningTrain a network on a table you writeyes
names.pyA language model you can trainA language model that invents namesyes
attention.pyAttention: a fuzzy lookupA network learns where to lookyes
neuro10.pyA model that decides instead of writesSort 10,000 headlines, with honest confidenceyes
my_messages.pyCheckpoint: languageSort messages you writeyes
digits.pyRead my handwritingRead handwritten digitsyes
convnet.pyHow networks seeTrains the convolutional network the chapter usesno: needs PyTorch
diffusion.pyPictures from noiseTurn static into shapesyes (a quick version)
maze.pyLearning by trial and errorA creature learns a maze from rewardsyes
autograd.pyWhat the professionals useA 60-line engine that finds every gradient for youyes
neuro8.pyWhat the professionals useThe iris network in PyTorchno: needs PyTorch
neuro9.pyWhat the professionals useHandwritten digits in PyTorch, on all of MNISTno: needs PyTorch

Understanding something and being able to build it are different skills, and the second one only comes from building. Start smaller than you think you should, change one thing at a time, and show somebody what you made.

When you have picked a project, or finished one, chapter 37 has the best places to keep going.