Neuro 13
0 / 37 chapters finished

Chapter 31: Checkpoint: seeing and creating

What to remember, a mixed quiz, and a project you can show off.

Parts 1 to 3 were about numbers in a table and words in a row. In this part the networks got out more. One learned to read handwriting. One was given eyes. One turned a screen of static into a heart, and a creature with no network at all learned a maze from nothing but points. This is the place to check that it stuck, and then to make a picture nobody has ever drawn, of a shape that you design.

Eight things to remember

  1. To a computer a picture is a grid of numbers, one per pixel, and a plain network reads it by flattening that grid into one long list.
  2. A fully connected layer has no idea which pixels are neighbors, which is why it read scrambled digits as well as normal ones and fell apart when a digit slid three pixels sideways.
  3. A model only works on inputs that look like the ones it trained on, so getting data into the shape it expects (preprocessing) is a real part of the job.
  4. A filter is a tiny grid of weights slid across the whole picture, and because the same weights are used at every stop, it finds its pattern wherever the pattern turns up.
  5. Stack layers of filters with pooling in between and a network finds small patterns, then patterns made of patterns, and nobody designs the filters: training finds them.
  6. A diffusion model learns one small skill, guessing the noise that was mixed into something, and makes a picture by using that skill over and over on pure static.
  7. In a diffusion model the training data decides what is possible, the prompt decides what you get, and the starting static decides the details.
  8. In reinforcement learning nobody supplies the right answer: an agent collects rewards, passes the credit backward through time, has to balance exploring against doing what already works, and will chase exactly the reward you wrote.

Mixed quiz

Seven questions from all over Part 4. If one trips you up, the "why" under it says which chapter to revisit.

A digit reader gets 95% of its test digits right. You hand it a photo of your own 7, written small in the top corner of the frame, and it says "2, 98% sure". What is the best explanation?

The 95% was measured on digits prepared exactly like the training ones. Softmax always hands out probabilities that add up to 1, so the network picks a digit and sounds sure, even for a picture unlike anything it has seen. That was the tidy-up switch in chapter 27.

You shuffle the pixels of every picture in the same way, then train a plain network and a convolutional network on the shuffled pictures. Which one is hurt?

The plain network never knew which pixels sat side by side, so a new order costs it nothing. A filter looks at a small patch and bets that the pixels in it belong together. Shuffle the picture and that bet stops paying. See chapter 28.

A 3 × 3 filter has −1 down its left column, 0 down the middle and +1 down the right. Which patch gives the biggest output?

A filter answers "how much does this patch look like my pattern?" On solid ink the −1s cancel the +1s and the sum is 0. Ink on the left gives a negative number. Only paper-then-ink lines the ink up under the +1s. See chapter 28.

The convolutional network still read digits that had been pushed 4 pixels sideways, though it never trained on a shifted digit. A friend says: "Then it will read a digit lying on its side too." Are they right?

The same filter runs at every position, so a pattern that slides is still found. Turn the pattern and the filter no longer matches it. To cope with that, a network has to be shown turned and resized copies in training, which is called data augmentation. See the fine print in chapter 28.

A diffusion model takes 50 steps to turn static into a heart. What changes from one step to the next?

Making a picture is all forward passes. The weights were fixed when training ended, and the prompt stays the same. What improves is the question being asked: slightly cleaner dots have a slightly sharper answer. See chapter 29.

The same 2,000 specks of static became a star with one switch on and a spiral with another. Where was the knowledge of what a star looks like?

Every picture ruins into the same blob, so the static holds no star. The switch is a single on/off input. Everything about the shape was learned during training, when each ruined dot arrived with a label saying which picture it came from. The switch only picks which of the learned answers to use. See chapter 29.

A trained creature has exploring set to 0%. A shortcut opens that is half as long as its usual route. It never takes it. Why?

Its table still learns from every move it makes. The trouble is which moves it makes: always the one it already believes is best. The number that says "that way hurts" was written when it was true, and only trying the move again could correct it. That is the price of never exploring, from chapter 30.

Build it: a picture nobody drew, of a shape you designed

The diffusion model in chapter 29 knew five pictures, and we chose all five. In this project you teach it a sixth, of your own design, and then watch your shape come out of pure static: 2,000 dots that were never in your drawing. It takes paper, a pencil and about ten lines of typing.

We did the whole project ourselves first, with a rocket. Everything below is what happened.

1. Design it on graph paper

The script draws with a pen. A shape is a list of pen strokes, and a stroke is a list of corners that the pen visits in order, joined by straight lines. So draw a square on paper, mark −1 to 1 across the bottom and −1 to 1 up the side, and sketch something made of straight lines. Then read off the corners.

−1−0.500.51 −1−0.500.51 (0, 1) (0.35, 0.35) (0.35, −0.6) (−0.35, 0.35) (−0.35, −0.6) x: across
Our rocket: a hull (orange, five corners), two fins (blue), a flame (red) and a round window (teal).

Three rules that will save you trouble:

  • Lift the pen, start a new stroke. Our rocket has five strokes: the hull, a fin, the other fin, the flame and the window.
  • To close a shape, come back to where you started. The hull's list begins and ends with (0, 1).
  • For a circle, borrow the script's helper. circle(0, 0.1, 0.15) is a circle centered at (0, 0.1) with radius 0.15.

The size of your drawing and where it sits on the paper make no difference. The script moves every shape to the middle and rescales it.

2. Type it in

In the editor below, find PEN_STROKES (around line 83) and add your shape as a new entry, right after the line for "heart". Ours:

    "rocket": [[(0, 1), (0.35, 0.35), (0.35, -0.6), (-0.35, -0.6), (-0.35, 0.35), (0, 1)],
               [(-0.35, -0.2), (-0.75, -0.9), (-0.35, -0.6)],
               [(0.35, -0.2), (0.75, -0.9), (0.35, -0.6)],
               [(-0.2, -0.6), (0, -1), (0.2, -0.6)],
               circle(0, 0.1, 0.15)],

Count the brackets: square brackets around the whole shape, square brackets around each stroke, round brackets around each corner, and a comma after every stroke.

3. Ask for it

Near the top of the script, line 42 says which shapes to train on. In this page the script uses the short list. Change ["heart", "star"] to ["rocket", "heart"], with your shape's name in place of ours. Put your shape first: the script prints a picture of the first shape's training dots before it does anything else, and you want that picture to be yours.

▸ run real python

Teach the diffusion model your shape

This is diffusion.py from chapter 29, unchanged, waiting for your two edits. Inside this page it runs on its quick setting: two shapes, a small network of 5,906 weights and 4,000 training steps. Press run and read the output from the top: your shape as 2,000 training dots, the same dots being ruined, the loss, and then the pictures it makes from pure static.

diffusion.py Python loads on first run
(click "run" to execute)

4. Read what came out

The training picture comes first, within a second. Check it before you look at anything else. This is your drawing as the network will see it: 2,000 dots sprinkled along your pen strokes. If a line is missing or a fin points the wrong way, the mistake is in your list of corners, and no amount of training will fix it.

Then the loss. Ours began at 0.585 after the first 400 steps and ended at 0.477. It never gets near 0, for the reason chapter 29 gave: static that slides a dot along its own line can't be guessed.

Then the pictures from static. Here is the rocket our run made, from 2,000 dots of pure static. Compare the fins, the flame and the window with the drawing above.

generated from pure static, asked for "rocket"
                      .#oo
                       o##.
                     .#o..#..
                     ##.. .#oo
                   .#o.    o#o
                   #o      o.ooo
                  o#.      .  o#
                  #. o....  .. o#
                 o    o##### . .#o
                 ##.  #o...#oo.#oo
                 #. .#o  o #o. ##
                .#o oo#ooo#o#.. #
                .#o  . .. ...o.o#    .
                o#o   .    ... ##o   .
               o##o      .     ##oo.
              #o#ooo. . .      o##oo
            o## .#.ooo#       oo.#### .
           o#o..#######.o.ooo###o ####..
          .##ooo. . .oo...o#oo .o o####o
         .##..   ..   .#.o#o  .     .o##o.o
         o#...        ..###            #.
                        .              o
   average gap to the nearest training dot: 0.040   (for brand-new real dots it is 0.017)

It is fuzzier than the training picture, because the quick setting gives a small network very little practice. Now read the last line. A copy of a training dot would sit 0.000 away from one. These dots sit 0.040 away on average, on a rocket about 3 wide and 4 tall. Nothing was copied. The network learned where the dots of a rocket tend to be, and put new ones there.

Honesty note: the picture and numbers above are from our run on a laptop. The script's dice are fixed, so your run of the same rocket should come out close to this, but different computers round their arithmetic slightly differently, and over 4,000 steps that can move a few dots and the last digit of a number.

5. When it goes wrong

We made each of these mistakes on purpose to see what the script would say.

What you seeWhat happened
KeyError: 'rokcet'The name in the list on line 42 does not match the name in PEN_STROKES. They have to be spelled the same.
TypeError: list indices must be integers or slices, not tupleA comma is missing between two strokes. Python even asks: "perhaps you missed a comma?"
One side of your shape is missing from the training pictureThe stroke was not closed. Repeat its first corner at the end of the list.
A small part is faint, or missing from the generated pictureThe dots are shared out by length, so a short stroke gets few of them. Draw that part bigger.

6. Make a film strip

The widget in chapter 29 let you scrub through the 50 steps. You can make your script do the same for your own shape. Find the function generate near the bottom. Right after its line that begins x = keep[level - 1], add these two lines, indented to match it:

        if level - 1 in (35, 20, 10):
            show(x, f"cleaned down to noise level {level - 1}")

Run it again. Before each finished picture you now get three more: the cloud at noise level 35, at 20 and at 10. In ours, level 35 was a blob with nothing to see. At level 20 the base and the fins had turned up. At level 10 it was plainly a rocket in a haze, and the last ten steps pulled the haze into lines. Rough decisions first, fine ones last, exactly as chapter 29 said.

One more experiment. Near the top of the script, change TRAIN_STEPS = 4000 if QUICK to TRAIN_STEPS = 200 if QUICK, so the network gets one twentieth of the practice. What will it make from static now?

A scatter with a hint of shape in it. In our run the loss stopped at 0.555, the bottom of the rocket was just about there and the top was a smear. The gap to the nearest training dot was 0.256, more than six times what it was after 4,000 steps. After 200 steps the network has learned roughly where rocket dots live and nothing finer. Put it back to 4000 when you are done.

7. Show someone

Print the training picture and the generated one, or put them side by side on the screen, and ask someone at home which one you drew and which one a network made out of static. Then show them the gap number and explain what a 0.000 would have meant. If you made the film strip, show that too: it is the whole idea of an image generator in four pictures.

Reading a picture, making one and learning a game all ran on ideas you already had. What changed each time was the wiring, such as filters that treat pixels as neighbors, and where the feedback came from: a label, noise we made ourselves, or nothing but a reward.

Challenge

Teach it two shapes of your own at once, say your two initials, by putting both names in the list on line 42. Does each prompt still give the right shape, or do they leak into each other? Then try it on your own computer, where the script does its full job: a bigger network and 30,000 steps. Add your shape's name to the second list on line 42 as well, the one that starts with "smiley", and run python3 diffusion.py. Chapter 36 explains how to set that up. Compare the gap number with the one you got in this page.

Next: your turn

That is the end of the guided tour. You have now met every kind of network this book covers, and Part 5 hands you the controls. Chapter 32 shows the tools professionals use, by having you build a small version of the most important one. Chapter 33 is a lab where you choose the data, the layers and every knob. Chapter 35 is a map of what the biggest systems are doing right now. And chapter 36 has three projects that are yours from start to finish.