Neuro 13
0 / 37 chapters finished

Chapter 19: Checkpoint: learning

What to remember, a mixed quiz, and a project with your own data.

At the start of this part a network could say yes or no about four examples you could hold in your head. Since then you have made one choose from a list, looked inside its activation function, rolled a ball down a loss landscape, passed blame backward by hand, trained on real flowers, watched a network memorize, and stacked layers until a spiral came apart. This is the place to check it all stuck, and then to try it on data that is yours.

Eight things to remember

  1. To choose between several answers, a network gives each one its own output neuron, and softmax turns the scores into probabilities that add up to 1.
  2. The loss is one number for how wrong the network is, and cross-entropy sets it by how surprised the network was by the right answer.
  3. An activation function puts a bend between layers, which is what makes extra layers worth having, and its slope decides how much of the error signal survives the trip backward.
  4. Words go into a network as one-hot inputs, and the hidden layer learns which combinations of clues matter.
  5. Learning is gradient descent: find the slope of the loss for every weight and move each one a small step downhill, with the learning rate setting the size of the step.
  6. Backpropagation finds all of those slopes in one sweep, by passing the blame for a mistake backward from the output with the chain rule.
  7. The only score that counts is the one on a test set the network never trained on, because a network can ace its training data just by memorizing it.
  8. Depth lets each layer reuse what the layers below it built, and scaling up changes how much there is of everything (layers, parameters, data, chips) but not the math.

Mixed quiz

Seven questions from all over Part 2. If one trips you up, the "why" under it says which chapter to revisit.

A network looks at a picture and says: cat 0.7, dog 0.2, fox 0.1. The picture is a dog. Which number sets the cross-entropy loss for this example?

The loss is −log of the probability given to the right answer, here −log(0.2) ≈ 1.61. The network was fairly surprised, so the loss is fairly big. See chapter 10.

After a lot of training, the last layers of a deep sigmoid network have changed a great deal, but its first layers have hardly moved. What is the most likely reason?

Each sigmoid layer multiplies the signal by a slope of 0.25 at best. A few layers of that and there is almost nothing left. This is the vanishing gradient from chapter 11, and ReLU's slope of 1 is the usual cure.

Which of these does training not change?

The shape of the network is a hyperparameter: a person picks it before training starts. Training only adjusts the weights and biases inside that shape. See chapter 12.

Your loss is going down smoothly, but so slowly that an hour of training has barely moved it. Which knob would you try first?

Smooth but slow means the steps are too short. Bigger steps cover the ground faster, up to the point where they start to overshoot. You saw both ends on the slider in chapter 13.

During backpropagation on one example, a weight's input happened to be 0. What nudge does that weight get?

A wire's share of the blame is multiplied by what was flowing through it. An input of 0 means it carried nothing and contributed nothing to the mistake, so there is nothing to correct. See chapter 14.

A network scores 99% on its training set and 62% on its test set. What has happened, and what might help?

A big gap between the two scores is the signature of overfitting, and training for longer usually widens it. The three cures are in chapter 16.

A one-hidden-layer network gets 55% on a tangled two-color puzzle with equal numbers of each color. What should you make of that score?

A score only means something next to its baseline, and five points above a coin flip is close to nothing. More neurons, or the same neurons in two layers, give the network more to work with, as on the spiral in chapter 17.

Build it: your own dataset

Every dataset in this part was chosen for you. Time to make one. You are going to write a small table, hold part of it back, train a network on the rest, and find out whether it learned your pattern or just memorized your rows.

  1. Pick a question that has a short list of answers and that you can answer for lots of things. Would I like this game? Which of my friends would pick this snack? Is this animal a good pet for a small apartment?
  2. Pick three or four clues, each one a word from a short list, like pace: calm or frantic.
  3. Write about 20 rows. Real opinions make the best data.

The script comes with a starter table so you can see it work first. It is made up: the taste in games of an imaginary friend who will happily play almost anything with other people, can't stand puzzles in a group, and likes only calm games when playing alone. There are 20 rows, one for every mix of five kinds of game, solo or with friends, calm or frantic.

The script will hide 5 of the 20 rows and train on the other 15. One hidden row is puzzle, friends, frantic, where the truth is "no." Almost every other game with friends is a "yes." What will the network say about a row it has never seen?

It says no. The row puzzle, friends, calm is still in the training set, and that was enough for the network to learn that "puzzle" and "friends" together mean trouble. Run it and look for that line.

▸ run real python

Train a network on your own table

This is my_data.py. The table is MY_TABLE near the top: each row is a few clue words and then the answer. Press run with the starter table first and read the output from top to bottom: the loss falling, the five test rows with the network's guess for each, the two scores, the baseline, and a verdict.

my_data.py Python loads on first run
(click "run" to execute)

What you should have seen

It learned. With random starting weights the network gets 6 of the 15 training rows and 2 of the 5 test rows. By epoch 300 it has all 15 and all 5. The baseline, always answering "yes," would have scored 2 of 5 on those test rows. And the hidden puzzle row comes out as "no," 85% sure.

Now break it. Change SCRAMBLE = False to True and run again. This shuffles the answers so they have nothing to do with the clues. The network still gets 15 of 15 training rows right. But on the test rows it scores 1 of 5, when the baseline would have scored 4. It memorized fifteen random answers. Notice that the training score was perfect both times. Only the held-back rows could tell learning from memorizing.

Now doubt it. Put SCRAMBLE back to False and change SEED, which decides which five rows are hidden and what the starting weights are. With the starter table, the seeds from 0 to 15 give test scores all the way from 0 of 5 (try SEED = 15) to 5 of 5. Five test rows make a very blurry measuring stick, and with only 20 rows in total, hiding the wrong five can remove exactly the examples the network needed. This is why real test sets have thousands of rows.

Now make it yours. Replace the rows in MY_TABLE with your own, and change COLUMNS and ASK to match. A few tips:

  • The answer goes last in every row, and you can have more than two different answers.
  • Spelling counts. To the network, "Calm" and "calm" are two different words.
  • If the verdict says it memorized, that is a real finding, not a failure. Either your clues do not predict the answer, or 15 rows were not enough to show how.
Challenge

Add a column that cannot possibly matter, such as the day of the week you typed each row. Does the test score change? Then, with your own table, try HIDDEN = 64 and EPOCHS = 5000. Look at how sure the network is about each test row, right or wrong. Is a bigger network trained for longer a better one here? And one with no computer at all: explain how a network learns to someone at home, using only these six words for the technical parts: input, weight, mistake, nudge, layer, prediction.

Next: machines that read and write

Your table was made of words, and the script turned every one of them into a yes/no switch: nine switches for the starter table. That works for a handful of words. A language has hundreds of thousands, and they arrive in sentences of any length, where the order matters. Part 3 is about how networks cope with that. It starts with turning text into numbers (chapter 20), builds a small model that invents names one letter at a time (chapter 21), and works up to what a chatbot is really doing when it answers you (chapter 23).