Chapter 15: Training on a real dataset
150 real flowers and the four habits of honest training.
Every dataset so far was typed in by hand: 4 rows for the screen-time rule, 8 for the snack picker, 8 meals. Real machine learning learns from data somebody collected out in the world, and that changes how careful you have to be.
The classic first real dataset is iris. In the 1930s the botanist Edgar Anderson measured 150 iris flowers, 50 from each of three species, and the statistician Ronald Fisher made the table famous in a 1936 paper. Each flower has four measurements in centimeters: the length and width of a petal, and the length and width of a sepal (the green leaf-like part under the petals). The job is to name the species from those four numbers.
It is tiny by modern standards (chapter 18 has the numbers), but it is big enough to need the four habits that every real project depends on.
Habit 1: hold back a test set
You cannot find out how well a network learned by testing it on the same examples it trained on. That is like marking your own homework with the answer sheet open. So before training, set some of the data aside (here 30 flowers out of 150) and never train on it. The 120 you train on are the training set. The 30 you hold back are the test set. Accuracy on the test set is the only number that tells you how the network will do on flowers it has never seen.
Habit 2: put the inputs on the same scale
Iris petals are between 1.0 and 6.9 cm long but only 0.1 to 2.5 cm wide. A network multiplies inputs by weights, so a column with bigger numbers shouts louder and gets bigger weight updates, for no good reason. The fix is to standardize each column: subtract the column's average, then divide by its standard deviation (a measure of how spread out the column is). Afterward every column is centered on 0 with about the same spread.
One detail matters: work out the average and spread from the training flowers only, then apply that same shift to the test flowers. If the test flowers helped set the scale, a little information about them would have leaked into training.
Habit 3: feed it in mini-batches
You met mini-batches in chapter 13: shuffle the training set, take a small handful, do forward pass, backward pass and weight update on just that handful, then move on. The script below uses handfuls of 16 flowers. One trip through all 120 training flowers is still one epoch; it now contains 8 weight updates instead of 1.
Habit 4: watch both scores, on both sets
Loss is what the math pushes down. Accuracy, the fraction of examples the network gets right, is what people care about. Print accuracy for the training set and the test set as you go. When the two stay close, the network is learning something general. When training accuracy keeps climbing while test accuracy stalls or drops, the network has started memorizing. That is overfitting, and it is the whole subject of chapter 16.
Before you run anything: there are three species with the same number of flowers each. What accuracy would a "model" get by guessing at random?
One in three. That number is the baseline: the score you get for knowing nothing. An accuracy only means something compared with its baseline. 60% would be poor for a coin-flip question and impressive for a hundred-way one.
Train on 150 real flowers
This is neuro7.py. It is the snack picker's network grown slightly (4 → 16 → 3 instead of 3 → 6 → 3) with all four habits added. It loads iris from scikit-learn, a popular Python toolkit, and prints training and test accuracy every 20 epochs.
Heads up: the first run on this page has to download scikit-learn into your browser as well as Python, so it takes noticeably longer than the other scripts. Later runs are quick. To run it on your own computer instead, get the code download described in Build your own.
What just happened
After a single epoch the network is already at 0.567 on the test flowers, well above the one-in-three baseline. It then climbs and levels off at 0.967 on both sets: 116 of the 120 training flowers and 29 of the 30 test flowers. Training and test accuracy finishing together is exactly what you hope to see. The network learned something about irises in general, not just about the 120 it studied.
Keep the result in proportion, though. Iris is a very easy dataset: almost any method scores above 90% on it. It is a warm-up, and the habits are the point.
The only score that counts is the one on examples the network never trained on. Hold back a test set, scale your inputs using the training set alone, train in mini-batches, and watch training and test scores side by side.
In the call to train_test_split, change random_state=1 to 2, then 3, 4 and 5, running the script each time. That number picks which 30 flowers are held back. The final test accuracy comes out as 1.000, 0.867, 1.000 and 0.967. The network and the training recipe did not change at all. So what does that spread tell you about trusting a score measured on only 30 examples?
Why is a test set held back and never used for training?
A network can score well on its training examples just by memorizing them. Only unseen examples can show whether it learned the pattern.
One input column runs from 0 to 1 and another from 0 to 1,000. What is the risk if you train without scaling them?
Inputs are multiplied by weights, so a column a thousand times bigger pushes a thousand times harder, both on the output and on its own weight's gradient. Training can still work, but it becomes lopsided and touchy. Standardizing puts every column on an equal footing.
Your model scores 97% on a test set of 30 flowers. A friend's scores 93% on the same 30. Is yours definitely the better model?
Each test flower is worth 3.3 points, so 97% against 93% is 29 right against 28 right. A different choice of 30 test flowers could easily flip it. Small test sets give blurry scores.