Chapter 5: What training actually changes
A trained model is its numbers and nothing else.
Before the puzzles get harder, be very clear about what training is allowed to change.
Training does not change the examples. The input table stays the same. The right answers stay the same. If the row says [1, 0] → YES, training is not allowed to rewrite that row.
Training does not grow new neurons either. If the code says there is one neuron, training has to work with one neuron. If the code says there are 4 hidden neurons, training has to work with those 4. The network does not quietly add a fifth because it got stuck.
The only things training changes are the weights and biases. Together they are called the model's parameters: the adjustable numbers inside the network. When you hear that a famous model has "seven billion parameters," that is a count of these numbers and nothing more. You have already watched this happen: the dark "model" panels in chapter 4 showed the stored numbers being rewritten, and nothing else, while training ran.
| Thing | Who changes it? |
|---|---|
Input examples, like [1, 0] | The human writing the dataset |
Correct answers, like YES | The human writing the dataset |
| Number of neurons, and how they connect | The human designing the network |
| Weights and biases | Training |
So when chapter 8 asks you to change hidden = 4 to hidden = 2, that is you changing the design before training starts. After that, training tries to find weights and biases that make your design work. Sometimes no such numbers exist, and chapter 7 is about exactly that.
The human chooses the shape of the network. Training fills that shape with useful numbers. A trained model is those numbers and nothing else.
If a one-neuron network cannot solve a puzzle, what is training allowed to try changing? What is it not allowed to change? Who would have to step in?
You download a trained model from the internet. What is in the file?
The examples are not kept and no rules are written down. Whatever the model "knows" is stored as parameters, and the file is a list of them (plus a note describing the network's shape).
A network with one neuron keeps failing at a puzzle. What can training do about it?
Training only turns the knobs that already exist. Adding neurons or changing the data is a decision for the human.
You train the same network design twice on the same examples, starting from different random weights. What is certain to be the same both times?
The shape is fixed by the human before training begins. The numbers training finds can differ from run to run, as you will see in chapter 8.