Chapter 3: One tiny neuron
Multiply, add, squish. Drag the sliders and watch it decide.
Imagine a little machine. It has some inputs and one output. Each input is a number: it could be the brightness of a pixel, or "did I practice piano? 1 for yes, 0 for no".
The machine does three things in order:
- Multiply each input by its own "importance number" (a weight)
- Add them all together (plus one extra fudge factor called a bias)
- Squish the answer into a number between 0 and 1 using a curve called sigmoid
That output between 0 and 1 is how strongly the neuron "fires": how confident it is that the answer is yes. That's one neuron. Yes, that's the whole thing.
What does sigmoid actually look like?
Sigmoid is a curve that turns "any number" into "a number between 0 and 1". Feed it a very negative number and out comes nearly 0. Feed it a very positive number and out comes nearly 1. Feed it exactly 0 and out comes exactly 0.5, the perfectly unsure middle.
That middle point clears up something from chapter 1. There, the neuron's question was "is the total bigger than 0?" Here, the decision is "is the output bigger than 0.5?" Those are the same question, because sigmoid turns a total of exactly 0 into an output of exactly 0.5. A total above 0 always lands above 0.5, and a total below 0 always lands below it.
Why sigmoid and not some other shape? Hold that question. Chapter 11 introduces its modern replacement, ReLU, and shows why the choice of "squisher" matters.
A lean, and a vote against
The weights decide how much each input matters. The bias is the neuron's built-in lean: how eager it is to fire before any input arrives. A big positive bias is an optimist. It leans YES and needs the inputs to talk it out of firing. A negative bias is a skeptic. It leans NO and needs the inputs to push it over the line. If every input is 0, the bias is the only thing left, so it alone decides the answer.
A weight can also be negative. A positive weight means "this input is evidence for YES." A negative weight means "this input argues against YES": when that input switches on, it drags the total down, toward NO. Keep that idea in your pocket. It becomes the secret ingredient when you reach the XOR puzzle in chapter 7.
Your turn
The playground below uses two yes/no inputs: piano done? and homework done? A value of 1 means yes and 0 means no. The neuron is pretending to answer "should screen time be allowed?" Chapter 4 teaches it that rule for real. Here you move the parts by hand to see what a neuron does.
Suppose you set both weights and the bias to exactly 0. What will the output be?
With every weight at 0 the inputs are multiplied away, so the total is always 0, and sigmoid turns a total of 0 into exactly 0.5. The neuron has no opinion. Try it in the playground.
Play with a neuron
Drag the sliders and watch the output bar react. It starts with both weights at +1 and the bias at -0.5. Flip the two inputs between 0 and 1 and see which combinations get a YES. Then make one weight negative and switch its input on.
What just happened
With both weights at +1 and the bias at −0.5, switching on either input gives a total of 0.5 and an output of 0.62: YES. With both inputs off, only the bias is left, the total is −0.5 and the output is 0.38: NO. That is the screen-time rule, "piano OR homework," made out of three numbers. Nothing in the neuron says "or." The rule lives entirely in the weights and the bias.
You set those three numbers by hand. In chapter 4 the neuron finds them on its own.
A neuron multiplies each input by a weight, adds a bias, and squishes the total into a number between 0 and 1. Change the weights and the bias, and the same little machine follows a different rule.
Make the neuron behave like AND instead of OR: YES only when piano and homework are both done. Hint: leave the weights alone and think about how skeptical the bias needs to be.
A neuron has a weight of −2 on its "it is raining" input. What happens when that input switches on?
A negative weight is a vote against. Ignoring an input is what a weight of 0 does.
Both inputs are 0. What decides the neuron's output?
Each weight is multiplied by an input of 0, so the weights add nothing. The total is just the bias. A positive bias gives an output above 0.5, a negative bias gives one below it.
What is sigmoid for?
Sigmoid is a smooth curve, so outputs like 0.62 are normal. The yes/no decision comes afterwards, by asking whether the output is above 0.5. How much each input matters is the job of the weights.