Neuro 13
0 / 37 chapters finished

Chapter 11: What does that "squish" actually do?

Sigmoid, ReLU, and why the bend matters.

Every neuron so far has ended the same way: multiply, add, then squish with sigmoid. The squish step has a proper name, the activation function, and it is doing two jobs you have not seen yet. This chapter looks at both, and then meets the activation that replaced sigmoid inside modern networks: ReLU.

Job one: the bend

Imagine deleting the squish. Every neuron just multiplies and adds. Now stack 100 layers of those neurons. What can that network learn?

Only what one layer could. Multiply-and-add of a multiply-and-add is still just a multiply-and-add with different numbers, so the whole stack collapses into one layer. One layer means straight-line rules only, and XOR stays impossible.

That is the first job of the activation: it puts a bend between the layers, so each layer hands on something that multiplying and adding alone could not have produced. It is what let the hidden neurons in chapter 8 fence off XOR's corners.

Job two: the slope

Here is what an activation looks like on graph paper. The horizontal axis is what goes into the neuron (the weighted sum). The vertical axis is what comes out.

Before you drag anything: where is the sigmoid curve steepest?

In the middle. Out at both ends the curve has flattened against 0 or 1 and barely rises at all. Drag the dot and watch the slope number to check.

▸ try it

Drag the dot. Watch the curve.

Slide the input left and right. The red dot is where the neuron sits right now. The orange dashed line shows how steep the curve is at that exact point. That steepness is called the derivative, which is just the math word for slope.

sigmoid
ReLU
0.00
output (y): 0.50 slope (derivative): 0.25

Why care about the slope? Because training uses it. When blame for a mistake is passed backward through a neuron, it gets multiplied by the slope at the spot where that neuron is sitting.

  • Steep slope: the blame gets through, and the neuron's weights move.
  • Flat slope (close to 0): the blame is multiplied by almost nothing, and the weights barely move.

Now read sigmoid's slope off the widget:

  • At an input of 0 the slope is 0.25, the biggest it ever gets. The neuron is at its most unsure here, and learns fastest.
  • At +5 or −5 the slope is 0.007. The neuron has made up its mind, and learning has almost stopped. A neuron stuck out here is called saturated.

Why sigmoid struggles in deep networks

Sigmoid's slope is 0.25 at best, and usually much less. In a deep network the blame has to travel back through every layer, and each sigmoid layer multiplies it by a slope of at most 0.25. (The weights get multiplied in too, but the slopes are the part that always shrinks.) Even in the best case:

10 layers:  0.25 × 0.25 × ... (10 times)  ≈  0.000001          (a millionth)
20 layers:  0.25 × 0.25 × ... (20 times)  ≈  0.000000000001    (a trillionth)

By the time the signal reaches the early layers there is almost nothing left of it, so those layers stop learning. This is the famous vanishing gradient problem, and it is a big part of why deep networks were so hard to train for years after backpropagation was known. (Chapter 14 shows exactly where that multiplying happens.)

ReLU: the fix that looks too simple

Toggle "ReLU" in the widget. ReLU stands for rectified linear unit, and its whole rule is: if the input is negative, output 0; otherwise output the input unchanged. The curve is flat on the left, then a straight ramp. It still has a bend (at 0), so job one is covered. And its slope is either exactly 1 or exactly 0.

That slope of 1 is the point. Multiply by 1 as many times as you like and nothing shrinks, so the blame survives the trip back through many layers.

It has one weakness of its own. A neuron whose input is negative for every example outputs 0 every time, with slope 0, so no blame ever reaches its weights. It is a dead neuron and it stays dead. Relatives of ReLU such as LeakyReLU, GELU and SiLU soften the bend to make that rarer (Wikipedia's activation function page shows them all). Sigmoid still has a job at the output, where you want a probability. It is in the hidden layers that ReLU took over.

▸ run real python

Run the ReLU version

This is neuro5.py: the snack picker from chapter 10 with ReLU in the hidden layer. Only two lines changed, one in the forward pass and one in the backward pass. Look for the <- arrows in the code. The script also uses a smaller learning rate than before, 0.1 instead of 0.5.

neuro5.py Python loads on first run
(click "run" to execute)

The activation's bend is what makes extra layers worth having, and its slope decides how much of the error signal survives the trip backward. Sigmoid's flat ends choke that signal; ReLU's slope of 1 lets it through.

Challenge

In the ReLU script, lr is the learning rate: the "small step" from chapter 4's nudge recipe, which chapter 13 digs into. Change lr = 0.1 to 0.5 and run it. It still works. Now try lr = 5. Every row comes out as 0.25, 0.25, 0.50, whatever the inputs. The first giant steps pushed all six hidden neurons into dead territory, and a network with no working hidden neurons can only report how common each snack is. Count the cookies in the table and check.

Why does a network need an activation function between its layers?

Multiply-and-add stacked on multiply-and-add is still one multiply-and-add. The bend is what lets each new layer do something the previous ones could not.

A sigmoid neuron's weighted sum is +8 on every training example. What happens to that neuron's weights during training?

At +8 the sigmoid's slope is about 0.0003. Blame passing back through the neuron is multiplied by that, so almost nothing reaches its weights. It is saturated.

What is the main reason ReLU made very deep networks easier to train than sigmoid?

Sigmoid multiplies the signal by 0.25 or less at each layer; ReLU multiplies it by 1 wherever the neuron is active. (And ReLU does have a bend, at 0. Without one it would be useless.)