Neuro 13
0 / 37 chapters finished

Chapter 6: Tables, dots, and lines

Why a neuron's decision is always a straight line.

Chapter 7 is about a puzzle called XOR, and it is where lots of people first get confused. The cure is to slow down and watch a table of answers turn into dots on graph paper.

With two yes/no inputs there are only four possible input pairs, and each pair is a dot. A rule is nothing more than a YES or NO label on each of the four dots. A table that lists every possible input with its answer is called a truth table. Here are the truth tables for three rules at once:

input 1input 2dotORANDXOR
00(0, 0)NONONO
10(1, 0)YESNOYES
01(0, 1)YESNOYES
11(1, 1)YESYESNO

OR says YES when either input is on. AND says YES only when both are on. XOR (short for "exclusive or") says YES when exactly one is on, and NO when both are.

For one of these three rules, no single straight line can put all the YES dots on one side and all the NO dots on the other. Which one? Sketch the four dots on paper if it helps.

XOR's two YES dots sit on opposite corners, with the NO dots on the other two. Any line that keeps both YES corners on one side drags a NO corner along with them. The pictures below show all three.

01 01 OR NO YES YES YES one line can split it AND NO NO NO YES one line can split it XOR NO YES YES NO no single line can split it

In each picture, input 1 runs left to right and input 2 runs bottom to top, so the bottom-left dot is (0, 0).

Why it has to be a line

A single neuron always does the same thing: multiply, add, and ask whether the total is above 0.

weight1 × input1 + weight2 × input2 + bias > 0

On graph paper, the points where that total is exactly 0 form a straight line. (If you have met y = mx + b at school, this is the same thing rearranged.) On one side of the line the total is positive and the neuron says YES. On the other side it is negative and the neuron says NO. That line is the decision boundary you dragged around in chapter 4.

Training can slide the line, tilt it, or flip which side means YES. It cannot bend it, and it cannot draw a second one. So a single neuron can only ever ask one question: "is this dot on the YES side of my line?"

That is enough for OR and for AND. XOR needs a different kind of question: "is this dot in the top-left corner or the bottom-right corner?" One straight line cannot ask about two separate corners.

There is a name for the rules a line can handle. A rule is linearly separable when one straight line can split its YES examples from its NO examples. OR is linearly separable. AND is linearly separable. XOR is not.

One neuron draws one straight line. It can learn a rule only if a single straight line can separate that rule's YES dots from its NO dots.

Challenge

Cover the labels on the three mini-graphs. Which one must be XOR? Hint: look for the two YES dots sitting on opposite corners.

Why is a single neuron's decision boundary always straight?

The straightness comes from the neuron's arithmetic: multiplying by fixed weights and adding can only make a straight edge. More dots or more training do not change that.

Training changes a neuron's two weights and its bias. What does that do to its line?

Three numbers can describe any straight line on the page, and nothing else. A bend or a second line needs more neurons.

Which of these rules is not linearly separable?

"Different from each other" is XOR in other words: the YES dots are (1, 0) and (0, 1), on opposite corners. The first rule is AND, and the second is split by a straight up-and-down line between input 1 = 0 and input 1 = 1.