Multi-Layer Neural Networks

The Problem That Nearly Killed Neural Networks

At the end of the previous chapter, we discovered a major limitation.

A perceptron can learn:

• AND
• OR

But cannot learn:

• XOR

The reason is simple.

A perceptron creates only one linear decision boundary.

In two dimensions: w1x1 + w2x2 + b = 0 represents a straight line.

If a problem cannot be separated by a straight line, a single perceptron fails.

This became known as the linear separability problem.

Understanding XOR Geometrically

Consider XOR:

Graphically:

(0, 1)			(1, 1)
  +				   -
  
  
  
  -                +
(0, 0)           (1, 0)

No single line can separate positive and negative examples.

Therefore:

Single Layer Perceptron = Failure

The question became:

What kind of network can solve XOR?

The Hidden Layer Idea

Instead of using one neuron:

Input
  ↓
Neuron
  ↓
Output

Use multiple neurons.

Inputs
  ↓ 
Hidden Layer
  ↓ 
Output Layer

This creates a Multi-Layer Perceptron (MLP).

The hidden layer transforms data into a new representation.

Why Hidden Layer Matter

Imagine classifying houses.

Inputs:

Size
Bedrooms
Age
Location

The network may learn hidden concepts such as:

Luxury Score
Neighborhood Quality
Family Suitability

These concepts were never explicitly programmed.

The network discovers them automatically.

This is called:

Representation Learning

Modern AI is largely about learning useful representations.

The Three Types of Layers

[ 1 ] Input Layer

The input layer does not perform computation.

It simply holds the input values.

Example:

Age = 20

Income = 50000

Experience = 3

Vector:

[ 2 ] Hidden Layer

This is where learning happens.

Each neuron:

1. Receives all inputs.
2. Computes a weighted sum.
3. Adds a bias.
4. Applies an activation function.
5. Sends the result to the next layer.

[ 3 ] Output Layer

Produces the final prediction.

Examples:

• Binary classification -> 1 neuron
• Multi-class classification -> one neuron per class
• Regression -> 1 neuron with a linear output

Matrix Representation

As networks grow, individual equations become impractical.

Instead we use matrices.

Forward Propagation

The process of producing predictions is called propagation.

Why Non-Linearity Is Essential