Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multilayer Perceptrons › From Single Neurons to Hidden Layers

Michael BrenndoerferApril 29, 20256 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 63Multilayer Perceptrons › From Single Neurons to Hidden Layers

Reading section 2 of 13 · From Single Neurons to Hidden Layers · 6 minSection 2 of 13 · 6 min

Section 2 of 13 · 6 min read

Multilayer Perceptrons › From Single Neurons to Hidden LayersOne section · Sections in this chapter2 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A single neuron computes a weighted sum of its inputs, adds a bias, and passes the result through an activation function. Given an input vector $\mathbf{x} = [x_1, x_2, \ldots, x_n]$ with $n$ features, the neuron computes:

y = \sigma(\mathbf{w}^\top \mathbf{x} + b)

where:

$\mathbf{x}$ : the input vector containing $n$ features
$\mathbf{w}$ : the weight vector, where each $w_i$ controls how much input $x_i$ influences the output
$\mathbf{w}^\top \mathbf{x}$ : the dot product $\sum_{i=1}^{n} w_i x_i$ , computing a weighted sum of inputs
$b$ : the bias term, which shifts the decision boundary
$\sigma$ : the activation function (e.g., ReLU, sigmoid), which introduces non-linearity
$y$ : the scalar output of the neuron

This single neuron can learn a linear decision boundary (made non-linear by $\sigma$ ), but it cannot solve problems requiring more complex boundaries.

Hidden Layer

A hidden layer is a collection of neurons that sits between the input and output of a neural network. Each neuron in a hidden layer receives the full input (or the output of the previous layer), applies its own weights and bias, and produces one scalar output. The term "hidden" reflects that these intermediate computations are not directly observed, only the final output layer is.

The XOR ProblemLink Copied

Consider the classic XOR problem: given two binary inputs, output 1 if exactly one input is 1, and 0 otherwise. No single linear boundary can separate the positive from negative examples. But with a hidden layer, we can first transform the inputs into a new representation where the classes become linearly separable.

The XOR truth table is:

XOR truth table. No straight line through the 2D input space can separate the two classes (output 0 vs output 1).
$x_1$	$x_2$	XOR output
0	0	0
0	1	1
1	0	1
1	1	0

No matter how you try to draw a straight line through the four points, some misclassification will remain. XOR requires a non-linear boundary, and that is exactly what a hidden layer provides.

In[3]:

Code

import numpy as np

# XOR inputs and outputs
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 0])

Out[4]:

Visualization

Scatter plot of four points in 2D space with XOR labels showing non-linear separability. — The XOR problem in 2D input space. Blue circles represent class 0 and red crosses represent class 1. No single straight line can separate the two classes, which demonstrates the fundamental limitation of linear classifiers on this task.

The magic happens when we add a hidden layer. Each hidden neuron learns to detect a different feature or pattern in the input. The output layer then combines these learned features to make the final prediction.

To see this in action, let's manually construct a hidden layer that solves XOR and visualize how it transforms the input space:

Out[5]:

Visualization

Scatter plot of four XOR points in the original 2D input space. — Original input space where XOR classes cannot be separated by a single straight line. Blue circles are class 0, red crosses are class 1.

Scatter plot of XOR points after hidden layer transformation, now linearly separable. — After passing through the hidden layer, points are remapped to a new space where they become linearly separable, with a vertical boundary at approximately 0.4.

The hidden layer has transformed the input space so that a simple linear classifier can now separate the classes. The first hidden neuron activates only when both inputs are high (AND-like behavior), while the second activates when either input is high (OR-like behavior). In this new coordinate system, the XOR pattern becomes trivially separable.