Backpropagation › Backpropagation Implementation from Scratch
Language AI HandbookAll chapters Backpropagation › Backpropagation Implementation from ScratchOne section · Sections in this chapter12 / 20 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement backpropagation from scratch using only NumPy. We'll build a small framework with Value nodes that record their computational history and can propagate gradients backward.
class Value:
"""A scalar value that supports automatic differentiation."""
def __init__(self, data, _children=(), _op="", label=""):
self.data = float(data)
self.grad = 0.0 # Gradient accumulated here
self._backward = lambda: None # Function to compute local gradient
self._prev = set(_children) # Input nodes
self._op = _op
self.label = label
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), "+")
def _backward():
# Gradient flows through addition unchanged
self.grad += out.grad
other.grad += out.grad
out._backward = _backward
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), "*")
def _backward():
# Product rule: d/dx(x*y) = y, d/dy(x*y) = x
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = _backward
return out
def relu(self):
out = Value(max(0, self.data), (self,), "relu")
def _backward():
# ReLU gradient: 1 if input > 0, else 0
self.grad += (self.data > 0) * out.grad
out._backward = _backward
return out
def backward(self):
"""Compute gradients via reverse-mode autodiff (backpropagation)."""
# Topological sort: process nodes in reverse order
topo = []
visited = set()
def build_topo(v):
if v not in visited:
visited.add(v)
for child in v._prev:
build_topo(child)
topo.append(v)
build_topo(self)
# Gradient of root with respect to itself is 1
self.grad = 1.0
# Propagate gradients backward
for node in reversed(topo):
node._backward()
def __repr__(self):
return f"Value(data={self.data:.4f}, grad={self.grad:.4f})"
def __radd__(self, other):
return self + other
def __rmul__(self, other):
return self * other
def __neg__(self):
return self * -1
def __sub__(self, other):
return self + (-other)
def __pow__(self, n):
out = Value(self.data**n, (self,), f"**{n}")
def _backward():
self.grad += (n * self.data ** (n - 1)) * out.grad
out._backward = _backward
return outThe topological sort is worth examining closely. The build_topo function performs a depth-first traversal of the computation graph, adding each node to the topo list only after all nodes it depends on have been added. When we then iterate reversed(topo), we process nodes from output to input, which is exactly the order the backward pass needs. This mirrors how backpropagation conceptually works: we start at the loss and propagate gradients backward, and each node can only compute its backward function once all downstream gradients have been accumulated.
Forward pass: a = x + y = 5.0 f = a * z = -20.0 Backward pass (gradients): df/dx = -4.0 (should be z = -4.0) df/dy = -4.0 (should be z = -4.0) df/dz = 5.0 (should be a = 5.0)
The gradients match the analytical result: , and .
Now let's build a simple neural network layer and verify backpropagation computes the same gradients as PyTorch:
class LinearLayer:
"""A fully-connected layer with backpropagation support."""
def __init__(self, n_in, n_out, rng=None):
if rng is None:
rng = np.random.default_rng(42)
# Xavier initialization
scale = np.sqrt(2.0 / (n_in + n_out))
self.W = rng.normal(0, scale, (n_out, n_in))
self.b = np.zeros(n_out)
self.dW = np.zeros_like(self.W)
self.db = np.zeros_like(self.b)
self._cache = {}
def forward(self, x):
"""Forward pass: z = W @ x + b"""
self._cache["x"] = x
return self.W @ x + self.b
def backward(self, dz):
"""Backward pass: compute gradients given upstream gradient dz."""
x = self._cache["x"]
# Gradient for weights: outer product of upstream grad and input
self.dW = np.outer(dz, x)
# Gradient for bias: same as upstream gradient
self.db = dz
# Gradient for input: W^T @ upstream_grad
dx = self.W.T @ dz
return dx
class SigmoidLayer:
"""Sigmoid activation with backward pass."""
def __init__(self):
self._cache = {}
def forward(self, z):
a = 1 / (1 + np.exp(-z))
self._cache["a"] = a
return a
def backward(self, da):
a = self._cache["a"]
# sigmoid'(z) = a * (1 - a)
return da * a * (1 - a)
class TwoLayerNet:
"""Two-layer MLP with sigmoid activation."""
def __init__(self, n_in, n_hidden, n_out):
rng = np.random.default_rng(42)
self.layer1 = LinearLayer(n_in, n_hidden, rng)
self.sigmoid = SigmoidLayer()
self.layer2 = LinearLayer(n_hidden, n_out, rng)
def forward(self, x, y):
"""Forward pass, returns loss."""
z1 = self.layer1.forward(x)
a1 = self.sigmoid.forward(z1)
z2 = self.layer2.forward(a1)
# Softmax + cross-entropy
logits = z2 - z2.max()
probs = np.exp(logits) / np.exp(logits).sum()
loss = -np.log(probs[y] + 1e-10)
self._cache = {"a1": a1, "probs": probs, "y": y}
return loss, probs
def backward(self):
"""Backward pass: compute all gradients."""
probs = self._cache["probs"]
y = self._cache["y"]
# Gradient of softmax+cross-entropy: pred - target
dz2 = probs.copy()
dz2[y] -= 1.0
# Backward through layer 2
da1 = self.layer2.backward(dz2)
# Backward through sigmoid
dz1 = self.sigmoid.backward(da1)
# Backward through layer 1
self.layer1.backward(dz1)Training epochs: 200 Final training loss: 0.0115 Training accuracy: 0.995 Loss reduction: 0.2869 -> 0.0115 (96.0% decrease)
The network learned the task successfully using pure NumPy backpropagation. The loss decreased substantially and accuracy is high on the linearly separable data.
