Backpropagation › Computational Graphs

Before diving into backpropagation, we need a way to visualize how computations flow through a network. A computational graph represents a mathematical expression as a directed graph where nodes are operations and edges show data flow.

Every neural network computation breaks down into elementary operations: addition, multiplication, matrix products, and activation functions. Consider the simple expression $f(x, y, z) = (x + y) \cdot z$. This decomposes into two operations:

1. $a = x + y$ (addition)
2. $f = a \cdot z$ (multiplication)

where:

• $x, y, z$: input variables (leaf nodes in the graph)
• $a$: an intermediate value storing the result of the first operation
• $f$: the final output (the root node)

The graph shows data flowing left to right through intermediate nodes. Each node computes something simple, and the structure of the graph encodes the data dependencies. Each node only needs to know about its immediate inputs and output, not the rest of the network. This locality is what makes the chain rule tractable.

A neural network layer performs the computation $\mathbf{a} = \sigma(\mathbf{W}\mathbf{x} + \mathbf{b})$, where:

• $\mathbf{x}$: the input vector
• $\mathbf{W}$: the weight matrix
• $\mathbf{b}$: the bias vector
• $\sigma$: the activation function
• $\mathbf{a}$: the output activations

This breaks into three atomic operations:

1. $\mathbf{z}_1 = \mathbf{W}\mathbf{x}$ (matrix multiplication)
2. $\mathbf{z}_2 = \mathbf{z}_1 + \mathbf{b}$ (bias addition)
3. $\mathbf{a} = \sigma(\mathbf{z}_2)$ (nonlinear activation)

Each node has a well-defined local gradient. The power of computational graphs is that they make the structure of computation explicit, which is exactly what backpropagation needs. The forward pass computes values flowing left to right; the backward pass computes gradients flowing right to left.

When we compose multiple layers, we are simply building a larger graph. The loss node at the end is connected to the output layer, which is connected to the layer before it, and so on back to the input. The entire network is one big directed acyclic graph of operations, with parameters as leaf nodes and the loss as the root.

Computational graphs provide two benefits for gradient computation:

• Modularity: Each node has a well-defined local gradient. We don't need to derive the gradient of the entire network at once, only the gradient of each primitive operation. This is the key insight that makes backprop practical: instead of deriving one giant formula for the gradient of the loss with respect to every parameter, we simply need to know the derivative of each individual operation, and the chain rule handles the composition.
• Efficiency: By storing intermediate values during the forward pass, we reuse them during the backward pass, avoiding redundant computation. Without this storage, we would have to recompute every intermediate value every time we needed it, which would make the backward pass much more expensive.

Deep learning frameworks like PyTorch and TensorFlow build exactly this kind of dynamic computational graph as code runs. Every arithmetic operation on a tensor implicitly creates a node in the graph, recording what inputs were used so the backward pass can reconstruct the gradient computation. You never see this graph directly, but it is being built silently behind every +, @, and activation function call you write.