Backpropagation › Vectorized Backpropagation

Real implementations work with mini-batches rather than single examples. With a batch of $B$ examples, the input is $\mathbf{X} \in \mathbb{R}^{B \times n}$, and all forward and backward computations generalize naturally.

For a linear layer $\mathbf{Z} = \mathbf{X} \mathbf{W}^T + \mathbf{b}$:

• Forward: $\mathbf{Z} = \mathbf{X} \mathbf{W}^T$ (matrix multiplication, $\mathbf{Z} \in \mathbb{R}^{B \times m}$)
• Backward gradient w.r.t. weights: $\frac{\partial \mathcal{L}}{\partial \mathbf{W}} = \left(\frac{\partial \mathcal{L}}{\partial \mathbf{Z}}\right)^T \mathbf{X}$ (result shape $m \times n$)
• Backward gradient w.r.t. input: $\frac{\partial \mathcal{L}}{\partial \mathbf{X}} = \frac{\partial \mathcal{L}}{\partial \mathbf{Z}} \mathbf{W}$ (result shape $B \times n$)
• Backward gradient w.r.t. bias: $\frac{\partial \mathcal{L}}{\partial \mathbf{b}} = \sum_{i=1}^{B} \frac{\partial \mathcal{L}}{\partial \mathbf{z}_i}$ (sum over batch dimension)

The bias gradient requires summing over the batch dimension because the same bias vector is added to all examples. This is another example of gradient accumulation: the bias participates in $B$ computations (one per example), so its gradient is the sum over all $B$ paths.

Vectorized backpropagation improves performance and naturally averaged (or summed, depending on how the loss is normalized) over the batch. A batch gradient descent step moves the parameters in a direction that reduces the average loss across all examples in the batch. With large batches, this average is a better approximation of the true gradient over the full dataset, leading to more stable optimization. With small batches (stochastic gradient descent), the noisier gradient estimate can help escape sharp local minima, which is why SGD with a small batch size often generalizes better than full-batch gradient descent.