Backpropagation › Limitations and Impact

Backpropagation has some practical limitations that shape how neural networks are designed and trained.

The vanishing gradient problem remains the most persistent challenge. When gradients are multiplied across many sigmoid or tanh layers, they can shrink to numerically zero before reaching the earliest layers. The introduction of ReLU activations, residual connections (which add skip paths that provide direct gradient highways), batch normalization, and careful weight initialization all exist primarily to address this problem. As we'll see in subsequent chapters, architectural choices like residual connections in ResNets and layer normalization in Transformers are fundamentally about keeping gradient flow healthy.

Backpropagation requires storing all intermediate activations for the backward pass. For large models and large batches, this memory requirement can exceed GPU capacity. Gradient checkpointing (also called activation recomputation) trades computation for memory by not storing all intermediate values and recomputing them during the backward pass. Modern large language models routinely use gradient checkpointing during training. The memory savings can be substantial: for a 100-layer transformer with long sequences, checkpointing can reduce activation memory by 10-15x at the cost of roughly 30-40% additional computation time.

Backpropagation computes exact gradients with respect to the loss, which is both a strength and a limitation. The gradients are correct for the exact computation performed, but they contain no information about the loss landscape beyond the current point. Second-order methods like Newton's method use the Hessian (matrix of second derivatives) to better inform the update direction, but computing and storing the full Hessian for even moderately sized networks is infeasible. Methods like K-FAC and Shampoo approximate the Hessian in structured ways to get some second-order benefits without full Hessian computation, but they remain significantly more expensive than first-order methods like Adam.

Non-differentiable operations break backpropagation's clean mathematical story. Operations like argmax, sorting, or branching on data values have zero gradient almost everywhere. Modern deep learning works around this through soft approximations (softmax instead of argmax), straight-through estimators (for binary operations), and reinforcement learning signals (for discrete decisions). The straight-through estimator is particularly elegant: during the forward pass, you apply a non-differentiable operation; during the backward pass, you simply pretend the operation was the identity function and pass gradients through unchanged. This is mathematically unjustified but empirically works well for many applications including training binary neural networks and quantization-aware training.

Biological plausibility is another limitation, though it matters more for neuroscience than for engineering. Backpropagation requires exact knowledge of the forward weights to compute the backward pass (the $\mathbf{W}^T$ in the gradient formula). Neurons in biological brains do not have access to the exact synaptic weights of the neurons they synapse onto, making backpropagation unlikely to be the mechanism by which biological networks learn. Alternative learning rules like forward-forward, target propagation, and perturbation-based methods are active research areas trying to achieve the efficiency of backpropagation while respecting biological constraints. For practical deep learning, though, biological plausibility is not a concern.

Despite these limitations, backpropagation fundamentally changed what was computationally possible. Before efficient gradient computation, training more than two or three layer networks was practically infeasible. The algorithm enabled CNNs for image recognition, RNNs for sequence modeling, Transformers for language AI, and the billion-parameter models that power today's AI systems. In the next chapter, we'll use these gradients with stochastic gradient descent and its variants to train networks efficiently.