Backpropagation › Numerical Gradient Checking

How do we verify that our backpropagation implementation is correct? The standard technique is numerical gradient checking, which approximates gradients using finite differences and compares them to the analytical gradients from backprop.

For a small $\epsilon > 0$, the centered finite difference approximation of $\frac{\partial \mathcal{L}}{\partial w}$ is:

$$\frac{\partial \mathcal{L}}{\partial w} \approx \frac{\mathcal{L}(w + \epsilon) - \mathcal{L}(w - \epsilon)}{2\epsilon}$$

where:

• $\mathcal{L}(w + \epsilon)$: the loss when we perturb weight $w$ by $+\epsilon$
• $\mathcal{L}(w - \epsilon)$: the loss when we perturb weight $w$ by $-\epsilon$
• $2\epsilon$: the denominator giving the centered difference (more accurate than one-sided)

The centered difference approximates the true derivative with $O(\epsilon^2)$ error, compared to $O(\epsilon)$ for the one-sided difference. Using $\epsilon \approx 10^{-5}$ typically gives sufficient accuracy. Choosing $\epsilon$ too small causes numerical catastrophe: floating-point subtraction of nearly equal numbers loses precision, making the approximation noisy. Choosing $\epsilon$ too large increases the approximation error from the Taylor series truncation. The sweet spot is around $10^{-5}$ for float64 arithmetic.

To check whether the analytical and numerical gradients agree, compute the relative error:

$$\text{relative error} = \frac{|\mathbf{g}_{\text{analytical}} - \mathbf{g}_{\text{numerical}}|}{\max(|\mathbf{g}_{\text{analytical}}|, |\mathbf{g}_{\text{numerical}}|) + 10^{-8}}$$

where:

• $\mathbf{g}_{\text{analytical}}$: the gradient computed by backpropagation
• $\mathbf{g}_{\text{numerical}}$: the gradient approximated by finite differences
• The $10^{-8}$ in the denominator prevents division by zero for near-zero gradients

Guidelines for acceptable relative error:

• Below $10^{-5}$: excellent, backprop is almost certainly correct
• Between $10^{-5}$ and $10^{-3}$: possibly suspicious, check further
• Above $10^{-2}$: likely a bug in backpropagation

All relative errors are well below $10^{-4}$, confirming our backpropagation implementation is correct.

Gradient checking is slow: it requires two forward passes per parameter, making it $O(P)$ in the number of parameters. Use it for debugging, not production training. The standard workflow is:

1. Run gradient check on a small version of your model with a handful of inputs
2. Once the check passes, disable it and train normally
3. If training behaves unexpectedly, re-enable the check

It is important to do gradient checking at the beginning of a project, before you have spent significant time training. Finding a gradient bug after 10 hours of training is much more costly than finding it after 10 minutes. Experienced practitioners routinely run gradient checks as a sanity step before launching any serious training run.

There are a few situations where gradient checking gives misleading results. First, if your model uses random dropout during training, you need to disable dropout before running the check, since the numerical approximation and the analytical gradient will use different dropout masks. Second, if your loss function has flat regions (like a hinge loss away from the margin), the numerical gradient might be zero while the analytical gradient is nonzero, giving large relative error that is not actually a bug. Third, very small gradients can have large relative errors due to the denominator normalization, even when the absolute difference is tiny.