Backpropagation › Key Parameters

The key implementation choices when working with backpropagation are:

• Learning rate ($\eta$): Controls how large each gradient step is. Too large causes divergence; too small causes slow convergence. We'll cover learning rate selection in detail in the Stochastic Gradient Descent chapter.
• Numerical precision: Float32 is the standard for training (good speed/memory tradeoff); float64 is used for gradient checking (higher precision). Mixed precision training (float16 for forward, float32 for gradients) is common in large model training.
• Gradient clipping threshold: Prevents exploding gradients in deep or recurrent networks by capping the gradient norm. Covered in the Gradient Clipping chapter.
• Gradient checkpoint interval: The memory vs. compute tradeoff for very deep networks. Checkpointing at every $k$ layers reduces memory by a factor of $k$ at the cost of recomputing $k-1$ forward passes.