Differential Privacy › Key Parameters and Their Interactions

Understanding how DP-SGD hyperparameters interact is essential for getting good results. The key parameters are:

• max_grad_norm (C): The per-example gradient clipping threshold. Smaller values mean more clipping (more bias) but less noise needed for a given epsilon. Typical range: 0.1 to 2.0. Values near the median gradient norm are generally best.
• noise_multiplier (sigma): Scales the Gaussian noise added to gradients. Higher values provide stronger privacy but hurt utility. Determined automatically by a privacy accountant when targeting a specific epsilon.
• target_epsilon: The privacy budget. Smaller values give stronger privacy. Practical range for LLMs: 1 to 8. Below 1, utility typically collapses for all but the largest models.
• target_delta: Failure probability. Should be much smaller than $1/n$ (dataset size). Typical: $10^{-5}$ to $10^{-6}$.
• batch_size: Larger batches improve the signal-to-noise ratio and are strongly preferred for DP training. Batch sizes of 256 to 4,096 are common for DP LLM fine-tuning.
• epochs: More training epochs consume more privacy budget. Budget-aware early stopping is essential.

These parameters do not act independently. Increasing the batch size while holding everything else constant reduces the subsampling rate $q$, which reduces the privacy cost per step, allowing more training steps within the same budget. This means you can train for more epochs or use a lower $\sigma$ to get better accuracy. For a fixed $\varepsilon$ target, the optimal strategy often involves maximizing batch size (to improve SNR) while setting $\sigma$ just high enough to stay within budget over the desired number of epochs.