Differential Privacy › DP for LLMs

Training large language models with differential privacy presents unique challenges and opportunities. LLMs have billions of parameters, are typically trained on massive public corpora where privacy concerns may seem lower, but the fine-tuning stage often involves sensitive private data: medical records, financial documents, personal communications, legal files. The combination of a powerful pre-trained model and a private fine-tuning dataset is precisely where DP provides the most value.

The most practical application of DP to LLMs is differentially private fine-tuning: start from a publicly pre-trained model and fine-tune on a private dataset with DP guarantees. Because the pre-trained weights already encode rich language understanding, the fine-tuning process only needs to make small adjustments, and the DP noise budget is spent more efficiently.

Research on DP fine-tuning of LLMs has shown that large pre-trained models can be fine-tuned with meaningful DP guarantees ($\varepsilon \approx 3$) with only modest accuracy degradation on many NLP benchmarks. Li et al. (2022) demonstrated that GPT-2-XL fine-tuned with DP on the E2E dataset produced text comparable in quality to non-DP fine-tuned smaller models. The key finding is consistent across studies: model scale is the most important factor for DP utility. A large pre-trained model fine-tuned with DP often performs comparably to a small model without DP on the same task.

This has an important practical implication: if you have a fixed privacy budget and need to choose between a small model trained without DP or a large model trained with DP, the large model with DP is often the better choice from both privacy and utility perspectives.

A particularly promising approach combines DP with parameter-efficient fine-tuning methods like LoRA (Low-Rank Adaptation), which we covered in the Parameter-Efficient Fine-Tuning chapter. Since LoRA only trains a small number of parameters (the low-rank adapter matrices), the per-step gradient is defined over a much smaller parameter space. This has two important benefits for DP.

First, the noise dimensionality is reduced. Instead of adding noise to a billion-dimensional gradient vector, you add noise to a much smaller vector of adapter parameters. The signal-to-noise ratio improves dramatically, because the same amount of gradient signal is now competing with noise in a much lower-dimensional space.

Second, the effective clipping is applied to a smaller gradient, which has less variability across examples, reducing clipping bias. The adapter gradients are more uniform across training examples than full-model gradients, because they capture only high-level task adaptation rather than fine-grained language patterns.

The combination of DP and LoRA, sometimes called DP-LoRA, has emerged as the practical default for DP LLM fine-tuning. Research has shown that DP-LoRA can match or exceed the accuracy of full DP fine-tuning while consuming less effective privacy budget per step, because the reduced dimensionality and lower gradient variability combine to provide more useful updates per unit of noise.

Choosing the rank hyperparameter for DP-LoRA involves a tradeoff. Lower rank means fewer parameters to update (better for DP, since less noise is spread across the gradient) but also less expressive adaptation (worse for utility). Empirically, ranks in the range of 4-16 work well for DP fine-tuning, which is lower than the typical 32-64 used without DP. The intuition is that DP noise already regularizes the fine-tuning heavily, so lower-rank adapters are less likely to overfit.

Theoretical DP guarantees are worst-case bounds. In practice, the actual privacy leakage may be lower than $\varepsilon$ suggests, because the worst-case scenario may not arise in realistic datasets. Privacy auditing provides empirical lower bounds on privacy leakage by attempting membership inference attacks and measuring their success rate. If the attack succeeds better than the DP guarantee allows, there is a bug. If the attack does no better than the theoretical bound predicts, you have evidence that the implementation is correct.

A common auditing technique uses canary examples: training examples that are deliberately inserted into the dataset and then tested to see if they can be detected. Canaries are often crafted to be highly distinctive (unusual sequences, rare patterns, synthetic data points) so they are maximally easy to memorize. If the attacker can detect canaries with high confidence, the privacy guarantee is tight or may be violated. If not, the mechanism provides at least as much privacy as the theoretical $\varepsilon$ suggests.

Auditing with canaries works as a shadow training exercise. The auditor selects a set of canary examples, some of which are added to the training data and some are held out. After training, the auditor runs a membership inference test: which examples can be distinguished as members versus non-members based on the model's behavior? A common test uses the model's loss on the example as a signal. The auditor measures the advantage of the membership inference attack over random guessing. If the advantage is bounded by $e^\varepsilon - 1$ (the theoretical maximum advantage under $\varepsilon$-DP), the empirical evidence is consistent with the theoretical guarantee. If the advantage exceeds this bound, there is a violation.

Empirical auditing is essential in practice because DP implementations can harbor bugs that invalidate the theoretical guarantee while the code appears to run correctly. Common implementation failures include:

• Incorrect subsampling: Using fixed, deterministic batches instead of Poisson subsampling invalidates the subsampling amplification used in the privacy accounting.
• Missing noise in some components: If certain gradient components (e.g., embeddings for rare tokens) are updated without adding noise, those components can memorize sensitive data.
• Numerical precision issues: Floating-point rounding in the clipping computation can introduce small but non-zero sensitivities beyond $C$.
• Gradient accumulation bugs: When using gradient accumulation over multiple micro-batches, the noise must be added once to the aggregated gradient, not per micro-batch.

Auditing catches these implementation failures in a way that code review alone cannot. It is good practice to run a privacy audit on any new DP implementation before deploying it with real sensitive data.

Tools like the autodp library and Opacus (from Meta) provide both theoretical accounting and empirical auditing capabilities. Opacus, in particular, is the most widely used library for DP training of PyTorch models and handles the per-example gradient computation efficiently.