Memorization and Privacy in Language Models › Mitigations

No single technique eliminates memorization entirely. Effective privacy protection requires a layered approach combining data preprocessing, training modifications, and deployment safeguards. Each layer addresses different aspects of the risk and has its own cost-benefit profile. Understanding each layer clearly helps practitioners make informed decisions about which combinations are appropriate for their deployment context.

The single most effective preprocessing step is deduplicating the training corpus. Since repetition is the primary driver of memorization, removing or downsampling duplicates dramatically reduces the memorization rate for affected sequences. Lee and colleagues demonstrated that deduplicating the C4 and GitHub training datasets reduced memorization rates by a factor of 10 or more for sequences that had been repeated many times. This is arguably the most effective privacy intervention available at the data preparation stage.

Deduplication can be exact (remove sequences that appear byte-for-byte more than $n$ times) or approximate (use MinHash or SimHash to identify near-duplicate documents and remove them). Both approaches are straightforward engineering tasks that pay large privacy dividends.

MinHash-based deduplication works by computing a sketch of each document as a set of token $k$-grams, then using locality-sensitive hashing to find documents whose sketches overlap substantially. Given a document $d$ with token $k$-grams $G(d)$, the Jaccard similarity between two documents $d_1$ and $d_2$ is:

$$J(d_1, d_2) = \frac{|G(d_1) \cap G(d_2)|}{|G(d_1) \cup G(d_2)|}$$

where:

• $G(d)$ is the set of $k$-gram token sequences in document $d$
• $|G(d_1) \cap G(d_2)|$ is the number of $k$-grams that both documents share
• $|G(d_1) \cup G(d_2)|$ is the total number of distinct $k$-grams across both documents

Two documents with a Jaccard similarity above a threshold (typically 0.8) are considered near-duplicates and the lower-priority copy is removed. This scales to trillion-token corpora because comparisons use compact MinHash sketches rather than the raw documents. Each sketch is a vector of $h$ hash values derived from the $k$-gram set, and similarity can be estimated from these vectors without comparing the full documents.

The one limitation of deduplication is that it cannot help with information that appears only once in the training data but is still memorized, such as the UUID example from Carlini et al. For single-occurrence memorization, other techniques are needed. Deduplication is a strong mitigation for the most easily-exploited class of memorized content, but it does not eliminate the long tail of weakly-memorized, single-occurrence data.

Automated PII detection pipelines can identify and redact personally identifiable information before training. Named entity recognition, regular expressions for known PII patterns (email addresses, phone numbers, Social Security numbers), and machine learning classifiers trained on PII examples can all contribute to reducing PII density in training data.

PII scrubbing is imperfect by design. Entity recognition misses novel formats and uncommon names. Redaction can be circumvented by indirect references. And the definition of PII is context-dependent in ways that automatic classifiers struggle with. Nevertheless, PII scrubbing reduces the quantity of sensitive information available for memorization and is considered a best practice for responsible training data curation. The goal is not perfect recall (which is impossible) but a substantial reduction in the density of readily-extractable PII.

A practical PII scrubbing pipeline typically operates in layers. A rules-based layer first handles high-confidence patterns: email addresses matching RFC 5321 syntax, US phone numbers in standard formats, Social Security numbers matching the standard 9-digit format, and credit card numbers passing the Luhn checksum. This layer has very low false positive rates and handles the majority of structurally distinctive PII.

A second layer applies a sequence labeling model, typically a fine-tuned BERT or similar model, to identify names, addresses, and other entities that require contextual understanding. This layer catches PII that does not follow a fixed pattern, such as a person's name appearing in the middle of a sentence. The tradeoff is higher false positive and false negative rates compared to the rules-based layer.

Finally, a post-hoc review step samples a small fraction of scrubbed documents to estimate false negative rates. The output replaces detected PII with typed placeholders like [EMAIL] or [PHONE] rather than removing the surrounding text entirely. This preserves the sentence structure needed for language learning. A model trained on text with [EMAIL] placeholders learns that email addresses exist in certain contexts, which is useful general knowledge, without learning specific email addresses, which is the privacy risk.

Differential privacy (DP) provides a formal mathematical guarantee that the model's outputs reveal limited information about any individual training example. The DP-SGD algorithm, developed by Abadi and colleagues, adds calibrated Gaussian noise to gradients during training, making it provably harder for an attacker to determine whether any specific example was in the training set.

The formal guarantee is stated in terms of $(\epsilon, \delta)$-differential privacy. A training algorithm $\mathcal{M}$ satisfies $(\epsilon, \delta)$-DP if for any two training datasets $D$ and $D'$ that differ by a single example, and for any set of possible outputs $S$:

$$\Pr[\mathcal{M}(D) \in S] \leq e^{\epsilon} \cdot \Pr[\mathcal{M}(D') \in S] + \delta$$

where:

• $\epsilon$ is the privacy budget: smaller values mean stronger privacy guarantees, with $\epsilon = 0$ meaning the outputs are statistically identical whether or not any single example was included
• $\delta$ is a small probability of the guarantee failing, typically set to $10^{-5}$ or smaller
• $D$ and $D'$ are neighboring datasets differing by exactly one training example
• $\mathcal{M}$ is the training algorithm (including randomness in mini-batch selection, gradient computation, and noise addition)

The privacy guarantee comes at a cost: DP-SGD typically reduces model quality, sometimes significantly. The tradeoff between privacy (measured by $\epsilon$, the privacy budget) and utility is steep for large language models. At $\epsilon = 1$ (strong privacy), model quality can degrade substantially. At $\epsilon = 10$ (weaker privacy), quality is closer to non-private training but the privacy guarantee is weaker. Calibrating this tradeoff for specific applications requires empirical experimentation and domain expertise. We will explore differential privacy in full detail in the next chapter, including how to select privacy budgets and how fine-tuning with DP differs from pre-training with DP.

Machine unlearning seeks to remove the influence of specific training examples from a trained model without full retraining. This addresses the GDPR right-to-erasure problem: if a user requests deletion of their data, unlearning would remove that data's influence from the model without the expense of training from scratch.

The conceptual foundation of unlearning is simple: reverse the effect of the gradient updates that were influenced by the target examples. In practice, this is significantly harder than it sounds, because gradient descent is not reversible and the influence of any single example is distributed diffusely across all the model's parameters through thousands of parameter updates.

Current unlearning techniques for neural networks fall into two broad categories:

• Exact unlearning: Provably removes all influence of the target examples. Requires either full retraining or techniques based on SISA training (Sharded, Isolated, Sliced, and Aggregated), which partitions training data so that only a fraction needs to be retrained when data is removed.
• Approximate unlearning: Uses gradient-based methods to nudge the model away from the target examples. Faster but provides weaker guarantees. Also called gradient ascent on the forgotten examples, this approach directly penalizes the model for assigning high probability to the target sequences, but cannot guarantee that all influence has been removed.

For large language models, exact unlearning is computationally intractable without architectural changes. Approximate unlearning is fast but has been shown to inadequately protect against membership inference attacks. A model that has been approximately unlearned may no longer verbatim reproduce the target sequence, but may still exhibit elevated probability for related content, and membership inference attacks may still correctly identify the example as training data.

SISA training is worth understanding in some detail because it represents the most principled approach to the unlearning problem. The training corpus is divided into $S$ shards of equal size. Each shard is further divided into $R$ slices. Models are trained incrementally: first on slice 1 of each shard, then on slices 1-2, and so on, with a checkpoint saved after each slice is processed. When a deletion request arrives, only the shards containing the target example need to be retrained from the most recent checkpoint that predates the introduction of that example.

In the best case, when the target example is in the last slice of a single shard, only $1/S$ of the total training cost is incurred for unlearning. In the worst case, when it appears in the first slice, the cost is $1/S \times R$, which is still a fraction of full retraining. With $S = 20$ shards and $R = 10$ slices per shard, the worst-case unlearning cost is $1/200$ of full retraining. This makes SISA practical for applications where unlearning requests are infrequent and the training data can be structured into shards at the outset. The main limitation is the need to plan for unlearning before training begins, which requires knowing the sharding strategy in advance.

At deployment time, systems can screen inputs for prompts that seem designed to extract training data and screen outputs for patterns matching known PII formats. This is a defense-in-depth measure rather than a root-cause fix: it catches obvious extraction attempts and obvious PII leaks, but determined adversaries can evade these filters, and the filters themselves may miss PII that does not match known patterns.

Output scanning is typically implemented as a post-processing step that applies the same PII classifiers used during data preprocessing to model outputs before they are returned to users. When a high-confidence PII detection occurs, the output can be blocked, redacted, or flagged for human review. Several deployed systems use this approach as a last-resort safety net, with the expectation that it will catch some leaks but not all.

The principal limitation of output scanning is indirect PII. Information that identifies someone in combination with other context but not on its own is difficult to detect without access to the full context of who the user is and what they might infer from the output. If a model generates a response that mentions a first name, a job title, and a city, that combination might uniquely identify a person even though none of the three pieces of information is PII on its own. No output scanner can reliably detect this without deep knowledge of the deployment context and the user's identity.

Input filtering faces a similar limitation. Simple extraction attacks that use unusual prompts or repeated tokens are easy to detect and block. Sophisticated attacks that construct prompts designed to blend with legitimate usage are much harder. The arms race dynamic is unfavorable: defenders must block all attacks, while attackers need only find one that works.

The mitigations described above are most effective when combined. Deduplication addresses the most common source of memorization (repetition) but leaves single-occurrence PII exposed. PII scrubbing reduces the density of sensitive information but has imperfect recall. Differential privacy provides formal guarantees but degrades model quality. Output scanning catches obvious leaks but misses indirect PII. Together, these layers create a defense that resists more failure modes than any single technique alone.

The chart below summarizes how each mitigation layer reduces extractable memorization, based on empirical results from published studies. The values represent approximate reduction in extracted examples relative to a baseline model with no mitigations.