Memorization and Privacy in Language Models › Summary

Language model memorization occurs when a model encodes specific training examples so precisely that those examples can be extracted from the model's outputs under adversarial conditions. The key concepts from this chapter are:

• $k$-memorization measures how easily a training example can be triggered with a $k$-token prefix. Larger prefixes make memorization easier to demonstrate, and smaller prefixes indicate stronger, more easily-exploitable memorization.
• Extractable memorization is the stronger property: an adversary without prior knowledge of the training data can craft prompts that cause the model to reproduce it. The 2021 Carlini et al. paper demonstrated this concretely for GPT-2, including extraction of a UUID seen only once in training.
• Repetition in training data is the primary driver of memorization. Sequences seen hundreds of times are reliably memorized; sequences seen once may still be memorized but are harder to extract.
• Larger models memorize more, not less. Model size and memorization rate are positively correlated, counter to the intuition that more capacity enables more generalization.
• PII in training corpora is prevalent in web-scale datasets, arising from scraped contact directories, forum posts, legal documents, and code repositories. Near-duplicate amplification increases the effective repetition count of PII beyond what occurrence counts suggest.
• Membership inference attacks can determine whether a specific sequence was in the training data by comparing the model's perplexity on that sequence against a baseline distribution.
• Mitigations include data deduplication (most effective for repeated content), PII scrubbing (targeted at sensitive information), differential privacy during training (formal guarantees at a quality cost), machine unlearning (planned for in SISA training), and output scanning (last-resort defense-in-depth). Each has different cost-benefit tradeoffs and no single technique is sufficient on its own.
• Regulatory pressure from GDPR, CCPA, HIPAA, and emerging AI-specific regulations requires organizations to have concrete, auditable answers about how training data privacy is managed.

The next chapter explores differential privacy in depth, covering the DP-SGD algorithm and how the privacy-utility tradeoff can be managed systematically for language model training.