Memorization and Privacy in Language Models › Worked Example: Memorization Risk Assessment

Consider a practical scenario: you are training a customer service language model on historical support tickets. The tickets contain real customer names, email addresses, and account numbers. What is the memorization risk, and what mitigations should you apply?

Step 1: Count repetitions. First, determine how often specific pieces of information appear. A single customer's email address may appear in every support ticket they sent. If a customer sent 50 tickets, their email appears 50 times. Based on the memorization curves discussed earlier, this creates high memorization risk for that customer's contact information. Run an occurrence count across your entire corpus to identify the highest-risk sequences before training begins.

Step 2: Assess model size. Larger models have higher memorization rates. If you are fine-tuning a large foundation model on your support tickets, the base model's parameter capacity means it will memorize more of your fine-tuning data than a smaller model would. Consider whether a smaller model meets your performance requirements, especially if the primary use case does not require the additional capacity of the largest models.

Step 3: Apply deduplication. Remove exact and near-exact duplicates from the training corpus. For customer emails that appear in many tickets, keep only one occurrence (or a small number, enough for the model to learn the domain vocabulary without memorizing the specific address). This is the most effective step: it directly attacks the repetition mechanism that drives most high-probability extraction.

Step 4: Scrub PII. Use a PII detection pipeline to replace names, email addresses, phone numbers, and account numbers with generic placeholders before training. Your model will learn customer service patterns without learning specific customer identities. Test the pipeline's recall rate on a hand-labeled sample of your data to understand what fraction of PII it misses.

Step 5: Consider differential privacy. If the support tickets contain particularly sensitive information (health issues, billing disputes, security incidents), evaluate whether DP-SGD is warranted. The privacy guarantee it provides may justify the quality tradeoff. Start with a high $\epsilon$ (weak privacy) to measure the quality impact, then tighten $\epsilon$ until the quality degradation becomes unacceptable.

Step 6: Audit after training. After training, run a memorization audit: attempt to extract training data using the extraction procedure, and check outputs for PII patterns. This validates whether your mitigations worked and provides evidence of due diligence for regulatory purposes. An audit that finds zero extracted PII provides much stronger regulatory protection than a stated intention to prevent memorization.

The worked example illustrates that memorization risk management is not a single decision but a sequence of decisions made at the data preparation, training configuration, and deployment stages. The earlier in the pipeline you intervene, the more effective and less expensive the intervention tends to be.