Memorization and Privacy in Language Models › PII in Training Data

The memorization problem would be less urgent if training corpora contained no private information. Unfortunately, the opposite is true: large web-scale training datasets contain substantial amounts of personally identifiable information, and this is not accidental. The web evolved to host contact information, discussion forums, legal records, and social profiles. Any system that systematically scrapes the web will capture all of these categories.

Common Crawl and similar web archives contain snapshots of public web pages. Many of these pages include PII through a variety of mechanisms.

Contact pages and directories are the highest-density PII source. Business directories, professional association membership lists, and alumni networks explicitly list names, addresses, phone numbers, and email addresses. These listings are publicly accessible and frequently crawled. They are also frequently out of date: a person's contact information may remain in a directory for years after they have changed jobs, moved, or requested removal.

Forum posts and comment sections are a qualitatively different source of PII. Users discuss personal situations and sometimes include identifying details, either intentionally (asking for help with a specific situation) or inadvertently (mentioning a person's name in context, sharing a screenshot that includes metadata, or referencing a real address in the course of a discussion). Forum posts are rich with narrative context, which means PII in forums often appears with more surrounding information than in a plain directory listing.

News articles mention individuals by name in connection with events. People who were newsworthy for any reason, including crime victims, accident witnesses, parties in public disputes, or simply attendees at a public event, may find their names appearing in training data. The distinction between "news-relevant" and "private" is culturally and legally contested. A person mentioned in a local newspaper story about a neighborhood dispute did not necessarily consent to having their name encoded in a language model.

Government and legal documents contain names and addresses of parties in public records. Court filings, property records, business registrations, and voter rolls are often technically public but were not expected by individuals to be machine-readable and searchable in the way that training corpora make possible. The aggregation of this information with other sources dramatically increases privacy risk beyond what any individual record creates on its own.

Code repositories present a distinctive PII risk. Developers inadvertently commit API keys, credentials, and test data containing real user information with surprising regularity. Studies of GitHub and similar repositories have found thousands of real email addresses, authentication tokens, and database connection strings. Even when these are subsequently revoked or removed, they may persist in git history and web snapshots that end up in training data.

The PII in these sources ranges from clearly public (a politician's official phone number) to ambiguously public (a person's name on a company directory) to clearly private (a medical professional's personal cell phone number scraped from a small business listing they no longer maintain). Training data curation cannot easily distinguish these categories at scale, which is why the privacy problem is fundamentally difficult to solve through filtering alone.

Research by Subramani and colleagues at AI researchers quantified PII in Common Crawl-derived datasets. They found that a sample of 15 million training documents contained tens of thousands of unique email addresses, phone numbers, and social security numbers. The density of PII was not uniform: certain categories of websites (job boards, member directories, legal filings) contributed disproportionately.

An important complication is that what counts as PII depends on context and jurisdiction. Under GDPR, even an IP address can be PII. Under HIPAA, a patient's date of birth combined with their diagnosis is protected health information. Under California's CCPA, a person's inferences drawn from professional or employment information, education information, or commercial information can be protected. Language models trained on web data potentially encode all of these categories, and the model itself does not know which sequences are sensitive.

The aggregation problem compounds this. A model might memorize a person's first and last name from one document, their city of residence from another, their employer from a third, and their approximate age from a fourth. None of these facts is particularly sensitive in isolation. But a user who queries the model with enough context might receive a response that combines all four, effectively reconstructing a private profile from separately innocuous facts. This aggregation risk is much harder to measure and mitigate than direct PII memorization.

Training data often contains near-duplicate documents: multiple versions of the same news article, forum discussions that quote earlier posts, and scraped websites with repeated contact information across multiple pages. Near-duplicates amplify memorization risk because they increase the effective repetition count of the contained text without appearing as exact duplicates to standard deduplication filters.

If a person's contact information appears on their business's website, a cached version of that website, three scraped directories that aggregated the original, and two forum posts that mentioned the business, the same PII may appear six or more times in the training data. The model will memorize it reliably, and the information may be extractable even from brief prompts.

The insidious aspect of near-duplicate amplification is that it is invisible to standard occurrence counting. If you check how many times the string "John Smith, 555-0123" appears verbatim in the training data, you might find only two exact matches. But the same information might appear in paraphrased form in ten other documents, and the model integrates all of these training signals into a coherent representation that makes it highly likely to reproduce the underlying information regardless of surface form.

This motivates more sophisticated deduplication strategies that operate at the document level rather than the string level, using techniques like MinHash locality-sensitive hashing to identify and remove semantically similar documents even when they differ in phrasing.