Memorization and Privacy in Language Models › What Is Memorization?

Before measuring memorization, we need a precise definition. Intuitively, a model has memorized a training example if it can reproduce that example. But this definition has several dimensions worth unpacking, and getting the definition right matters both for measurement and for understanding the severity of the risk.

Verbatim memorization means the model can reproduce a sequence of tokens character-for-character as it appeared in training data. This is the most straightforward case: given a prefix from the training text, the model generates the exact continuation.

Approximate memorization is broader. A model might reproduce a piece of text with minor variations: different whitespace, slightly reordered clauses, or a changed proper noun. These approximate matches are harder to detect but still represent real privacy risks. A model that reliably outputs someone's address with the street number off by one is still leaking private information. Approximate memorization also makes auditing harder because string-matching checks for exact reproduction will miss it.

There is a third category that sits between these two: semantic memorization. This occurs when the model does not reproduce the surface form of a training example but does reproduce its specific semantic content in a way that reveals private information. If a training document described "a 42-year-old male patient with diabetes and hypertension in Seattle admitted on March 3rd," a model that generates this fact in response to a medical query has effectively leaked private health information even if the phrasing differs. Semantic memorization is the hardest to measure and the hardest to defend against.

Some memorization is intentional. You would not want your language model to forget that Paris is the capital of France, or to mangle common phrases. This type of memorization captures broadly shared factual knowledge. It is benign and desirable.

Unintentional memorization occurs when a model happens to reproduce specific, private information that appeared in the training data. This includes personally identifiable information (PII) like names combined with contact details, unique text from private documents that ended up scraped into the training corpus, or rare sensitive facts that a user never intended to share publicly.

The distinction between intentional and unintentional memorization is easier to state than to implement. The model does not mark certain memories as "public knowledge" and others as "private." Both categories emerge from the same gradient descent process applied to the same training data. Context and training data curation are the only levers available before training; extraction resistance and privacy-preserving training techniques are the levers available after. This means the burden falls heavily on whoever constructs and curates the training corpus.

An important nuance is that memorization can shift from unintentional to functionally intentional based on how a model is fine-tuned or deployed. A base language model may have memorized a phone number incidentally during pretraining on web data. If that model is then fine-tuned on a customer support corpus where the same number appears frequently, and then deployed as a customer service agent, the model may now reliably produce that number in response to relevant prompts. At some point along that chain, the memorization crosses from accidental to embedded behavior.

Research has shown that memorization is not a binary property. It exists on a spectrum, and different parts of the training corpus are memorized to different degrees. Thinking about memorization as a continuous property rather than a yes/no distinction leads to better-calibrated risk assessments and more targeted mitigations.

The broad categories along the spectrum are:

• Highly memorized: Sequences that appeared many times across the training data or that are very long and specific. Book passages, code snippets, and contact information in templates all fall here. These can often be extracted with short prompts.
• Moderately memorized: Sequences that appeared a few times or that have partial uniqueness. These may be extractable with carefully crafted prompts but not with generic ones.
• Weakly memorized: One-off sequences that were encountered during training but are only recoverable under specific prompting conditions, if at all.
• Not memorized: Sequences with high perplexity that were not reinforced by repetition and show no evidence of being stored in the model's weights.

Understanding where on this spectrum a given piece of data falls helps predict privacy risk and informs data curation decisions. A practical implication is that the most dangerous memorization is also the easiest to measure and mitigate: repeated sequences stand out in audit procedures and are removed efficiently by deduplication. The harder problem is the long tail of weakly memorized content that does not show up clearly in automated checks but can still leak under the right adversarial conditions.

To understand memorization it helps to think about how gradient descent produces this behavior. During training, the model is rewarded for assigning high probability to training sequences. For sequences that appear many times, the gradient signal is correspondingly strong: every appearance of the sequence pushes the weights toward configurations that make that sequence highly probable. The model is not trying to memorize; it is trying to minimize loss. Memorizing frequently-repeated sequences is a natural consequence of doing so.

For very unusual sequences, a different mechanism can drive memorization. Highly specific text, like a UUID or a rare proper name, has very few plausible completions. The model learns that when it sees a certain unusual prefix, one particular continuation is overwhelmingly more probable than any alternative. This tight association between a distinctive prefix and its continuation is essentially memorization, even when the sequence appeared only once.

This means we have two distinct routes to memorization: repetition-driven memorization (common text reinforced across many training examples) and specificity-driven memorization (unique text with low entropy that the model finds easy to compress). Both contribute to the overall memorization rate, and they call for slightly different mitigations.