Memorization and Privacy in Language Models › Measuring Memorization

The challenge of measuring memorization is that we cannot simply ask the model "did you memorize this?" We need empirical methods that probe the model's behavior without assuming knowledge of what was memorized. The most influential work in this area developed both the formal definitions and the practical tooling that the field now uses.

The most influential framework for measuring memorization was introduced by Carlini and colleagues in a 2021 paper titled "Extracting Training Data from Large Language Models." Their approach defines memorization formally and provides a practical extraction method that demonstrated, empirically, that real private information could be extracted from deployed language models.

They define a training example $s$ as $k$-memorized if there exists a prefix $p$ of length $k$ tokens drawn from $s$ such that giving the model $p$ makes it dramatically more likely to produce the continuation of $s$ than a model would without any meaningful context. Formally:

$$\Pr[\text{LM produces continuation of } s \mid p] \gg \Pr[\text{LM produces continuation of } s \mid \text{random prompt}]$$

where:

• $s$ is the full training sequence being tested (the candidate memorized example)
• $p$ is a prefix of $k$ tokens taken from the beginning of $s$ (the prompt given to the model)
• $k$ is the prefix length, which controls how much context the attacker provides
• LM refers to the language model's autoregressive generation process
• The right-hand side is the baseline probability of producing the continuation when given only a random, uninformative prompt

The $\gg$ notation captures the essential idea: the conditional probability given the natural prefix should be orders of magnitude larger than the baseline. If the model is no more likely to produce a sequence when given its own prefix than when given a random prompt, the sequence is not memorized in any meaningful sense. If the probability ratio is extremely large, the sequence is almost certainly memorized verbatim.

The parameter $k$ controls the strength of the definition. When $k$ is small (say, 5 tokens), we are asking whether a very short prompt can elicit the sequence, which is a strong form of memorization. When $k$ is large (say, 100 tokens), we are giving the model almost the entire sequence and asking if it can complete the rest, which is a weak form of memorization. A sequence that is $5$-memorized is a greater privacy concern than one that is $100$-memorized, because extracting it requires less prior knowledge of the sequence.

In practice, the exact probability is difficult to compute for long sequences, so Carlini et al. operationalize this by comparing the model's perplexity. Perplexity is the exponential of the average negative log-likelihood per token:

$$\text{PPL}(s \mid p) = \exp\!\left(-\frac{1}{|s| - k} \sum_{i=k+1}^{|s|} \log P(s_i \mid s_1, \ldots, s_{i-1})\right)$$

where:

• $|s|$ is the total number of tokens in the sequence $s$
• $k$ is the number of prefix tokens provided
• $s_i$ is the $i$-th token in the sequence
• $P(s_i \mid s_1, \ldots, s_{i-1})$ is the model's probability of the $i$-th token given all preceding tokens

Lower perplexity means the model assigns higher probability to the sequence. A sequence is considered memorized when its perplexity given the natural prefix is dramatically lower than its perplexity given a random prompt of the same length. The ratio of these two perplexities quantifies how much the natural prefix raises the probability of the sequence compared to a generic prompt. This ratio, which Carlini et al. call the memorization score, is the central empirical quantity in their framework.

A related but stronger concept is extractable memorization: a training example is extractable if an adversary who does not know the example can nevertheless craft a prompt that causes the model to generate it.

The extraction procedure works as follows. An attacker generates a large number of model completions (hundreds of thousands to millions) using a variety of prompts, including common prefixes and intentionally unusual prompts designed to shift the model out of generic behavior. The attacker then compares these completions against any available dataset or applies deduplication heuristics to identify repetitive exact sequences. Sequences that appear verbatim in the model's output and are unique enough to be unlikely coincidences are flagged as extracted training data.

The key insight is that an adversary does not need access to the training data to run this attack. The extraction procedure requires only black-box access to the model's text generation API. This makes it applicable to any deployed model, including models where the training data is not publicly known. The attacker's investment is computation, not information.

Carlini et al. applied this method to GPT-2 (the 1.5 billion parameter version) and successfully extracted hundreds of examples of training data, including:

• Names, phone numbers, and email addresses of specific individuals
• IRC chat logs and personal blog posts
• News articles and Wikipedia text
• A 128-bit UUID that appeared only once in the training data

The single-occurrence UUID extraction is particularly striking. A model trained on a 40 GB corpus was able to reproduce a unique 32-character string that appeared exactly once. The probability of generating that string by random chance is vanishingly small, which confirms true memorization rather than coincidence. This result demolished the intuitive assumption that neural network compression would prevent direct recovery of specific training examples.

A useful numerical summary is the fraction of training examples that are $k$-memorized for a given prefix length $k$. Larger $k$ means we give the model more of the target sequence before asking it to complete, which makes memorization easier to trigger. By varying $k$ from 0 (no prefix) to 50 (50 tokens of prefix), we can construct a memorization curve that characterizes how much of the training data is recoverable at different levels of prompting.

Research on GPT-2 found that roughly 0.1% to 1% of training examples could be extracted with prefix lengths between 10 and 50 tokens. While this sounds small, GPT-2 was trained on approximately 40 GB of text. A 1% memorization rate across such a corpus amounts to hundreds of megabytes of extractable training data. When scaled to models trained on trillions of tokens, even a fraction of a percent represents a staggering quantity of memorized content.

A follow-up study by Carlini and colleagues in 2022 examined larger models and found that memorization scaled predictably with model size, training duration, and repetition in the training data. The study introduced the concept of a "memorization curve": a plot of memorization rate against prefix length. Models that memorize more aggressively show steeper memorization curves, reaching high extraction rates with shorter prefixes.

Not all models memorize at the same rate, and not all types of training data are equally at risk. Several factors influence how much a specific example is memorized:

Repetition in training data. The single strongest predictor of memorization is how many times an example appears in the training corpus. A sequence that appears 100 times across the training data is far more likely to be memorized than one that appears once. This makes data deduplication a practical privacy mitigation, because deduplication directly reduces this main risk factor.

Model size. Larger models memorize more. A 6 billion parameter model memorizes proportionally more training examples than a 117 million parameter model, even when both are trained on the same data. This is counterintuitive to many practitioners who assume that larger models would generalize better and rely less on specific examples, but the relationship runs in the opposite direction: more capacity enables more memorization. The model has room in its parameter space to store more specific patterns, and gradient descent fills that space.

Training duration. Models trained for more steps memorize more. Early in training, the model is learning broad patterns. As training continues, it gradually encodes more specific information. This means that longer training runs carry higher memorization risk, even when the training data is fixed.

Position in context. Some research suggests that training examples near the end of long documents are memorized more reliably than examples near the beginning, possibly because the model encounters the end of a document more often when data is chunked. Context window chunking strategies can inadvertently influence which parts of a document receive more training signal.

Template and structured text. Highly formulaic text, such as legal boilerplate, contact information templates, or code with repeated patterns, is memorized at higher rates than diverse natural prose. The regularity reduces the model's need to learn generalizable patterns and instead rewards verbatim reproduction. If you have ever seen a language model correctly reproduce a standard legal disclaimer, you have observed template memorization in action.

Data source characteristics. Different data sources contribute to memorization at different rates. Code repositories, for instance, contain large amounts of repeated boilerplate (license headers, standard imports, framework scaffolding) that appears across thousands of files. Legal documents often contain standardized language that appears verbatim across many filings. These structural features of certain data sources make them particularly prone to producing memorized content.