Memorization and Privacy in Language Models › Extraction Attacks

Measuring memorization in a controlled setting is one thing. Extracting training data in an adversarial setting is another. Extraction attacks demonstrate that memorization creates real privacy risks, not just theoretical ones.

The extraction attack described by Carlini and colleagues operates as follows. The attacker has access to the model's generation API but does not have direct access to the training data. The attacker generates a large number of text completions from the model using diverse prompts, then analyzes these completions to identify likely training data.

The prompts can take several forms:

• Empty or near-empty prompts: Completions from very short prompts tend to capture the model's most strongly memorized content, since there is little context to guide generation toward anything other than familiar sequences.
• Style-shifting prefixes: Prompts designed to shift the model into a particular style or domain (for example, a few lines of code to shift into code generation mode, or a news article lead sentence to shift into news content) can expose memorized content from those domains.
• Repeated token attacks: Feeding the same token repeated many times (for example, 50 repetitions of the newline token) can destabilize the model's standard generation behavior and cause it to fall back to memorized sequences.

After generating completions, the attacker applies several filters to identify likely memorized text. Membership inference heuristics assess whether a completion looks "too specific" to have been generated by chance. Deduplication identifies sequences that appear verbatim in multiple completions, suggesting they are reliably produced rather than randomly generated. Cross-referencing with known datasets identifies completions that match known documents. Each of these filters reduces false positives, helping the attacker distinguish extraction from coincidentally matching text.

The scale of this attack is important to appreciate. Carlini et al. generated 600,000 completions from GPT-2 using a mix of prompts. From those 600,000 completions, they manually verified hundreds of training data extractions. The false positive rate was manageable, and the true positive rate was significant enough to make the attack practically useful. An adversary with access to more compute could scale this to millions of completions.

A related attack is membership inference: given a specific text sequence, determine whether it was part of the training data. This is weaker than extraction (we already know the sequence) but still useful for privacy auditing and for adversaries trying to confirm whether a target's data was used in training.

The simplest membership inference attack computes the model's perplexity on the candidate sequence. Training data tends to have lower perplexity (the model assigns it higher probability) than non-training data, because the model was optimized to predict training sequences. An attacker who obtains a target document can feed it to the model, compute the perplexity, and compare it to the distribution of perplexities for known non-training documents. If the target document's perplexity is anomalously low, it was likely in the training set.

More sophisticated attacks use the loss trajectory over the course of training, or compare the target model's behavior against a reference model trained on similar but non-overlapping data. The latter approach, called likelihood ratio attacks, achieves better accuracy by accounting for the fact that some text is inherently easy to predict regardless of whether it appeared in training. A Wikipedia article about a major historical event will have low perplexity under many language models simply because the content is commonly known and frequently discussed, not necessarily because it appeared in this particular model's training data. By comparing against a reference model, the likelihood ratio attack controls for this baseline difficulty.

The practical import of membership inference attacks extends beyond technical curiosity. A company could use membership inference to audit whether a third-party model was trained on proprietary data. A regulator could use it to verify whether a company's claimed data sources match the model's actual behavior. A journalist investigating a model provider could use it to confirm whether user-submitted content was incorporated into a subsequently-released model without consent.

The fundamental difficulty is that reducing memorization requires the model to forget specific training examples, but gradient descent does not selectively forget individual examples. Every update adjusts all weights in response to the entire batch. Targeted forgetting, sometimes called machine unlearning, is an active research area (discussed in detail in the Mitigations section), but current techniques work imperfectly and can degrade model performance.

Additionally, extraction attacks can exploit the model's context window. By providing a specific prefix from a document, the model's attention mechanism increases the probability of producing the document's continuation. Removing this context sensitivity would fundamentally undermine the model's ability to do useful tasks like summarization, completion, and question answering. The same mechanism that makes language models useful, their ability to pick up on contextual cues and generate contextually appropriate continuations, also makes them vulnerable to extraction with well-chosen prefixes.

There is also a fundamental information-theoretic dimension to this problem. A language model with $n$ parameters trained on $T$ tokens of data is a compression of that data. For a large model (tens of billions of parameters) trained on a moderately-sized dataset (hundreds of billions of tokens), the compression ratio is less than 1 bit per token. At this compression ratio, the model cannot store all training sequences verbatim. But it can store many short, specific sequences very compactly, because those sequences have low entropy relative to their length. This is precisely the profile of PII: email addresses, phone numbers, and social security numbers are short strings with very low entropy (they follow predictable patterns), so they are efficiently memorizable.

An emerging category of extraction attack combines memorization vulnerabilities with prompt injection. In a system that uses a language model as a backend for a user-facing application, an attacker may be able to craft inputs that cause the model to reveal memorized training data in a context where the system would normally filter or refuse to disclose it.

For example, a customer service chatbot might have instructions to never discuss competitor products. But an attacker who knows the model was trained on data containing competitor pricing could potentially craft indirect prompts that cause the model to summarize or quote content from those documents without explicitly framing it as competitor information. The model does not enforce policies through understanding; it generates the most probable continuation given the prompt, and if that continuation happens to include memorized data, it will produce it.

This category of attack is particularly relevant for retrieval-augmented generation (RAG) systems, where external documents are inserted into the model's context at inference time. If those documents contain sensitive information and the system's isolation mechanisms are imperfect, adversarial inputs may be able to extract document contents that were intended to inform the response but not be directly disclosed.