PII Removal › PII Removal Strategies

Once PII is detected, you must decide what to do with it. The options form a spectrum from aggressive to conservative, with different consequences for downstream model quality. The choice of strategy interacts with the choice of detection method: a high-recall detector with a conservative strategy (placeholder replacement) may produce better outcomes than a moderate-recall detector with an aggressive strategy (document removal), because you preserve more documents while still removing the most dangerous content.

The most common strategy replaces detected PII spans with generic placeholder tokens. Email addresses become [EMAIL], phone numbers become [PHONE], person names become [NAME]. The surrounding text remains intact with only the PII replaced.

Placeholder replacement preserves the syntactic structure of the original sentence. A model trained on "Please contact [EMAIL] for the invoice" learns valid sentence patterns, the pragmatics of how contact information is requested, and the linguistic context around email addresses, without learning any specific email address. This is the dominant approach in practice because it balances privacy protection with data utility. The redacted text is still useful for learning language, even if it cannot be used to identify individuals.

The choice of placeholder token matters for downstream tokenization. If the placeholder is a rare token or an out-of-vocabulary string, the model's tokenizer may fragment it into many subwords, adding noise to the learned representation. Using placeholder tokens that are single vocabulary items, or adding them as special tokens in the vocabulary, produces cleaner training examples. The Pile dataset, for example, used the token <|pii|> as a placeholder, adding it as a special token to the vocabulary so it would never be split during tokenization.

A refinement is to use type-specific placeholders so the model learns that different PII categories appear in different contexts. [EMAIL_ADDRESS] for emails, [PHONE_NUMBER] for phones, and [PERSON_NAME] for names lets the model distinguish between "contact [EMAIL_ADDRESS] to schedule" and "speak with [PERSON_NAME] in room 302", which have different semantic implications even though both contain PII. Type-specific placeholders also preserve useful syntactic patterns: the model learns that [EMAIL_ADDRESS] typically appears as an object of contact verbs and as a string inside angle brackets in email headers, while [PHONE_NUMBER] appears after "call" or "fax" or in formatted contact blocks.

The main limitation of placeholder replacement is that it introduces tokens that will appear in the training data but not in most real-world text the model will process at inference time. A model that has learned "speak with [PERSON_NAME] in room 302" as a training example may generalize the pattern correctly to "speak with Dr. Chen in room 302" at inference time, or it may not, depending on how the placeholder token is represented in the embedding space. Using widely-used placeholder tokens that are already in the model's vocabulary helps bridge this gap.

Redaction removes the PII span and replaces it with nothing, or with a short indicator like REDACTED or ****. The resulting text may be grammatically incomplete, since the original sentence may have relied on the PII token for subject-verb agreement or pronoun resolution. "The report was submitted by REDACTED" is less natural than "The report was submitted by [NAME]", but both obscure the actual person.

Redaction is common in legal and government document processing where the goal is to match the appearance of physically redacted paper documents. Legal practitioners and government agencies have decades of experience with REDACTED as a marker, and their downstream readers understand what it means. For training data preparation, placeholder replacement is generally preferred over redaction because it maintains grammatical structure better and produces text that a language model can learn from without the spurious REDACTED token appearing everywhere.

One context where redaction is appropriate is when the PII is part of a URL, filename, or other structured string where placeholder insertion would break the format. The URL https://patient-portal.hospital.org/records/JOHN-SMITH/download contains a name in a structured position. Replacing "JOHN-SMITH" with "[NAME]" produces a valid-looking URL that may confuse a model that has learned URL syntax. Removing the path segment entirely produces a less informative but structurally valid URL.

When PII is so dense in a document that removing it would destroy the document's semantic coherence, the correct action is to remove the entire document rather than attempt inline redaction. A contact list, a roster of employees with personal details, or a thread where users repeatedly share identifying information about themselves or others will produce heavily redacted text that provides little useful signal to the model.

Consider a comment thread on a local community forum where users post their names, addresses, and phone numbers to coordinate a neighborhood meeting. After PII removal, the document might look like: "[NAME] at [ADDRESS] says [PHONE_NUMBER] works for me, or [NAME] at [ADDRESS] could host if [PHONE_NUMBER] calls first." This text teaches the model almost nothing useful about language while still potentially containing some residual identifying information. Document removal avoids this outcome by treating the document as non-salvageable.

Document-level removal decisions are typically rule-based: if more than some threshold fraction of tokens are redacted PII spans, remove the document. The threshold is calibrated empirically against a validation set of documents, with a human reviewer checking that documents near the threshold are appropriately classified. Values in the range of 5-15% PII token density are common triggers for document-level removal, depending on how aggressively the pipeline values corpus completeness versus privacy protection.

An alternative trigger for document removal is the presence of certain PII types regardless of density. A document containing a credit card number or a Social Security number is almost certainly not the kind of text you want in a general-purpose training corpus, regardless of what fraction of the document that number occupies. These categories can trigger document removal directly without a density threshold.

Pseudonymization replaces real PII with synthetic but realistic alternatives. Instead of replacing "John Smith" with [NAME], the pipeline substitutes a different real-sounding name like "Robert Chen". Instead of replacing a real email with [EMAIL], it substitutes user4921@example.com. The result is text that reads naturally and contains names and contact information in the positions where the original contained them.

Pseudonymization preserves the realism of the training text at the cost of more complex processing. A model trained on pseudonymized data learns natural name patterns, the syntactic positions where names appear, and coreference patterns (the same name appearing multiple times to refer to the same entity) rather than learning that [NAME] appears in name-bearing contexts. This is particularly valuable for tasks where learning name syntax matters, such as dialogue summarization or structured information extraction.

The challenge is ensuring consistency: if "John Smith" appears ten times in a document, all occurrences must be replaced with the same pseudonym, or the model learns incoherent coreference patterns. A coreference resolution module must first identify all mentions of the same entity and then apply a consistent replacement. The coreference step adds complexity and potential for error: "John" alone and "Mr. Smith" alone and "John Smith" in full may all refer to the same person and must be linked before pseudonymization.

Pseudonymization also requires a synthetic data generation component that can produce realistic fakes for each PII category. Name lists for common first and last names in the relevant language are widely available. Random email generators that combine realistic-looking usernames with common domain names are straightforward. Phone number simulators that produce numbers in valid format ranges are also simple. More complex categories like medical record numbers or national ID numbers with embedded checksum constraints require format-aware generation that can produce checksum-valid synthetic values.

The consistency requirement also applies across documents in the same corpus. If your corpus contains multiple documents from the same forum, and "johnsmith42" appears as a username across all of them, pseudonymization must either replace all occurrences with the same pseudonym or replace each occurrence independently. The former approach better preserves coreference but requires a cross-document identity resolution step that is computationally expensive at corpus scale.