PII Removal › Summary

PII removal is a targeted privacy protection step that occurs after deduplication and quality filtering and before the cleaned data enters the training pipeline. The key takeaways from this chapter are:

• PII spans a spectrum from direct identifiers (email, SSN) that are pattern-detectable to quasi-identifiers and sensitive context that require full-document understanding to assess. Legal frameworks like GDPR define PII expansively because re-identification attacks can combine apparently innocuous fields.
• Detection combines three methods: regex for format-constrained PII, NER for names and contextual entities, and specialized classifiers for domain-specific categories like medical or financial information. Each method covers a different part of the PII surface area, and production pipelines layer all three.
• Redaction strategies differ by use case: placeholder replacement preserves linguistic structure and is preferred for training data; pseudonymization preserves realism at greater complexity by replacing real PII with synthetic but realistic alternatives; document removal is appropriate for heavily PII-dense texts where inline redaction would leave incoherent output.
• Evaluation prioritizes recall: for training data, missing PII (false negative) is worse than over-redacting (false positive). Recall targets above 0.95 are common, with precision accepted at 0.80-0.85. Synthetic test sets, canary extraction tests, and membership inference attacks together give a complete picture of pipeline quality.
• Canary extraction tests measure the actual privacy risk by checking whether the model trained on processed data can be induced to reproduce PII from the original corpus. They catch systematic failures that precision-recall evaluation on synthetic data misses.
• Fundamental limitations remain: quasi-identifier combinations can re-identify individuals even after explicit identifiers are removed; non-English text requires separate language-specific pipelines; and the recall-precision trade-off forces a choice between privacy protection and data quality. PII removal reduces but does not eliminate memorization risk, and combining it with training-time differential privacy provides stronger formal guarantees.