PII Removal › PII Removal Evaluation

Evaluating PII removal is harder than evaluating most NLP tasks because the ground truth is inherently private. You cannot publish a labeled dataset of real PII without defeating the purpose. Evaluation therefore relies on synthetic test sets, held-out annotated samples, and indirect metrics. Each approach captures different aspects of the pipeline's performance, so a complete evaluation uses multiple complementary methods.

The most common evaluation approach generates a synthetic test set by embedding fake-but-realistic PII into real or synthetic document text. The PII is generated to cover the target categories (email, phone, name, and so on) and to appear in diverse contexts: mid-sentence, at the start of a document, in structured tables, in informal prose, and in multi-line addresses. The diversity of contexts matters because many PII detectors perform well on clean prose but fail on noisy or structured formats that appear frequently in web crawls.

Precision and recall are computed in the usual way. Precision measures the fraction of detected spans that are PII, catching the rate at which the pipeline flags legitimate content. Recall measures the fraction of true PII that the pipeline detects, showing how much PII leaks through. These two quantities are in tension: a detector that flags everything achieves perfect recall but terrible precision. A detector that flags nothing achieves perfect precision on an empty detection set but terrible recall.

The F1 score combines these into a harmonic mean, but the F1 score conceals the asymmetry in the costs of false positives and false negatives for privacy applications. A missed SSN (false negative) is much worse than a falsely flagged date (false positive). For training data PII removal, the desirable operating point strongly favors recall over precision. Recall values above 0.95 are typical targets, with precision accepted as low as 0.80-0.85. This means you are willing to erroneously remove some legitimate text in exchange for catching nearly all real PII.

Synthetic test sets have a known limitation: the PII they contain was generated to be detectable by the methods you used to design the test. If your evaluation uses the same templating approach as your training data, you may overestimate performance on the distribution of PII found in web text. Wild-web PII is more diverse in format, context, and language than any synthetic test set can capture. A good evaluation supplements synthetic test sets with held-out samples from the source corpus, annotated by human reviewers under controlled privacy conditions.

A stronger evaluation uses canary sequences: carefully crafted strings that are inserted into the training data and then searched for in the model's outputs. If the canary can be extracted from a trained model, it demonstrates that the model memorized the training example containing it. This technique, introduced by Carlini et al. (2019), directly measures what PII removal is trying to prevent.

For PII removal evaluation, the canary test adapts as follows. Known PII-containing documents are processed by the PII removal pipeline, and a model is trained on the processed corpus. At evaluation time, the model is prompted with partial contexts that surrounded the original PII and checked whether it completes with the original private information. If the pipeline missed the PII and it remained in training data, the model may reproduce it. If the PII was correctly removed and replaced with a placeholder, the model should complete the prompt with the placeholder rather than the original value.

The extraction success rate is measured as the fraction of canary prompts for which the model reproduces the original PII within its top-k completions for some value of k. A lower extraction rate indicates better PII removal. The test is typically run with k values of 1 (exact greedy completion), 10, and 100 to capture both highly confident memorization and probabilistic memorization.

Canary extraction tests are expensive because they require training a model, but they directly measure the privacy risk that PII removal is meant to prevent. They also catch systematic failures that synthetic precision-recall evaluation misses: if the pipeline fails on a specific writing style or document format that was well-represented in the training data, canary tests will reveal this because the model will have memorized those documents and can reproduce them under extraction.

A variant of the canary test, proposed in research on differential privacy auditing, uses membership inference rather than exact extraction. A membership inference attack attempts to determine whether a given document was in the training set by measuring the model's behavior on that document relative to similar non-training documents. Successful membership inference for PII-containing documents indicates that those documents' content influenced the model's weights, which is the mechanism through which PII becomes extractable.

A complementary approach uses membership inference attacks to estimate how much information about individual training examples survives in model weights. A successful membership inference attack can distinguish, better than chance, whether a given document was in the training set. When applied to documents containing PII, this measures how much the PII contributed to memorization.

Membership inference is an indirect measure of PII removal quality. A pipeline that removes PII successfully should reduce the memorization signal for those documents, making membership inference harder. If membership inference success rates remain high after PII removal, the pipeline is not effectively reducing privacy risk, even if it achieves high recall on synthetic precision-recall benchmarks. The persistence of the memorization signal despite PII removal can indicate that the pipeline is removing the explicit identifiers but leaving enough co-occurring context to still uniquely identify documents in the training set.

The practical use of differential privacy auditing is to set a target: after PII removal, the membership inference advantage for PII-containing documents should be no higher than for clean documents. If it is higher, the PII removal pipeline is not doing its job. This target provides a quantifiable goal that precision-recall metrics cannot capture directly.

For the structured PII categories detectable by regex (email, phone, SSN), accuracy can be evaluated programmatically without requiring annotated test data. A post-processing check scans the output corpus for known-format PII patterns and reports the fraction remaining. This is not a complete evaluation because it cannot check for names or contextual PII, but it provides a fast, cheap sanity check that the regex layer is functioning correctly after any pipeline changes.

A practical approach is to run the regex scan on a random 1% sample of the processed corpus and report the residual PII rate per category. If the email residual rate is above a threshold, it indicates a bug in the pipeline (perhaps a regex that was recently updated and introduced a regression) rather than a fundamental limitation. This check should run automatically as part of every pipeline deployment to catch accidental regressions before they affect the full corpus.