Toxicity Filtering › Toxicity Filter Evaluation

Evaluating a toxicity filter requires more than measuring classifier performance on a test set. You need to measure the downstream effect on the training data distribution and, ideally, on model behavior. A filter that achieves 95% AUC on a held-out test set may still cause serious problems if the test set is not representative of the corpus or if the filter's false positive biases are concentrated in important content categories.

At the classifier level, standard evaluation metrics apply:

• AUC-ROC: Measures classifier discrimination across all thresholds, appropriate for comparing classifiers without committing to a threshold choice. A random classifier has AUC 0.5; a perfect classifier has AUC 1.0.
• Precision at fixed recall: Choose a target recall level (e.g., 90% of toxic documents filtered) and measure what fraction of filtered documents are truly toxic. This directly measures how much collateral filtering you accept to achieve your recall target.
• Recall at fixed precision: Choose a target precision level (e.g., 80% of filtered documents are truly toxic) and measure what fraction of toxic documents are caught at that precision level.
• Average Precision (AP): The area under the precision-recall curve, which summarizes classifier quality across all thresholds. AP is more informative than AUC-ROC for highly imbalanced datasets where true positives are rare.
• Calibration (Expected Calibration Error): The degree to which classifier scores reflect true probabilities of toxicity. A score of 0.7 should correspond to approximately 70% probability of being toxic. Calibration matters for threshold selection: if a classifier is poorly calibrated, your threshold choices will be systematically off.

For calibration evaluation, reliability diagrams plot classifier output probability on the x-axis against the fraction of truly toxic examples in each probability bin on the y-axis. A well-calibrated classifier produces points near the diagonal. The Expected Calibration Error (ECE) is the weighted mean absolute deviation from the diagonal:

$$\text{ECE} = \sum_{b=1}^{B} \frac{|B_b|}{n} \left| \text{acc}(B_b) - \text{conf}(B_b) \right|$$

where:

• $B_b$: the set of documents whose predicted probability falls in bin $b$
• $|B_b|$: the number of documents in bin $b$
• $n$: total number of documents
• $\text{acc}(B_b)$: the true fraction of toxic documents in bin $b$ (the "accuracy" in the calibration sense)
• $\text{conf}(B_b)$: the mean predicted probability for documents in bin $b$

Low ECE means predicted probabilities are trustworthy as probability estimates. High ECE means the classifier's scores are poorly calibrated and should not be treated as probabilities when choosing thresholds.

Beyond standard accuracy metrics, toxicity classifiers should be evaluated for disparate impact across demographic groups. The key metrics are:

Subgroup AUC: Measure classifier AUC separately on subsets of examples that explicitly mention a specific group. Significant drops in subgroup AUC indicate the classifier performs differently for different groups. For example, if overall AUC is 0.92 but AUC on examples mentioning LGBTQ+ people is 0.78, the classifier is substantially less reliable for that category.

False positive rate parity: The fraction of non-toxic documents that are incorrectly flagged as toxic should be approximately equal across groups. If the false positive rate for documents mentioning Muslim people is 15% while the false positive rate for documents mentioning Christian people is 3%, the filter will disproportionately remove content about Muslim people from your training corpus, regardless of whether that content is benign.

Counterfactual consistency: Pairs of sentences that differ only in a demographic identifier (e.g., "I am a [group] person" versus "I am a [other group] person") should receive similar toxicity scores. Significant differences indicate group-specific sensitivity rather than content-based toxicity detection. The Jigsaw Unintended Bias in Toxicity Classification dataset provides labeled pairs of this type, and it remains the standard benchmark for this evaluation.

Pinned AUC (BPSN and BNSP): The Jigsaw competition introduced two specialized metrics. BPSN (Background Positive, Subgroup Negative) measures AUC on examples that are toxic overall but do not mention a specific group, paired with examples that mention the group but are not toxic. BNSP (Background Negative, Subgroup Positive) measures the reverse. Together they diagnose whether the classifier is biased toward associating a group with toxicity (high BPSN, low BNSP) or toward treating group-mentioning toxic content as benign (low BPSN, high BNSP).

After applying a toxicity filter to a training corpus, measure:

• Filtered fraction by domain: What percentage of web pages, books, Wikipedia articles, and other sources are removed? Large variation by domain may indicate miscalibration or domain mismatch between the classifier's training data and the corpus.
• Filtered fraction by language: Multilingual corpora often have language-specific false positive rates. A classifier trained primarily on English may be poorly calibrated for other languages, assigning higher toxicity scores to non-English text simply because it is unusual relative to the training distribution.
• Manual audit of filtered content: A random sample of filtered documents should be reviewed by human annotators to estimate the empirical precision of the filter. If your estimated precision is below 70-80%, you are over-filtering substantially and should raise the threshold.
• Vocabulary drift: Compare token frequencies before and after filtering. Unusual drops in frequency for terms associated with specific communities indicate disparate impact. This is most efficiently computed by computing the ratio of pre-filter to post-filter frequency for every token and flagging the tokens with the largest drops.

The ultimate test of toxicity filtering is its effect on model behavior after training. Corpus-level metrics tell you what was removed; model-level metrics tell you whether the removal had the intended effect. Required evaluations include:

• Toxicity benchmarks: Held-out test sets of known toxic and non-toxic examples that the model never saw during training. Measure generation toxicity by prompting the model with topic openings and evaluating the completions.
• Targeted probe sets: Prompts designed to elicit toxic completions. A model trained on insufficiently filtered data will complete "People of [group] are..." with negative stereotypes more often than a model trained on filtered data. The RealToxicityPrompts dataset provides a large collection of such probes.
• Fairness probes: Measure whether model toxicity levels differ by demographic group mentioned in the prompt. A model that generates more toxic content about some groups than others has learned biased associations that the training data filter did not fully address.
• Counter-speech performance: Test whether the model can accurately refute hate speech and discuss hate-related topics in an educational context. Over-filtering degrades this capability, and a model that cannot engage with hate speech topics educationally has limited utility for content moderation, historical education, and safety research applications.