Toxicity Filtering › Implementation Walkthrough
Language AI HandbookAll chapters Toxicity Filtering › Implementation WalkthroughOne section · Sections in this chapter7 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement a toxicity filtering pipeline. We will work with a simulated dataset that captures the statistical properties of a realistic pretraining corpus: a mix of toxic content, borderline content, and benign content that includes the sensitive categories (counter-speech, group mentions) where over-filtering is most common. This allows us to measure classifier performance, calibrate a threshold, and audit the filter's behavior across demographic categories.
Setup and Installation
# Install required packages
# uv pip install detoxify scikit-learn numpy pandas matplotlib
import warnings
warnings.filterwarnings("ignore")Synthetic Toxicity Dataset
We will create a realistic synthetic dataset that simulates the key challenge: a mix of toxic content, borderline content, and benign content that a naive classifier might falsely flag. The category distributions reflect a realistic pretraining corpus, where toxic content is a minority of the total and the challenging cases make up a substantial fraction of the benign documents.
import numpy as np
np.random.seed(42)
# Simulate toxicity classifier scores and ground truth labels
# This represents a corpus of 2000 documents with ground truth labels
# The classifier scores are calibrated to match realistic Detoxify output distributions
n_docs = 2000
# Simulate different document categories
categories = {
"clearly_toxic": {"n": 200, "mean_score": 0.85, "std": 0.10, "label": 1},
"mildly_toxic": {"n": 300, "mean_score": 0.60, "std": 0.15, "label": 1},
"borderline": {"n": 200, "mean_score": 0.40, "std": 0.15, "label": 0},
"group_mention": {
"n": 300,
"mean_score": 0.30,
"std": 0.20,
"label": 0,
}, # Benign but mentions minority groups
"counter_speech": {
"n": 200,
"mean_score": 0.35,
"std": 0.18,
"label": 0,
}, # Benign counter-speech
"clearly_benign": {"n": 800, "mean_score": 0.08, "std": 0.07, "label": 0},
}
scores = []
labels = []
category_tags = []
for cat_name, cat in categories.items():
n = cat["n"]
cat_scores = np.clip(
np.random.normal(cat["mean_score"], cat["std"], n), 0.001, 0.999
)
scores.extend(cat_scores)
labels.extend([cat["label"]] * n)
category_tags.extend([cat_name] * n)
scores = np.array(scores)
labels = np.array(labels)
category_tags = np.array(category_tags)
# Calculate ground truth metrics
n_toxic = labels.sum()
n_total = len(labels)
prevalence = n_toxic / n_totalDataset: 2000 documents Toxic documents: 500 (25.0% of corpus) Non-toxic documents: 1500 (75.0% of corpus) Documents by category: clearly_toxic 200 docs [TOXIC] mildly_toxic 300 docs [TOXIC] borderline 200 docs [benign] group_mention 300 docs [benign] counter_speech 200 docs [benign] clearly_benign 800 docs [benign]
This distribution reflects a realistic pretraining corpus: toxic content is a minority of the total (about 25%), and the challenging cases (borderline, group mentions, counter-speech) make up a substantial fraction of the non-toxic documents. The group-mention category represents documents discussing minority communities in neutral or positive terms; the counter-speech category represents documents that engage directly with harmful claims in order to refute them.
Classifier Performance Analysis
With our dataset in place, we compute the standard classifier evaluation metrics. These give a threshold-independent view of classifier quality before we commit to a specific operating point.
from sklearn.metrics import (
average_precision_score,
precision_recall_curve,
roc_auc_score,
roc_curve,
)
# Calculate ROC curve and AUC
fpr, tpr, roc_thresholds = roc_curve(labels, scores)
auc_roc = roc_auc_score(labels, scores)
# Calculate Precision-Recall curve and Average Precision
precision, recall, pr_thresholds = precision_recall_curve(labels, scores)
avg_precision = average_precision_score(labels, scores)
# Find threshold that maximizes F-beta for beta=2 (recall-weighted)
beta = 2
f_beta_scores = []
for p, r in zip(precision, recall):
if p + r > 0:
f_beta = (1 + beta**2) * p * r / (beta**2 * p + r)
else:
f_beta = 0.0
f_beta_scores.append(f_beta)
f_beta_scores = np.array(f_beta_scores)
best_idx = np.argmax(f_beta_scores)
best_threshold = (
pr_thresholds[best_idx]
if best_idx < len(pr_thresholds)
else pr_thresholds[-1]
)
best_precision = precision[best_idx]
best_recall = recall[best_idx]
best_f_beta = f_beta_scores[best_idx]Classifier Performance Summary AUC-ROC: 0.958 Average Precision: 0.891 Optimal threshold (F-beta=2): Threshold: 0.383 Precision: 0.610 Recall: 0.966 F-2 score: 0.865
The AUC-ROC and average precision give us a threshold-independent view of classifier quality. The optimal threshold for our recall-weighted objective sits below 0.5, showing the asymmetric cost: we accept lower precision to achieve higher recall on toxic content. The threshold is selected by maximizing , which weights recall twice as heavily as precision.
Precision-Recall Tradeoff Visualization
False Positive Rate by Category
A critical diagnostic is how false positive rates differ across document categories. This directly measures the disparate impact of a given threshold: which categories of benign content are disproportionately removed? A threshold that is appropriate overall may still cause serious over-filtering for specific content types.
# Apply threshold and compute false positive rates by category
applied_threshold = best_threshold
filtered = scores >= applied_threshold
# Compute per-category false positive rates (among non-toxic docs)
fp_rates = {}
for cat_name in categories:
if categories[cat_name]["label"] == 0: # Only non-toxic categories
mask = category_tags == cat_name
cat_n = mask.sum()
cat_fp = (filtered[mask]).sum()
fp_rates[cat_name] = {
"total": cat_n,
"filtered": int(cat_fp),
"fp_rate": cat_fp / cat_n if cat_n > 0 else 0.0,
}
# Overall stats
n_filtered = filtered.sum()
tp = (filtered & (labels == 1)).sum()
fp = (filtered & (labels == 0)).sum()
fn = (~filtered & (labels == 1)).sum()
tn = (~filtered & (labels == 0)).sum()
overall_precision = tp / (tp + fp) if (tp + fp) > 0 else 0
overall_recall = tp / (tp + fn) if (tp + fn) > 0 else 0Filtering results at threshold tau=0.383 Documents filtered: 792 / 2000 (39.6%) True positives: 483 (toxic docs filtered) False positives: 309 (benign docs filtered) False negatives: 17 (toxic docs missed) Precision: 0.610 Recall: 0.966 False positive rates by category: borderline 51.0% (102/200 docs filtered) group_mention 35.3% (106/300 docs filtered) counter_speech 50.5% (101/200 docs filtered) clearly_benign 0.0% (0/800 docs filtered)
The false positive rates reveal the disparate impact directly. Categories like group_mention and counter_speech have substantially higher false positive rates than clearly_benign, even though none of these categories contain toxic content. This is the over-filtering risk in concrete numbers: at a threshold calibrated for high recall, you lose substantially more content from documents that discuss or critique hate speech than from neutral benign content. If those categories represent important community voices or educational content, the filter is causing harm.
Calibration Analysis
A well-calibrated classifier is essential for threshold selection. If the classifier outputs 0.7 for documents that are toxic only 40% of the time, your threshold selection will be systematically miscalibrated: you will believe you are operating at one precision-recall point when you are at another. The Expected Calibration Error (ECE) measures how far the classifier's probability estimates deviate from true frequencies.
from sklearn.calibration import calibration_curve
# Compute calibration curve
n_bins = 10
fraction_of_positives, mean_predicted_value = calibration_curve(
labels, scores, n_bins=n_bins, strategy="uniform"
)
# Compute Expected Calibration Error (ECE)
# ECE = weighted mean absolute difference between predicted probability and fraction positive
bin_counts = []
bin_width = 1.0 / n_bins
for i in range(n_bins):
lower = i * bin_width
upper = (i + 1) * bin_width
in_bin = (scores >= lower) & (scores < upper)
bin_counts.append(in_bin.sum())
bin_counts = np.array(bin_counts)
valid = bin_counts > 0
ece = (
np.sum(
bin_counts[valid]
/ n_total
* np.abs(fraction_of_positives - mean_predicted_value)
)
if valid.any()
else 0.0
)Calibration Analysis
Expected Calibration Error (ECE): 0.1155
(Lower is better; 0.0 = perfect calibration)
Calibration by probability bin:
Predicted Observed
0.045 0.000
0.144 0.003
0.250 0.034
0.351 0.105
0.448 0.286
0.545 0.435
0.648 0.654
0.749 0.862
0.850 0.973
0.952 1.000Calibration Reliability Diagram
The reliability diagram makes calibration quality visually obvious. Well-calibrated classifiers produce points near the diagonal; systematic deviations indicate that the classifier is overconfident (points below the diagonal) or underconfident (points above).
Threshold Sensitivity Analysis
Before committing to a threshold for a production pipeline, you should understand how key outcomes change as the threshold varies. This analysis reveals the relationship between threshold, filtering aggressiveness, and disparate impact, allowing you to make an informed choice rather than simply accepting the -optimal threshold.
import pandas as pd
thresholds = np.linspace(0.1, 0.9, 50)
results = []
for tau in thresholds:
filtered_mask = scores >= tau
n_filt = filtered_mask.sum()
tp_t = (filtered_mask & (labels == 1)).sum()
fp_t = (filtered_mask & (labels == 0)).sum()
fn_t = (~filtered_mask & (labels == 1)).sum()
prec = tp_t / (tp_t + fp_t) if (tp_t + fp_t) > 0 else 1.0
rec = tp_t / (tp_t + fn_t) if (tp_t + fn_t) > 0 else 0.0
pct_filtered = n_filt / n_total
# False positive rates for sensitive categories
counter_speech_mask = category_tags == "counter_speech"
group_mention_mask = category_tags == "group_mention"
cs_fp = (
filtered_mask & counter_speech_mask
).sum() / counter_speech_mask.sum()
gm_fp = (
filtered_mask & group_mention_mask
).sum() / group_mention_mask.sum()
results.append(
{
"threshold": tau,
"pct_filtered": pct_filtered,
"precision": prec,
"recall": rec,
"counter_speech_fp": cs_fp,
"group_mention_fp": gm_fp,
}
)
results_df = pd.DataFrame(results)
The left panel shows the precision-recall tradeoff that any filtering practitioner must manage. The right panel is equally important: it shows that the false positive rates for counter-speech and group-mention documents are substantially higher than for clearly benign content across the entire threshold range. This gap does not close at higher thresholds; it narrows, but both sensitive categories remain over-filtered relative to neutral benign content at any threshold that achieves meaningful recall. The pattern is structural and cannot be solved by threshold calibration alone; it reflects the underlying classifier's learned associations.
Applying the Filter and Auditing Results
# Simulate applying the filter to the corpus
final_threshold = best_threshold
corpus_filtered = scores >= final_threshold
corpus_kept = ~corpus_filtered
# Audit: compute fraction removed by category
audit_results = []
for cat_name in categories:
mask = category_tags == cat_name
n_cat = mask.sum()
n_removed = (corpus_filtered & mask).sum()
pct_removed = n_removed / n_cat if n_cat > 0 else 0.0
is_toxic = categories[cat_name]["label"] == 1
audit_results.append(
{
"category": cat_name,
"total": n_cat,
"removed": int(n_removed),
"pct_removed": pct_removed,
"expected_removal": "Yes" if is_toxic else "No (FP)",
}
)
audit_df = pd.DataFrame(audit_results)
total_removed = corpus_filtered.sum()Filter Audit at threshold=0.383 Total removed: 792 / 2000 (39.6%) Category Total Removed % Removed Expected? ------------------------------------------------------------ clearly_toxic 200 200 100.0% Yes mildly_toxic 300 283 94.3% Yes borderline 200 102 51.0% No (FP) group_mention 300 106 35.3% No (FP) counter_speech 200 101 50.5% No (FP) clearly_benign 800 0 0.0% No (FP)
This audit table is what every practitioner should examine before deploying a filter at scale. The expected toxic categories should have high removal rates; the non-toxic categories should have low ones. When categories like counter_speech and group_mention show higher removal rates, that is the concrete signal to investigate whether the filter is properly calibrated for your corpus, or whether the threshold needs to be raised to reduce disparate impact at the cost of some recall.
Key Parameters
The key parameters for a toxicity filtering pipeline are:
- Classifier architecture: BERT-based classifiers like Detoxify or LLaMA Guard capture semantic toxicity better than shallow models, at higher computational cost. For very large corpora, a two-stage pipeline (keyword pre-filter followed by neural classifier) reduces cost while preserving recall.
- Filtering threshold (): Values around 0.5-0.7 are typical starting points; calibrate to your precision-recall requirements using a labeled validation set. The -optimal threshold gives a systematic starting point, but manual review of filtered content should always follow.
- Beta for threshold selection: weights recall twice as heavily as precision, appropriate when missing toxic content is the primary concern. Adjust based on your application's sensitivity to over-filtering versus under-filtering.
- Attribute selection: Decide which toxicity attributes (severe toxicity, identity attacks, threats, obscenity) to filter independently versus jointly. Combined OR filtering compounds false positive rates; a meta-classifier on attribute scores is more precise.
- Audit frequency: Re-audit filter performance whenever the corpus changes substantially, when the filter is applied to a new language or domain, or when community norms shift in ways that may affect classifier calibration.