Toxicity Filtering › Toxicity Thresholds

A toxicity classifier produces a score between 0 and 1. To make a filtering decision, you must choose a threshold above which a document is considered toxic and removed from the corpus. The threshold is a technical choice and a values decision, with measurable consequences for what your model learns and what communities are represented in the training data.

At high thresholds (close to 1.0), you filter only text that the classifier considers almost certainly toxic. Precision is high: most filtered documents are problematic. Recall is low: many toxic documents that received scores between your threshold and 1.0 remain in the corpus.

At low thresholds (close to 0.0), you filter everything the classifier assigns any toxicity signal to. Recall is high, but precision collapses: you remove many acceptable documents.

For training data, the asymmetry matters in both directions. Missing toxic content (low recall) means the model learns from it. Filtering too aggressively (low precision) removes legitimate content and skews the training distribution. Research on this tradeoff uses a formal framing: given a toxicity classifier with score $s(d)$ for document $d$ and a threshold $\tau$, we filter documents where:

$$s(d) \geq \tau$$

where:

• $s(d)$: the classifier's toxicity score for document $d$, between 0 and 1
• $\tau$: the filtering threshold, a value in $[0, 1]$ chosen by the practitioner
• Documents with $s(d) \geq \tau$ are removed from the training set

The expected precision $P(\tau)$ and recall $R(\tau)$ as functions of threshold are:

$$P(\tau) = \frac{\text{true toxics filtered at } \tau}{\text{all documents filtered at } \tau}$$ $$R(\tau) = \frac{\text{true toxics filtered at } \tau}{\text{all true toxics in corpus}}$$

These quantities cannot be computed without ground truth labels for the full corpus, which is why calibrating a threshold requires a held-out labeled validation set that mirrors the distribution of your corpus. A common mistake is calibrating thresholds against a dataset of known toxic examples, which will give you optimistic recall estimates but tell you nothing about precision or false positive rates on benign content.

To pick a threshold, you need a validation set with ground truth labels. This set should be:

• Representative of the corpus you are filtering, including more than high-confidence toxic examples. The hard cases, the borderline documents and the sensitive-but-benign documents, should be proportionally represented.
• Balanced enough to allow measurement at both ends of the precision-recall curve. An all-toxic validation set cannot measure precision. An all-benign validation set cannot measure recall. You need a mix that reflects your production distribution.
• Annotated by multiple raters with adjudicated decisions for borderline cases. Single-annotator labels for borderline cases are unreliable and will introduce noise into your threshold selection.

With a validation set in hand, you plot the precision-recall curve across all threshold values from 0 to 1 and identify the operating point that satisfies your requirements. A common approach is to choose $\tau$ to maximize the $F_\beta$ score, which is a weighted harmonic mean of precision and recall:

$$F_\beta = (1 + \beta^2) \cdot \frac{P \cdot R}{\beta^2 \cdot P + R}$$

where:

• $\beta > 1$: weights recall more heavily, appropriate when missing toxic content is more costly than over-filtering
• $\beta < 1$: weights precision more heavily, appropriate when over-filtering is more costly
• $\beta = 1$: equal weighting (the standard $F_1$ score)

For most pretraining data curation pipelines, practitioners set $\beta > 1$ (commonly $\beta = 2$) because the cost of training on toxic content is considered higher than the cost of losing some acceptable documents. The right $\beta$ depends on your application. A model intended for use with vulnerable populations (mental health support, crisis intervention) warrants a very high $\beta$, accepting aggressive over-filtering to minimize exposure to harmful content. A model intended for security research that must understand adversarial content may warrant a lower $\beta$.

The $F_\beta$ approach gives you a single operating point, but often you want to see the full curve before committing. A practical approach is to plot the precision-recall curve, mark several candidate thresholds at different recall levels (e.g., "90% recall threshold" and "95% recall threshold"), and then examine a random sample of filtered documents at each threshold to assess whether the additional filtered content at the more aggressive threshold is problematic or benign.

A single toxicity score is often insufficient. Jigsaw's taxonomy distinguishes between:

• Toxic: A rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion
• Severe toxic: A hateful or threatening comment that calls for violence or demeans entire groups
• Obscene: Contains explicit language (profanity, sexual language)
• Threat: A direct or indirect threat of physical harm
• Insult: An insulting, inflammatory, or negative comment toward a person
• Identity attack: Negative or hateful language targeting a person's identity characteristics

You may want different thresholds for different attributes. A corpus intended for a children's educational assistant might filter at a very low threshold for obscenity even while allowing discussion of historical violence. A corpus for a medical assistant might allow clinical discussion of abuse without filtering it as toxicity, but filter any glorification of violence at a low threshold. A corpus for a legal research model might need to retain verbatim quotes from hate speech cases that would otherwise score high on identity attack.

The practical consequence of multi-attribute filtering is that you must maintain multiple classifiers or a multi-output classifier, and you must decide how to combine signals from multiple attributes. A common approach is to filter if any attribute exceeds its threshold (logical OR), which increases recall at the cost of precision. Alternatively, you can filter only if a primary attribute exceeds its threshold, and use secondary attributes to flag borderline cases for manual review.

The combination logic matters enormously at scale. If you use logical OR across six attributes each with a 10% false positive rate, the combined false positive rate for a benign document approaches:

$$1 - (1 - 0.10)^6 \approx 0.47$$

Nearly half of all benign documents would be filtered, even with individually well-calibrated classifiers. This calculation assumes attribute independence, which overstates the false positive rate when attributes are positively correlated (a document that scores high on "obscene" is likely to score somewhat higher on "insult" as well, so the effective number of independent classifiers is less than six). Practitioners often address this by treating the six Jigsaw attributes not as six independent filters but as a single multi-output model with a combined decision rule that accounts for correlations between attributes, typically by learning a logistic regression meta-classifier on top of the six attribute scores.