Toxicity Filtering › Toxicity Classifiers

The primary tool for toxicity filtering at scale is a learned classifier. Rule-based approaches, such as blocklists of slurs and hate speech terms, are too brittle. They fail on creative spellings, miss implicit forms entirely, and produce high false-positive rates on educational and journalistic content that discusses toxic language without endorsing it. The word "slur" is not itself a slur; blocking it damages content that discusses and critiques slurs. Learned classifiers, with all their limitations, are more flexible.

The most widely used toxicity classifier in the research literature is Google's Perspective API, developed by Jigsaw. Perspective uses a convolutional neural network trained on millions of comments labeled by human raters for attributes including toxicity, severe toxicity, obscenity, threat, insult, and identity attack. The model outputs a probability score for each attribute, and practitioners apply a threshold to decide what to filter.

Perspective was a landmark contribution because it demonstrated that classifier-based toxicity detection at scale was feasible and that the outputs were meaningful enough to support deployment. The API handles millions of requests per day and underpins moderation systems for major platforms. But it also revealed the core limitation of classifier-based approaches: the model's notion of toxicity reflects the values and blind spots of the annotators who labeled the training data.

Research published by Dixon et al. (2018) and subsequent work showed that Perspective assigned higher toxicity scores to text mentioning Black people, Muslim people, LGBTQ+ people, and women, even when the sentiment was neutral or positive. A sentence like "I am a gay man" received a higher toxicity score than "I am a straight man." A sentence like "She is a strong Black woman" scored higher than "She is a strong white woman." This bias arose because the training data contained more toxic speech targeting these groups, and the model learned superficial correlations with the group labels rather than the presence of harm. The classifier essentially learned "text that mentions marginalized groups is more likely to be toxic" from its training set, which reflected a real statistical pattern in the data, but that correlation does not hold for the vast majority of documents that simply discuss or mention those groups.

More recent classifiers have addressed some of these issues through better annotation design, diverse annotator pools, and adversarial training. The Detoxify library, for example, trains on the Jigsaw dataset with added robustness against identity bias, using a technique called "unintended bias mitigation" that explicitly penalizes differential false positive rates across groups. Meta's LLaMA Guard and similar models are trained specifically on safety taxonomies aligned with model deployment requirements, incorporating constitutional AI principles alongside empirical annotation. For training data filtering, you typically want a model optimized for recall (catching most toxic content) at the cost of precision (accepting some false positives), because the downstream cost of a missed toxic example is higher than the cost of losing one acceptable document.

Three broad architectural choices are used in practice, each with distinct tradeoffs:

Keyword-based models maintain lists of toxic terms and patterns. They are fast, interpretable, and easy to update, but they suffer from high false-positive rates and can be circumvented by creative misspellings, code words, or dog whistles. Writing "b1ack people" instead of "Black people" to evade a filter is a trivial evasion that does not require any sophistication. These models also struggle with context: the word "kill" appears in countless benign contexts ("kill the lights," "kill it at the presentation"), and a simple blocklist cannot distinguish them. Keyword models work well as a first-stage filter before applying a learned model, where their role is to quickly remove obvious content and reduce the load on the more expensive classifier.

Shallow ML classifiers (logistic regression, linear SVM on n-gram features) can be trained quickly on moderate labeled datasets and produce reasonably calibrated probabilities. Their main weakness is that they miss semantic toxicity: "I hope you get cancer" contains no explicit slurs but is clearly a threat. A logistic regression over unigrams will not learn this mapping without seeing many similar examples labeled explicitly, and it will not generalize to paraphrases. Bag-of-words features also ignore word order, which matters for distinguishing "The teacher killed the student's grade" from more violent phrasings.

Neural classifiers (fine-tuned BERT, RoBERTa, or purpose-built architectures) can capture semantic and contextual signals. A sentence like "You're so bright you must be a black hole" is benign, while the same sentence with an offensive substitution is toxic, and a neural model trained on enough examples can learn this distinction because it encodes the sentence as a whole rather than a bag of features. The tradeoff is computational cost, which matters when you are classifying hundreds of millions of documents. A single Detoxify inference call on a CPU takes on the order of milliseconds; at a billion documents, the cumulative cost is substantial.

For web-scale filtering, the common approach is a two-stage pipeline: fast keyword pre-filtering removes obvious cases, then a neural classifier processes the remaining documents. This reduces the expensive classifier calls by a large factor while preserving recall on the hard cases. A keyword filter might remove 30-50% of the most obvious toxic content in a single pass, leaving the neural classifier to handle context-dependent cases. The two-stage architecture also makes the filtering pipeline auditable: you can inspect what the keyword filter removed separately from what the neural classifier removed, which is useful for debugging and bias analysis.

Off-the-shelf classifiers like Perspective and Detoxify are trained on general web data, primarily English. If your corpus is multilingual, domain-specific, or from a particular time period, you will likely need to fine-tune or train a custom classifier. A classifier trained on English Reddit comments will perform poorly on Arabic web forums because of language differences and because the types and styles of toxicity differ across cultural contexts.

The key ingredients for a custom classifier are:

• Labeled examples: Both toxic and non-toxic. You need enough toxic examples to cover the specific types of harm you are targeting, and enough non-toxic examples that closely resemble toxic ones (to prevent false positives) to teach the model the boundary.
• Annotator guidelines: Explicit criteria about what counts as toxic, including worked examples of edge cases. Ambiguous cases should be adjudicated and documented, not just discarded. The documentation of adjudicated cases becomes a resource for understanding and updating the classifier's coverage.
• Calibration data: Examples specifically designed to test for false positives on content that superficially resembles toxic text, including counter-speech, academic discussion, fiction, and journalistic reporting. If your calibration data does not include these categories, your classifier may fail silently on exactly the content types where false positives are most costly.

One recurring finding in the literature is that annotation disagreement is high for toxicity. When multiple annotators label the same text, they often disagree, especially on borderline cases. This disagreement is informative in two ways. First, it marks the ambiguous region of the distribution, which is precisely where your threshold decisions matter most. Second, it signals how annotator demographics affect labeling: annotators from marginalized communities often assess text differently from majority-group annotators for the same content.

Rather than discarding disagreement, some approaches train with soft labels (the average annotation score across raters) or with uncertainty estimates that allow the model to express low confidence on ambiguous examples. Soft-label training produces models that assign intermediate probabilities to borderline content, which is exactly what you want when making threshold decisions: you want your model to be uncertain about uncertain cases, not overconfidently assign them to one class.

Inter-annotator agreement is typically measured with Cohen's kappa ($\kappa$) or Krippendorff's alpha ($\alpha$). The formulas differ but both measure agreement above chance. For Cohen's kappa between two annotators:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

where:

• $p_o$: observed agreement (fraction of examples both annotators labeled identically)
• $p_e$: expected agreement by chance, computed from the marginal label frequencies

For toxicity annotation, kappa values around 0.6-0.7 are considered acceptable, indicating substantial agreement. Values below 0.5 signal that the annotation task is too loosely defined and the guidelines need refinement. When kappa is low, the resulting model inherits the inconsistency of the annotation: similar documents may receive very different scores depending on which annotators labeled which documents, and the classifier's behavior becomes less predictable in exactly the region where you most need predictability.

One practical consideration when building a custom classifier is the need for hard negative mining. A classifier trained with randomly sampled negative examples (non-toxic documents from the web) will learn an easy boundary because the random negatives are clearly benign. At deployment time, the hard cases, the counter-speech, the academic discussion, the dark humor, all receive scores higher than the training negatives warranted, because the model has never seen negatives that look like those. This causes higher false positive rates in production than in evaluation. Hard negative mining, the deliberate inclusion of difficult non-toxic examples in training, is essential for closing this gap. Common sources of hard negatives include news articles about hate speech, academic papers on extremism, dictionaries and educational resources about slurs, court documents describing hate crimes, and fiction with morally complex characters.

The amount of hard negatives needed depends on the classifier architecture and the distribution of the corpus you are filtering. A practical approach is to run your trained classifier on a held-out validation set, identify the false positives with the highest toxicity scores, analyze their content to understand what class of hard negative is missing, and add representative examples of that class to the training set. Repeat until the false positive rate on the validation set is acceptable.