Toxicity Filtering › What Toxicity Means in This Context

Before you can filter toxic content, you need a working definition. In natural language, "toxic" covers a wide range of unpleasant content, from mildly rude comments to calls for genocide. For training data curation, you need a definition precise enough to guide annotation decisions and evaluate classifiers. In the context of pretraining data, toxicity generally refers to content that:

• Promotes or glorifies harm against people based on protected characteristics (race, religion, gender, sexual orientation, national origin, disability)
• Contains explicit threats, targeted harassment, or personal attacks
• Includes graphic depictions of sexual violence or child sexual abuse material (CSAM)
• Advocates for genocide, terrorism, or mass violence
• Dehumanizes individuals or groups through slurs, comparisons to vermin, or eliminationist language

Notice what this definition excludes. Adult content between consenting parties may be appropriate for some applications and not others, but it is not inherently toxic in the sense used here. Discussion of historical atrocities is not toxic even if it describes terrible events in detail. Fiction depicting villains is not toxic even when the villain espouses reprehensible views. A news article reporting on a hate crime may quote the perpetrator's language directly, and that direct quotation serves an important informational purpose. These distinctions are critical because naive toxicity classifiers frequently fail to make them, and we will return to this problem at length when we discuss over-filtering risks.

The practical framing that most practitioners adopt is that toxicity filtering targets content that, if the model memorized and reproduced it verbatim, would constitute a failure mode. This gives you a concrete test: would a reasonable person be alarmed if this exact text appeared in your model's output? If yes, consider filtering it from training. If the text describes harm in order to analyze or condemn it, and a reasonable person would understand the analytical framing, it probably belongs in your training corpus.

This framing also clarifies the relationship between toxicity and quality. Toxic content is not low quality in the conventional sense: a well-written manifesto calling for ethnic cleansing is far more dangerous as training material than a low-quality but benign spam message. Quality filters improve the information density of the corpus; toxicity filters shape the values and behavioral tendencies the model learns. You need both, and they are largely independent.

Defining exactly what counts as toxic is harder than it looks, and the difficulty has real engineering consequences. Consider these four texts:

1. "I hate [group]. They should be driven out."
2. "Surveys show that 45% of respondents expressed hostile attitudes toward [group]."
3. "[Group] people are lazy and untrustworthy." (an implicit stereotype without explicit threat)
4. "The claim that [group] people are lazy is a racist myth that studies consistently refute."

Text 1 is clearly toxic. Text 4 is clearly not. Texts 2 and 3 are harder. Text 2 is a factual statement about a social phenomenon; it is exactly the kind of evidence-based discussion you want in a training corpus that will be used to build an informed model. Text 3 contains no slurs, no threats, and no explicit calls for harm, but it promotes a harmful stereotype that has real-world consequences. Whether you filter text 3 depends on your application and your threshold.

The taxonomy problem compounds when you move to multi-class classification. Jigsaw's Perspective API taxonomy distinguishes between general toxicity, severe toxicity, obscenity, threats, insults, and identity attacks. Each of these categories has its own annotation guidelines, edge cases, and borderline instances, and they are not mutually exclusive. A single comment may score high on multiple attributes, and the interaction between attributes matters when you are making a filtering decision. We will return to multi-attribute filtering in the threshold section.