Toxicity Filtering › Toxicity Filtering

When you build a language model on web-scale data, you are training on a cross-section of human expression that includes encyclopedias, news articles, hate speech, violent content, graphic descriptions, and targeted harassment. Left unfiltered, this material embeds deeply into model weights. Models trained on toxic content reproduce it, amplify it, and sometimes generate it unprompted in polite conversation. Toxicity filtering is the process of identifying and removing this material from your training corpus before the model ever sees it.

This chapter builds on the quality filtering techniques we discussed in the previous chapter, which focused on removing low-quality, malformed, or uninformative text. Toxicity filtering addresses a different dimension: content that is structurally fine but normatively harmful. A document about white supremacist ideology may be grammatically perfect, richly formatted, and informationally dense. Quality filters will not catch it. Toxicity filters must. The distinction matters because the two types of filtering require different tools, different thresholds, and different tradeoffs.

The core challenge is not building a classifier that detects obvious slurs. Any simple heuristic can catch the most explicit content. The hard problem is balancing two competing errors: missing toxic content that degrades your model versus over-filtering legitimate text and inadvertently erasing minority voices, discussions of historical violence, or literary content that depicts harm in order to critique it. Both errors are costly, and the right balance depends on your application, your user population, and your values as a practitioner. This chapter works through both sides of that balance in detail, covering the classifiers you will use, the threshold decisions you will face, the over-filtering risks you must monitor, and how to evaluate a complete filtering pipeline.