Quality Filtering › Quality Filtering

When you scrape the web for training data, you collect far more than clean, coherent text. You get spam, SEO-stuffed keyword lists, machine-translated garbage, duplicate boilerplate, pornographic content, malware descriptions, and page after page of raw HTML fragments that escaped the parser. Even after deduplication (covered in the previous chapter), a substantial fraction of what remains is low-quality text that would hurt model performance if included. Quality filtering is the stage where you decide what belongs in a training corpus.

The core challenge is defining "quality" for a language model. It is not the same as "grammatically correct" or "well-written by human standards." Training data quality means text that teaches the model useful patterns: coherent sentence structure, meaningful word co-occurrences, factual statements, domain-appropriate vocabulary, and the kind of reasoning chains you want the model to reproduce. A Reddit comment with spelling errors can be higher quality than a perfectly formatted but semantically empty spam article. This distinction separates data curation from proofreading.

Quality filtering encompasses several approaches that operate at different levels of precision and cost. Heuristic filters apply simple, cheap rules that catch obvious garbage. Perplexity filtering uses a reference language model to score how natural the text is relative to known-good content. Classifier-based filtering trains a model to distinguish quality text from junk. Each method has its place in a production pipeline, and serious efforts like C4, The Pile, and RefinedWeb use combinations of all three. This chapter covers each approach in depth, including how to set filter thresholds without accidentally discarding too much good data or biasing the corpus against minority linguistic communities.

Understanding quality filtering matters because it can strongly affect any pretraining pipeline. A 10% improvement in filtering precision can translate to models that need substantially less compute to reach a given performance level. The "less is more" phenomenon, where a smaller but cleaner corpus outperforms a much larger noisier one, has been replicated across many independent training runs.