Quality Filtering › Why Quality Matters for LLM Training

Before diving into the methods, it helps to understand the stakes. Language models learn by predicting the next token in a sequence. Every piece of text in the training corpus implicitly teaches the model patterns. If that text is incoherent, the model learns incoherent patterns. If it consists of keyword repetitions, the model learns to repeat keywords. The relationship between data quality and model quality is tight, and the mechanisms are instructive.

Consider what happens when a model trains on a spam article that repeats the phrase "cheap insurance online" forty times. The model updates its weights to assign high probability to the word "online" following "cheap insurance", and high probability to "cheap" following "insurance". These updates compete with updates from legitimate text that use the same words in different, more meaningful contexts. Garbage data is not neutral: it actively corrupts the statistical patterns the model is trying to learn.

The Gopher paper from DeepMind demonstrated this clearly by ablating their quality filters and measuring perplexity on held-out benchmarks. Models trained on unfiltered Common Crawl data performed substantially worse than models trained on filtered versions of the same underlying source. More dramatically, training on a smaller but higher-quality corpus often outperforms training on a much larger but lower-quality one. The Phi-1 paper pushed this principle to an extreme by showing that a 1.3B parameter model trained on "textbook-quality" filtered and synthetic data achieved performance competitive with models trained on ten times more conventional web data.

Practically, the pipeline looks like this: web scraping and HTML extraction produce raw text. Deduplication removes redundant content. Quality filtering removes low-value content from what remains. What survives goes into the training corpus. The filtering stage is where you make the hardest judgment calls, because every filter has a precision-recall tradeoff. A strict filter removes more garbage but also more legitimate text. A permissive filter preserves edge cases but lets through noise. There is no setting that eliminates this tradeoff: you are always choosing a point on a curve, not escaping it.

The stakes scale with corpus size. When you process a trillion tokens, even a 1% false negative rate means ten billion tokens of garbage in the training set. When you process ten petabytes of raw web text, even a 0.1% false positive rate discards ten terabytes of legitimate content. At these scales, decisions about filter design that seem like engineering choices have material effects on what the resulting model knows, how well it handles different communities and dialects, and what failure modes it exhibits in production.