Quality Filtering: Heuristics, Perplexity, and Classifiers

When you scrape the web for training data, you collect far more than clean, coherent text. You get spam, SEO-stuffed keyword lists, machine-translated garbage, duplicate boilerplate, pornographic content, malware descriptions, and page after page of raw HTML fragments that escaped the parser. Even after deduplication (covered in the previous chapter), a substantial fraction of what remains is low-quality text that would hurt model performance if included. Quality filtering is the stage where you decide what belongs in a training corpus.

The core challenge is defining "quality" for a language model. It is not the same as "grammatically correct" or "well-written by human standards." Training data quality means text that teaches the model useful patterns: coherent sentence structure, meaningful word co-occurrences, factual statements, domain-appropriate vocabulary, and the kind of reasoning chains you want the model to reproduce. A Reddit comment with spelling errors can be higher quality than a perfectly formatted but semantically empty spam article. This distinction separates data curation from proofreading.

Quality filtering encompasses several approaches that operate at different levels of precision and cost. Heuristic filters apply simple, cheap rules that catch obvious garbage. Perplexity filtering uses a reference language model to score how natural the text is relative to known-good content. Classifier-based filtering trains a model to distinguish quality text from junk. Each method has its place in a production pipeline, and serious efforts like C4, The Pile, and RefinedWeb use combinations of all three. This chapter covers each approach in depth, including how to set filter thresholds without accidentally discarding too much good data or biasing the corpus against minority linguistic communities.

Understanding quality filtering matters because it can strongly affect any pretraining pipeline. A 10% improvement in filtering precision can translate to models that need substantially less compute to reach a given performance level. The "less is more" phenomenon, where a smaller but cleaner corpus outperforms a much larger noisier one, has been replicated across many independent training runs.

Before diving into the methods, it helps to understand the stakes. Language models learn by predicting the next token in a sequence. Every piece of text in the training corpus implicitly teaches the model patterns. If that text is incoherent, the model learns incoherent patterns. If it consists of keyword repetitions, the model learns to repeat keywords. The relationship between data quality and model quality is tight, and the mechanisms are instructive.

Consider what happens when a model trains on a spam article that repeats the phrase "cheap insurance online" forty times. The model updates its weights to assign high probability to the word "online" following "cheap insurance", and high probability to "cheap" following "insurance". These updates compete with updates from legitimate text that use the same words in different, more meaningful contexts. Garbage data is not neutral: it actively corrupts the statistical patterns the model is trying to learn.

The Gopher paper from DeepMind demonstrated this clearly by ablating their quality filters and measuring perplexity on held-out benchmarks. Models trained on unfiltered Common Crawl data performed substantially worse than models trained on filtered versions of the same underlying source. More dramatically, training on a smaller but higher-quality corpus often outperforms training on a much larger but lower-quality one. The Phi-1 paper pushed this principle to an extreme by showing that a 1.3B parameter model trained on "textbook-quality" filtered and synthetic data achieved performance competitive with models trained on ten times more conventional web data.

Practically, the pipeline looks like this: web scraping and HTML extraction produce raw text. Deduplication removes redundant content. Quality filtering removes low-value content from what remains. What survives goes into the training corpus. The filtering stage is where you make the hardest judgment calls, because every filter has a precision-recall tradeoff. A strict filter removes more garbage but also more legitimate text. A permissive filter preserves edge cases but lets through noise. There is no setting that eliminates this tradeoff: you are always choosing a point on a curve, not escaping it.

The stakes scale with corpus size. When you process a trillion tokens, even a 1% false negative rate means ten billion tokens of garbage in the training set. When you process ten petabytes of raw web text, even a 0.1% false positive rate discards ten terabytes of legitimate content. At these scales, decisions about filter design that seem like engineering choices have material effects on what the resulting model knows, how well it handles different communities and dialects, and what failure modes it exhibits in production.

Heuristic filtering is the first line of defense. These filters encode explicit rules about what bad text looks like. They run fast (often in microseconds per document), scale to petabytes, and catch the most obvious problems without requiring any model inference. The downside is that heuristics are brittle: they fail on text that looks superficially clean but is semantically poor, and they may incorrectly discard legitimate text that happens to match the wrong pattern.

The insight behind heuristic filters is that low-quality web content has characteristic surface properties that distinguish it from natural prose. Spam articles repeat keywords. Navigation menus have very short lines. Scraped template content has identical repeated blocks. Pages that failed HTML parsing have high ratios of angle brackets and escaped entities. Rather than modeling these problems semantically, heuristics directly measure the surface properties that correlate with low quality.

The heuristic filters used in production pipelines like C4, MassiveText, and RefinedWeb fall into several categories:

• Length-based filters: Remove documents that are too short (fewer than 200 words, for instance) or too long. Short documents provide too little context for the model to learn useful patterns. Extremely long documents are sometimes artifacts of parsing failures where many separate pages got concatenated.
• Punctuation and symbol ratio filters: Flag documents where a large fraction of characters are symbols, numbers, or non-letter characters. A document that is 30% curly braces is almost certainly code or template junk, not natural language.
• Alphabet coverage filters: Remove documents where most content is non-alphabetic or where the text comes from an unexpected script. This is also used for language identification when language-specific filtering is required.
• Bullet point or line-length filters: Documents composed almost entirely of short, newline-delimited items (like site navigation menus or keyword lists) tend to have poor narrative structure. The ratio of lines to words can identify these.
• Boilerplate detection: Many web pages share identical footers, cookie banners, and privacy policy text. Hard-coded string matching can remove pages where more than a fixed percentage of the text matches known boilerplate phrases. This complements deduplication, which operates at the document level, by catching pages where boilerplate comprises a large fraction of otherwise-unique content.
• Bad word lists: Some pipelines maintain lists of profanity, spam trigger words, or adult content markers. Any document containing words from these lists is flagged or removed. The C4 pipeline famously uses the "List of Dirty, Naughty, Obscene, or Otherwise Bad Words" maintained on GitHub.
• Repetition filters: A document that repeats the same sentence or paragraph is almost certainly low-quality. You can detect this by checking whether any n-gram above a certain length occurs more than once in the document, or by measuring the character-level entropy of the content. Low entropy means low informational diversity.

Each rule has an explicit failure mode. Length filters discard short but high-quality documents like dictionary definitions or aphorisms. Symbol ratio filters may incorrectly remove technical documentation with code snippets. Bad word lists may disproportionately remove text from communities where that vocabulary appears in legitimate cultural or literary contexts. Understanding these failure modes is not academic: it determines which heuristics to apply, which to skip for a given domain, and how to calibrate thresholds.

Let's look at what these filters detect and why they work.

The clean news article scores low on all three metrics. The keyword spam shows a very high word repetition ratio because the same words ("insurance") dominate the vocabulary after stopword removal. The template document is flagged by the duplicate line ratio. The symbol-heavy code fragment is caught by the symbol ratio.

These thresholds are tunable. If you set the symbol ratio cutoff at 0.3, you reject documents where more than 30% of characters are symbols. If you set the word repetition threshold at 0.4, you reject documents where the top 5 words account for more than 40% of non-stopword content. The right thresholds depend on your domain: code corpora need higher symbol ratio thresholds, while technical documentation might legitimately use more repetitive terminology than general prose.

The C4 dataset (used to train T5 and mT5) is one of the most influential filtered corpora, and its heuristic rules have become a baseline for many subsequent efforts. The C4 processing pipeline starts with Common Crawl's WET files (pre-extracted text), then applies the following filters. Lines are discarded if they:

• Do not end with terminal punctuation (a period, question mark, or exclamation point)
• Contain fewer than 3 words or more than 100,000 characters
• Contain the string "javascript" (to catch pages where the parser captured script blocks)
• Contain the string "lorem ipsum" (to catch placeholder content)
• Contain any words from the "List of Dirty, Naughty, Obscene, or Otherwise Bad Words"

Documents are then removed if fewer than 5 sentences remain after line filtering, or if they contain the phrase "lorem ipsum". The entire pipeline removed roughly 70% of raw Common Crawl documents, producing a cleaner but still large 750 GB corpus.

Later analyses questioned some of these choices and their consequences. The terminal punctuation filter disproportionately removes non-English text where punctuation conventions differ: many East Asian languages do not end sentences with a period. The JavaScript filter turned out to remove legitimate pages that happened to discuss JavaScript programming. The bad word filter significantly underrepresented content from communities that use that vocabulary in legitimate cultural, literary, or conversational contexts.

These are typical failure modes of heuristic filters: rules designed around one language or content type, or calibrated on a particular demographic's writing, can be inappropriate for others. Every rule encodes an assumption about what normal text looks like, and those assumptions tend to reflect the dominant culture and language of whoever designed the rules.

One less-obvious but useful heuristic is character-level entropy. Informative text has high character diversity: it uses a wide variety of words and transitions. Repetitive spam or boilerplate has low entropy because the same short n-grams repeat constantly.

Given a document with $N$ characters, the empirical character unigram distribution is $p_c = \text{count}(c) / N$ for each character $c$. The character-level entropy is:

where the sum runs over all distinct characters appearing in the document. Well-written prose in English typically has character entropy between 4.5 and 5.2 bits. Repetitive spam with limited vocabulary falls below 4.0. Compressed or encoded binary data exceeds 6.0. Setting a window of acceptable entropy values gives you a fast filter that catches a class of garbage that other rules miss.

The entropy measure is also language-aware in a useful way: it naturally adapts to the expected character distribution of the target language. Chinese text will have different typical entropy values than English, but the threshold "reject documents below the 5th percentile of entropy for this language" works without needing separate calibration.

Heuristic filters catch obvious garbage, but they cannot assess whether text is coherent and natural in a linguistic sense. A document can pass all heuristic checks and still be semantically nonsensical, machine-translated in a stilted way, or composed of grammatically valid but meaningless phrases. Perplexity filtering addresses this by scoring how well a reference language model can predict the text.

The key insight behind perplexity filtering is this: a language model trained on high-quality text assigns low perplexity to text that resembles its training distribution and high perplexity to text that does not. Coherent English prose gets low perplexity from a model trained on books or Wikipedia. Garbled machine translation, keyword soup, or foreign language text gets high perplexity.

This is powerful because it captures something heuristics cannot: the sequential coherence of language. A document can have perfect punctuation, normal symbol ratios, and varied vocabulary, but if the word sequences are unnatural (as happens with low-quality machine translation or automated content generation), a language model will find it surprising. The perplexity signal captures that surprise.

You train or obtain a small reference language model on known-good text. Wikipedia, books corpora, or curated news articles work well. Then you score each candidate document by running it through the reference model and measuring the average perplexity. Documents above a threshold are discarded.

The reference model does not need to be large. In practice, character-level n-gram models (KenLM) or small neural models work well for this purpose. The Gopher paper and CCNet pipeline both use KenLM models trained on Wikipedia for perplexity scoring. KenLM implements a modified Kneser-Ney smoothed n-gram language model stored in a compact hash structure. It is extremely fast: scoring billions of words takes hours, not days. For a trillion-token pipeline, KenLM's throughput is essential; running a multi-billion-parameter neural model at scoring time would be prohibitively expensive.

The choice of reference corpus matters enormously. A KenLM model trained on Wikipedia learns the patterns of encyclopedic English: neutral tone, third-person perspective, factual claims, formal vocabulary. It will assign high perplexity to legal documents, poetry, informal dialogue, scientific papers, and code, not because these are low quality, but because they differ from encyclopedic style. If you want all these content types in your training corpus, you need either multiple domain-specific reference models or a permissive threshold that only removes the most extreme outliers.

Let's demonstrate the concept using a simple n-gram language model, which captures the same core intuition as KenLM-based filtering.

The perplexity values separate the candidates along the expected lines. Coherent prose that matches the training distribution (science news, economic research sentences) gets low perplexity. Keyword spam and garbled text get high perplexity because their word sequences are unpredictable given the model's learned patterns. The HTML artifact scores high because isolated tag names and attributes have no sequential predictability from the reference model's perspective.

Notice that this works without any explicit rules about what constitutes "good" text. The n-gram model does not know that "buy cheap loans" is spam. It knows that those three words appearing together is unexpected given the contexts it has seen during training on science and technology sentences. Perplexity filtering is powerful precisely because it generalizes beyond any fixed rule set.

The CCNet pipeline (used for XLM-R and LLaMA pretraining data) filters documents by perplexity into three buckets:

• Head: Bottom 30th percentile by perplexity (highest quality text, closest to the reference distribution)
• Middle: 30th to 65th percentile
• Tail: Top 35th percentile by perplexity (lowest quality, often excluded entirely)

Rather than a fixed absolute threshold, CCNet uses percentile-based filtering. This is more robust because absolute perplexity values depend heavily on the reference model and the tokenization. A threshold of "reject documents with perplexity above 500" that works for English may be completely wrong for another language if you use the same model. Absolute thresholds also shift if you retrain the reference model with different hyperparameters. Percentile thresholds are stable across these variations because they are defined relative to the corpus distribution.

One important caveat is that perplexity filters can be too aggressive about domain specificity. A language model trained on Wikipedia will assign high perplexity to legal documents, poetry, and code, even though these are perfectly valid forms of text. If you want to include diverse domains in your training corpus, you need either domain-specific reference models or a more permissive threshold that only removes the truly extreme outliers. CCNet handles this partially by training separate language models for each language, but within a language the domain bias persists.

A second caveat is that perplexity filtering is circular in a subtle way. If you train your reference model on text that excludes certain dialects or styles, those dialects get high perplexity scores and are removed from the corpus. The resulting training corpus underrepresents those styles, producing a final model that performs poorly on them. The reference model's biases become the corpus's biases. This is distinct from heuristic bias (where a rule is explicitly wrong) but equally consequential.

The overlapping distributions reveal why perplexity thresholds are imperfect. The tails of the high-quality and medium-quality distributions overlap substantially. A threshold that catches most low-quality documents will inevitably discard some legitimate high-quality text. This overlap is the core precision-recall tradeoff in filtering. It is not a failure of the method: the task itself is ambiguous. Some text lies on the quality boundary, and no scalar metric will cleanly separate it. This is why production pipelines do not rely on perplexity filtering alone.

One practical subtlety is that perplexity scores depend on tokenization. A character-level model, a word-level model, and a BPE model trained on the same corpus will assign different perplexity values to the same document, even if all three models have similar quality. This makes it difficult to compare perplexity thresholds across different reference model implementations.

The CCNet approach of using percentile thresholds partially mitigates this: because the percentile is computed over the same corpus with the same model, relative rankings are stable even if absolute values vary. If you change the reference model (for example, updating from a 3-gram to a 5-gram KenLM model), you need to recompute the threshold percentiles on a representative sample rather than reusing old absolute cutoffs.

Classifier-based filtering takes a more direct approach: train a binary classifier to predict whether a given document is high-quality or not. Rather than encoding rules or using indirect signals like perplexity, the classifier learns from examples of good and bad text. This makes it more flexible and often more accurate than heuristic or perplexity-based approaches, at the cost of requiring labeled training data and higher inference cost.

The classic implementation from OpenAI (used for the WebText dataset, which GPT-2 trained on) trains a logistic regression classifier on n-gram features extracted from two classes of text:

• Positive examples: Text from outbound links on Reddit that received at least 3 upvotes. The assumption is that Reddit users curate quality content by voting, so pages linked from Reddit and upvoted represent human-approved text across a wide range of topics and styles.
• Negative examples: Random Common Crawl pages.

The classifier then scores every Common Crawl document by its predicted probability of being a positive example. Documents below a threshold are discarded. This approach uses the crowdsourced curation signal embedded in Reddit voting without requiring any manual labeling by the research team.

The Reddit upvote signal is not perfect. Reddit's user base skews toward certain demographics, interests, and languages. Content that Reddit users find interesting may not be the same as content that makes the best language model training data. Despite this, the resulting WebText dataset and models trained on it showed that the Reddit-curated filtering dramatically improved quality over unfiltered Common Crawl.

The classifier trained on TF-IDF features over character n-grams is surprisingly effective for this task. Character n-grams capture low-level patterns (spelling, punctuation, character repetition) that distinguish boilerplate and spam from natural prose without requiring linguistic analysis. The character-level representation is language-agnostic and robust to vocabulary differences between the training examples and the documents being scored.

The TF-IDF weighting (term frequency times inverse document frequency) ensures that features that appear uniformly across all documents (common characters, punctuation) get down-weighted, while features that are distinctive to certain document types get up-weighted. The logistic regression then learns to combine these features into a scalar quality score.

The quality scores reflect the classifier's learned pattern: text that resembles curated web content scores high, while boilerplate, spam, and keyword-stuffed text scores low. The casual social media post sits near the boundary, where its classification is ambiguous. Whether to include informal social text depends on your training objectives: if you want the model to handle conversational queries, discarding all informal text is counterproductive.

This example uses a small training set for illustration. In production, these classifiers are trained on millions of positive and negative examples, which substantially improves their accuracy and generalization. The WebText classifier reportedly used millions of Reddit-linked pages as positives and millions of random crawl pages as negatives.

It is worth understanding why character-level n-gram features are so effective for quality classification. Character n-grams at the 2-6 gram range capture patterns like:

• Repetition of exclamation marks ("!!!!") that signal spam
• Camel-case identifiers ("getElementById") that signal code
• Copyright boilerplate sequences ("rights reserved", "© 20")
• Structural patterns of URLs (".html", "href=")
• Distinctive spacing and punctuation patterns of natural prose

These patterns are robust because they occur at a finer granularity than words. A new spam campaign can use completely different vocabulary but will still use the same exclamation-mark repetition patterns. A new programming language will still produce code with distinctive symbol combinations. The character-level representation generalizes better than word-level features across the long tail of web content.

The char_wb analyzer in scikit-learn adds word-boundary tokens to the character n-grams, which captures the shapes of individual words (short words, long words, words with unusual character combinations) without building an explicit vocabulary. This makes it much harder to game: an adversarial spam author cannot simply replace flagged words with synonyms, because the character-level patterns of their writing style remain detectable.

For production pipelines that process billions of documents, even logistic regression with sparse features can become a bottleneck. Facebook's fastText classifier offers a faster alternative. Trained on subword n-gram embeddings, fastText achieves near-linear-time inference and can be trained on large datasets in minutes.

The CCNet pipeline uses fastText for language identification, assigning each document to a language with a confidence score, and the same architecture works for quality classification. FastText learns a shallow embedding for each subword n-gram and sums them to produce document representations, which are then used for classification. The model is shallow (no deep neural layers) but captures rich character-level patterns through the n-gram vocabulary.

The RefinedWeb dataset (used to train Falcon) extends this approach with a quality classifier that incorporates multiple signals: URL patterns, document length distribution, and language model scores, combined with a neural classifier that learns to weight these features. This ensemble approach is more robust than any single signal because different signals catch different failure modes, and the neural combination learns which signal is most reliable for which type of document.

A more recent approach avoids the need for manual labeling entirely. Instead of asking "does this look like Reddit-approved content?", you can train on implicit quality signals extracted automatically:

• Educational content detection: The Phi-1 paper introduced the idea of training classifiers to detect "textbook quality" text. A GPT-4 model rated a sample of documents on educational richness, and those ratings were used to train a smaller fastText classifier that then scored the full corpus. This transfers expensive LLM judgment to cheap classifier inference.
• Model-as-teacher at scale: Use a strong language model (like GPT-4) to rate a sample of documents, then train a smaller classifier on those ratings. This allows you to scale expensive LLM ratings across billions of documents that would be too costly to rate directly.
• Consistency with known facts: Score documents by how many verifiable factual claims they contain versus unsupported assertions. Pages with many specific, checkable claims (dates, measurements, named entities) tend to be more informative than vague or opinion-heavy content.
• Citation and evidence quality: Academic-style text that cites sources and qualifies claims with evidence tends to be more reliable training data than unsupported assertions. Some pipelines use structural markers (citation patterns, hedging language like "according to" or "researchers found") as lightweight quality signals.

The "textbook quality" approach from the Phi-1 paper showed that filtering for educationally rich text and synthetically generating high-quality examples could produce capable small models from surprisingly little data. A 1.3B parameter model trained on 6B tokens of textbook-quality data matched models trained on 300B tokens of conventional web data on several coding benchmarks. This finding reinforced that the marginal value of data quality exceeds the marginal value of data quantity in many training regimes.

The fundamental constraint of quality filtering is the precision-recall tradeoff. Precision measures what fraction of the documents you keep are high quality. Recall measures what fraction of the high-quality documents you successfully retain. A perfect filter would have both precision and recall equal to 1.0, but in practice the two values trade off against each other through the threshold parameter.

When you raise the threshold (require a higher quality score to pass), precision goes up (fewer garbage documents pass) but recall goes down (more legitimate documents fail). When you lower the threshold, recall goes up but precision goes down. The right operating point on this curve depends on the relative costs of the two errors.

In data curation, the cost of low recall depends on how much data you have. If you have an enormous raw corpus, you can afford to be strict and discard ambiguous documents. If you are targeting a niche domain with limited available text, discarding borderline documents could leave you with a corpus that is too small for effective training. The cost of low precision depends on how much the garbage hurts. For general-purpose pretraining, some noise may be tolerable. For safety-critical fine-tuning, even a small fraction of problematic content can have outsized effects.

The curve illustrates the fundamental tradeoff. Requiring high precision means setting a strict threshold, which means filtering out more documents including some legitimate ones (low recall). Requiring high recall means setting a lenient threshold, accepting more low-quality documents (lower precision). The right operating point depends on how much high-quality data you have available. If you have an abundance of data, you can afford to be strict. If your target domain is data-sparse, you may need to accept more noise.

The "balanced" operating point shown at roughly equal precision and recall is not always the right choice, even when the costs of both errors seem symmetric. For language model pretraining, most practitioners find that a recall-biased operating point (say, 0.85 recall, 0.70 precision) is preferable to a precision-biased one at the same area under the curve, because the model can partially compensate for noisy training data through its own averaging during pretraining, but it cannot compensate for data it never saw.

Choosing the right threshold is as important as choosing the right filtering method. Set it too strict and you discard legitimate training data, reducing diversity and potentially introducing biases. Set it too lenient and you let through too much noise. There is no universally correct answer, but several strategies help.

The most principled approach is downstream evaluation. You vary the threshold, train (or fine-tune) a model on each filtered corpus, and measure performance on held-out benchmarks. The threshold that maximizes downstream task performance is the right one. This is expensive (it requires training multiple models) but gives ground truth. For large-scale pretraining, this approach is often impractical, but it is feasible for domain-specific fine-tuning where training is cheap.

A cheaper proxy is to monitor the acceptance rate at different thresholds. If your filter accepts 15% of documents, you might be too strict unless you have an enormous raw corpus. If it accepts 85%, you are probably not filtering enough for a general-purpose quality pipeline. Most production pipelines aim for an acceptance rate in the 20-60% range for quality-focused filtering, though this depends heavily on the quality of the source crawl. Fresh, high-quality crawls from curated seeds may have 60% acceptance rates; broad Common Crawl data might have 20-30% acceptance rates.

Percentile-based thresholds avoid the sensitivity to reference model calibration. Instead of "reject documents with perplexity > 500", you set "reject the top 30% of documents by perplexity." The percentile threshold is stable across different reference models and corpora, making it easier to reproduce and compare across pipeline iterations.

Visual inspection remains important at every stage. After setting a threshold, sample 100 accepted documents and 100 rejected documents and review them manually. You will almost always find surprises: legitimate text being rejected for unexpected reasons, or garbage that somehow passes the filter. Manual inspection catches failure modes that quantitative metrics miss. In practice, teams at Google, OpenAI, and other organizations report spending significant time reading samples from their filtered corpora before finalizing thresholds.

Stratified threshold analysis is an advanced strategy for pipelines that care about corpus diversity. Rather than setting a single acceptance rate across all domains, you set domain-specific acceptance rates. You might accept 70% of Wikipedia-style documents (because they are almost all clean) while accepting only 25% of forum content (because it has a higher noise rate). The domain labels can come from a separate classifier, URL patterns, or the language identification stage. This ensures that stricter filtering on noisy sources does not eliminate the noisier-but-legitimate domains entirely.

Let's look at how different threshold choices affect corpus composition in a concrete way.

As the threshold increases, the acceptance rate drops and the domain composition shifts away from spam and boilerplate toward news and Wikipedia content. This is exactly the desired behavior. However, note that even at the strictest threshold, some spam still passes through, and some legitimate forum content is excluded. No threshold completely separates the classes.

Notice also how the accepted corpus becomes less diverse as the threshold rises. High-threshold filtering produces a corpus concentrated in formal, structured text (news, encyclopedia entries) and underrepresents informal but legitimate content (forums, personal blogs). This can affect the model's downstream capabilities: models trained only on formal text may struggle with casual conversational queries, colloquial language, or the writing styles of communities whose text disproportionately appears in lower-scored domains.

The right panel makes the cost of aggressive filtering concrete: moving from a threshold of 0.3 to 0.7 cuts the accepted corpus to a fraction of its former size. The left panel shows what you gain and lose in detail. The gains are obvious: less spam and boilerplate. The losses are subtler: you also lose forum content that includes useful informal language patterns, colloquialisms, and practical knowledge that does not appear in formal sources. A model trained on the high-threshold corpus will produce more formal text by default and will have worse calibration on casual queries.

This visualization also reveals an important property of corpus composition: the relative fractions of domains shift even when you do not explicitly target them. A quality filter calibrated entirely on spam versus clean text will still disproportionately remove forum content because forum text happens to share surface properties with lower-quality content. If forum coverage matters for your downstream use case, you need either a domain-stratified filter or a post-filtering resampling step that restores target domain proportions.

No single filtering method is complete on its own. Production pipelines chain multiple filters in sequence, with each stage removing different failure modes. The order of stages reflects a cost-accuracy tradeoff: cheap methods run first to reduce the volume reaching expensive methods, while more sophisticated methods handle the harder cases that simple rules cannot resolve.

The typical ordering of a production filtering pipeline proceeds through four stages. Language identification runs first. Documents classified as not being in the target language are removed immediately, and this alone eliminates a large fraction of raw web data for any target language. Even for a multilingual corpus, this stage assigns language confidence scores and routes documents to language-specific downstream processing.

Heuristic filters run next because they are fast (microseconds per document) and remove obvious garbage (symbol-heavy text, duplicated lines, too-short documents) before more expensive methods are applied. A heuristic stage that processes a trillion-token corpus in a few hours might prevent the perplexity stage from having to score 30-40% of the raw documents that would have been discarded anyway.

Perplexity filtering runs after heuristics. It is slower than heuristics (requiring n-gram model inference) but faster than neural classifiers. Documents surviving heuristic filtering are scored against a KenLM model and those above a percentile threshold are removed. This stage is particularly effective at catching coherently formatted but linguistically unnatural text: machine-translated boilerplate, auto-generated content, and documents in the wrong register for the target domain.

Classifier-based filtering is the final and most expensive stage. Only documents that pass the previous stages reach the classifier. This keeps inference cost manageable while getting the benefit of a high-capacity classifier for the hard borderline cases that heuristics and perplexity cannot cleanly resolve. The classifier has seen enough of the data distribution from the positive and negative examples to recognize quality patterns that are invisible to simpler methods.

The pipeline design reflects a deliberate cost-accuracy tradeoff. Each stage is more accurate but slower than the previous one. By placing cheap methods first, you ensure that expensive methods only process documents that need careful evaluation.

The pipeline successfully routes different failure modes to different filters. The word repetition filter catches keyword spam. The duplicate line filter catches navigation boilerplate. The perplexity filter catches language model-detectable incoherence. Documents that are legitimate prose pass through all stages. The rejection breakdown tells you which filter is doing the most work, which helps calibrate the pipeline: if the perplexity filter is removing very few documents after the heuristics, you might be able to drop it and save computation; if it is removing many documents, it is earning its cost.

In a real pipeline at scale, the rejection breakdown is also a diagnostic tool. If your "symbol_ratio" filter is suddenly rejecting 40% of documents in a new crawl batch, that is a signal that something changed in the upstream HTML extraction: perhaps a new template is generating more symbol-heavy artifacts. Monitoring rejection rates by stage over time turns the filtering pipeline into a data quality sensor.

The pipeline above uses hard binary decisions: each filter either keeps or discards a document. An alternative is soft scoring, where each stage assigns a real-valued quality estimate and the final inclusion decision is based on a combined score.

Soft scoring has several advantages. It allows a document that barely fails one filter but strongly passes all others to potentially be included, which a hard pipeline would miss. It enables downstream users of the corpus to apply different quality cuts without reprocessing the raw data. And it provides gradient-like signals that help calibrate thresholds by examining the score distribution.

The practical implementation assigns each document a vector of per-filter scores (heuristic score, perplexity score, classifier score) and combines them with a learned or hand-tuned weighting. The Dolma toolkit (Allen AI's open-source data curation pipeline) supports this soft-scoring approach, storing per-document quality attributes alongside the filtered text so that users can apply different thresholds for different downstream applications.

Quality filtering is powerful but has well-documented failure modes that practitioners need to understand. These are not edge cases to be noted and dismissed: they have real consequences for the models produced and the communities affected.

Demographic bias is the most thoroughly documented concern. Filters trained on "high quality" examples that are predominantly formal English text from mainstream sources will systematically score African American Vernacular English (AAVE), code-switching, and other non-standard dialects as low quality. The C4 dataset was shown to significantly underrepresent text associated with Black, Hispanic, and other minority communities relative to their representation in the raw web. Measurements by researchers at the University of Washington found that documents containing dialect features associated with minority communities were disproportionately removed by the C4 filtering pipeline. If a model is trained on such filtered data, it performs worse on text from those communities, generating more errors and less fluent completions. This is not hypothetical: it has been measured in multiple independent studies. Any filtering pipeline that uses a model trained on a biased reference corpus will inherit and potentially amplify that bias.

Domain exclusion is a subtler problem. Quality filters calibrated on news and encyclopedia text will deprioritize legal documents, scientific papers, programming documentation, and other specialized domains that have different linguistic patterns. This can hurt the model's performance on specialized tasks, even tasks that are highly valuable (legal AI assistants, scientific literature search, code generation). The solution is domain-stratified filtering: separate quality models for different content types, or domain-specific thresholds that account for expected differences in register and style. Implementing this requires a domain classifier that runs before the quality classifier, adding another component to the pipeline but substantially improving coverage.

Filtering for scale introduces a different pressure. When your raw corpus is in the petabyte range, even a 1% false negative rate (failing to remove low-quality documents) translates to hundreds of terabytes of garbage in the training set. This pushes practitioners toward conservative thresholds, which in turn makes demographic bias and domain exclusion worse. The tension between scale-appropriate strictness and representational fairness has no clean engineering resolution. It requires explicit policy decisions about acceptable tradeoffs.

Temporal consistency is another challenge. Web text quality distribution shifts over time. A quality model trained in 2020 may perform poorly on 2024 web crawl data because the patterns of spam, machine-generated content, and boilerplate have changed. In particular, the massive increase in LLM-generated content on the web since 2022 has introduced a new category of technically clean but potentially low-value text: content that passes all surface-level quality checks but was generated by a model rather than a human. Production pipelines need regular re-evaluation and potentially retraining on fresh samples. This is operationally non-trivial: it requires maintaining labeled datasets, retraining classifiers, and re-filtering previously processed crawl data.

Over-filtering and the homogeneity problem affect downstream model behavior in ways that are difficult to measure but consequential. Models trained on aggressively filtered data tend to produce more uniform, formal-sounding text. They handle formal queries well but may sound stilted in casual conversation or fail to match the register of informal queries. They may underperform on tasks involving non-standard writing (social media analysis, dialect translation, informal summarization). The filtering pipeline's threshold settings are effectively hyperparameters of the final model, and the effects of those hyperparameters on model behavior are poorly characterized in most published work.

For researchers and practitioners, quality filtering is both a technical and a policy decision. The thresholds you choose, the reference model you train on, and the heuristics you encode all embed assumptions about what constitutes good text and whose language is considered standard. Documenting these choices, measuring their demographic effects, and auditing samples from filtered corpora for unexpected biases is increasingly expected in responsible data curation practice. Several recent data papers (Dolma, RedPajama-V2, ROOTS) explicitly include quality scores as separate metadata rather than pre-applying hard thresholds, precisely to allow downstream users to apply different filtering policies appropriate to their use cases.

Quality filtering removes low-value text from web-scraped corpora to improve model training efficiency and output quality. The three main approaches differ in cost, accuracy, and failure mode:

• Heuristic filters: Fast rule-based filters that catch obvious garbage including symbol-heavy text, keyword repetition, boilerplate lines, and too-short documents. They cost microseconds per document and scale to petabytes. The limitations are brittleness (rules designed for one domain or language may fail on others) and potential demographic bias in the rules themselves.
• Perplexity filtering: Scores documents against a reference language model trained on known-good text. Documents with high perplexity (surprising to the model) are flagged for removal. More accurate than heuristics for detecting incoherent or non-target-domain text. Uses percentile-based thresholds for robustness across different reference models. The main limitations are domain sensitivity (the reference model's training distribution determines what gets high perplexity) and the circular bias risk when the reference corpus is itself demographically skewed.
• Classifier-based filtering: Trains a binary quality classifier from labeled examples. More flexible and often more accurate than perplexity-based approaches. Can use logistic regression on character n-gram features (fast, interpretable) or neural classifiers with multiple input signals (more accurate). The main limitations are the cost of labeled data and inference, and the risk of inheriting whatever biases exist in the labeling process.

Threshold selection involves tradeoffs between precision (fewer false passes) and recall (fewer false rejections). Downstream evaluation, percentile-based thresholds, acceptance rate monitoring, and manual inspection of samples are the key tools. Domain-stratified filtering and soft scoring extend the basic pipeline to handle heterogeneous corpora.

Real pipelines chain all three approaches in order of increasing cost. Each stage removes different failure modes, and careful threshold calibration determines the final quality-diversity tradeoff. The key limitation is that "quality" is defined relative to a reference corpus, making filters potentially biased against non-standard language and specialized domains. Documenting these choices, measuring their demographic effects, and auditing filtered corpora for unexpected biases is part of responsible data curation practice. The models that result from this pipeline will reflect the filtering decisions made here: every threshold, every heuristic, and every reference corpus choice shapes what the model learns and what communities it serves well.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about quality filtering for language model training data.