Quality Filtering › Heuristic Filters
Language AI HandbookAll chapters Quality Filtering › Heuristic FiltersOne section · Sections in this chapter3 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Heuristic filtering is the first line of defense. These filters encode explicit rules about what bad text looks like. They run fast (often in microseconds per document), scale to petabytes, and catch the most obvious problems without requiring any model inference. The downside is that heuristics are brittle: they fail on text that looks superficially clean but is semantically poor, and they may incorrectly discard legitimate text that happens to match the wrong pattern.
The insight behind heuristic filters is that low-quality web content has characteristic surface properties that distinguish it from natural prose. Spam articles repeat keywords. Navigation menus have very short lines. Scraped template content has identical repeated blocks. Pages that failed HTML parsing have high ratios of angle brackets and escaped entities. Rather than modeling these problems semantically, heuristics directly measure the surface properties that correlate with low quality.
The heuristic filters used in production pipelines like C4, MassiveText, and RefinedWeb fall into several categories:
- Length-based filters: Remove documents that are too short (fewer than 200 words, for instance) or too long. Short documents provide too little context for the model to learn useful patterns. Extremely long documents are sometimes artifacts of parsing failures where many separate pages got concatenated.
- Punctuation and symbol ratio filters: Flag documents where a large fraction of characters are symbols, numbers, or non-letter characters. A document that is 30% curly braces is almost certainly code or template junk, not natural language.
- Alphabet coverage filters: Remove documents where most content is non-alphabetic or where the text comes from an unexpected script. This is also used for language identification when language-specific filtering is required.
- Bullet point or line-length filters: Documents composed almost entirely of short, newline-delimited items (like site navigation menus or keyword lists) tend to have poor narrative structure. The ratio of lines to words can identify these.
- Boilerplate detection: Many web pages share identical footers, cookie banners, and privacy policy text. Hard-coded string matching can remove pages where more than a fixed percentage of the text matches known boilerplate phrases. This complements deduplication, which operates at the document level, by catching pages where boilerplate comprises a large fraction of otherwise-unique content.
- Bad word lists: Some pipelines maintain lists of profanity, spam trigger words, or adult content markers. Any document containing words from these lists is flagged or removed. The C4 pipeline famously uses the "List of Dirty, Naughty, Obscene, or Otherwise Bad Words" maintained on GitHub.
- Repetition filters: A document that repeats the same sentence or paragraph is almost certainly low-quality. You can detect this by checking whether any n-gram above a certain length occurs more than once in the document, or by measuring the character-level entropy of the content. Low entropy means low informational diversity.
Each rule has an explicit failure mode. Length filters discard short but high-quality documents like dictionary definitions or aphorisms. Symbol ratio filters may incorrectly remove technical documentation with code snippets. Bad word lists may disproportionately remove text from communities where that vocabulary appears in legitimate cultural or literary contexts. Understanding these failure modes is not academic: it determines which heuristics to apply, which to skip for a given domain, and how to calibrate thresholds.
Worked Example: Repetition and Symbol Filters
Let's look at what these filters detect and why they work.
import re
from collections import Counter
def symbol_ratio(text):
"""Fraction of characters that are non-alphabetic and non-space."""
if not text:
return 1.0
non_alpha = sum(1 for c in text if not c.isalpha() and not c.isspace())
return non_alpha / len(text)
def max_word_repetition(text, top_n=5):
"""Fraction of words accounted for by the top-N most common words (excluding stopwords)."""
stopwords = {
"the",
"a",
"an",
"and",
"or",
"in",
"on",
"at",
"to",
"of",
"is",
"it",
"this",
"that",
}
words = [
w.lower()
for w in text.split()
if w.isalpha() and w.lower() not in stopwords
]
if not words:
return 1.0
counts = Counter(words)
top_count = sum(v for _, v in counts.most_common(top_n))
return top_count / len(words)
def duplicate_line_ratio(text):
"""Fraction of lines that are duplicates."""
lines = [l.strip() for l in text.splitlines() if l.strip()]
if not lines:
return 0.0
duplicates = len(lines) - len(set(lines))
return duplicates / len(lines)
# Example documents
docs = {
"Clean news article": (
"Scientists at MIT have developed a new approach to carbon capture that "
"could reduce costs by up to 40 percent. The technique uses a novel "
"polymer membrane that selectively absorbs CO2 from industrial exhaust. "
"Initial tests suggest the system could be deployed at scale within five years."
),
"Keyword spam": (
"buy cheap insurance online get insurance quotes best insurance rates "
"cheap car insurance home insurance life insurance online insurance quotes "
"buy insurance now insurance deals insurance comparison insurance calculator"
),
"Template boilerplate": (
"Home | About | Contact | Privacy Policy | Terms of Service\n"
"Home | About | Contact | Privacy Policy | Terms of Service\n"
"Home | About | Contact | Privacy Policy | Terms of Service\n"
"Copyright 2023. All rights reserved."
),
"Symbol-heavy": (
">>> import sys; sys.path.append('/usr/local/lib') "
"{{config.key}}: [val1, val2, val3] "
"SELECT * FROM table WHERE id=42 AND status='active';"
),
}Document Symbol% WordRep% DupLine% -------------------------------------------------------------- Clean news article 2.09% 18.75% 0.00% Keyword spam 0.00% 65.52% 0.00% Template boilerplate 8.45% 62.50% 50.00% Symbol-heavy 24.82% 100.00% 0.00%
The clean news article scores low on all three metrics. The keyword spam shows a very high word repetition ratio because the same words ("insurance") dominate the vocabulary after stopword removal. The template document is flagged by the duplicate line ratio. The symbol-heavy code fragment is caught by the symbol ratio.
These thresholds are tunable. If you set the symbol ratio cutoff at 0.3, you reject documents where more than 30% of characters are symbols. If you set the word repetition threshold at 0.4, you reject documents where the top 5 words account for more than 40% of non-stopword content. The right thresholds depend on your domain: code corpora need higher symbol ratio thresholds, while technical documentation might legitimately use more repetitive terminology than general prose.
The C4 Heuristic Suite
The C4 dataset (used to train T5 and mT5) is one of the most influential filtered corpora, and its heuristic rules have become a baseline for many subsequent efforts. The C4 processing pipeline starts with Common Crawl's WET files (pre-extracted text), then applies the following filters. Lines are discarded if they:
- Do not end with terminal punctuation (a period, question mark, or exclamation point)
- Contain fewer than 3 words or more than 100,000 characters
- Contain the string "javascript" (to catch pages where the parser captured script blocks)
- Contain the string "lorem ipsum" (to catch placeholder content)
- Contain any words from the "List of Dirty, Naughty, Obscene, or Otherwise Bad Words"
Documents are then removed if fewer than 5 sentences remain after line filtering, or if they contain the phrase "lorem ipsum". The entire pipeline removed roughly 70% of raw Common Crawl documents, producing a cleaner but still large 750 GB corpus.
Later analyses questioned some of these choices and their consequences. The terminal punctuation filter disproportionately removes non-English text where punctuation conventions differ: many East Asian languages do not end sentences with a period. The JavaScript filter turned out to remove legitimate pages that happened to discuss JavaScript programming. The bad word filter significantly underrepresented content from communities that use that vocabulary in legitimate cultural, literary, or conversational contexts.
These are typical failure modes of heuristic filters: rules designed around one language or content type, or calibrated on a particular demographic's writing, can be inappropriate for others. Every rule encodes an assumption about what normal text looks like, and those assumptions tend to reflect the dominant culture and language of whoever designed the rules.
Character-Level Entropy as a Quality Signal
One less-obvious but useful heuristic is character-level entropy. Informative text has high character diversity: it uses a wide variety of words and transitions. Repetitive spam or boilerplate has low entropy because the same short n-grams repeat constantly.
Given a document with characters, the empirical character unigram distribution is for each character . The character-level entropy is:
where the sum runs over all distinct characters appearing in the document. Well-written prose in English typically has character entropy between 4.5 and 5.2 bits. Repetitive spam with limited vocabulary falls below 4.0. Compressed or encoded binary data exceeds 6.0. Setting a window of acceptable entropy values gives you a fast filter that catches a class of garbage that other rules miss.
The entropy measure is also language-aware in a useful way: it naturally adapts to the expected character distribution of the target language. Chinese text will have different typical entropy values than English, but the threshold "reject documents below the 5th percentile of entropy for this language" works without needing separate calibration.