Quality Filtering › Setting Filter Thresholds
Language AI HandbookAll chapters Quality Filtering › Setting Filter ThresholdsOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Choosing the right threshold is as important as choosing the right filtering method. Set it too strict and you discard legitimate training data, reducing diversity and potentially introducing biases. Set it too lenient and you let through too much noise. There is no universally correct answer, but several strategies help.
Threshold Selection Strategies
The most principled approach is downstream evaluation. You vary the threshold, train (or fine-tune) a model on each filtered corpus, and measure performance on held-out benchmarks. The threshold that maximizes downstream task performance is the right one. This is expensive (it requires training multiple models) but gives ground truth. For large-scale pretraining, this approach is often impractical, but it is feasible for domain-specific fine-tuning where training is cheap.
A cheaper proxy is to monitor the acceptance rate at different thresholds. If your filter accepts 15% of documents, you might be too strict unless you have an enormous raw corpus. If it accepts 85%, you are probably not filtering enough for a general-purpose quality pipeline. Most production pipelines aim for an acceptance rate in the 20-60% range for quality-focused filtering, though this depends heavily on the quality of the source crawl. Fresh, high-quality crawls from curated seeds may have 60% acceptance rates; broad Common Crawl data might have 20-30% acceptance rates.
Percentile-based thresholds avoid the sensitivity to reference model calibration. Instead of "reject documents with perplexity > 500", you set "reject the top 30% of documents by perplexity." The percentile threshold is stable across different reference models and corpora, making it easier to reproduce and compare across pipeline iterations.
Visual inspection remains important at every stage. After setting a threshold, sample 100 accepted documents and 100 rejected documents and review them manually. You will almost always find surprises: legitimate text being rejected for unexpected reasons, or garbage that somehow passes the filter. Manual inspection catches failure modes that quantitative metrics miss. In practice, teams at Google, OpenAI, and other organizations report spending significant time reading samples from their filtered corpora before finalizing thresholds.
Stratified threshold analysis is an advanced strategy for pipelines that care about corpus diversity. Rather than setting a single acceptance rate across all domains, you set domain-specific acceptance rates. You might accept 70% of Wikipedia-style documents (because they are almost all clean) while accepting only 25% of forum content (because it has a higher noise rate). The domain labels can come from a separate classifier, URL patterns, or the language identification stage. This ensures that stricter filtering on noisy sources does not eliminate the noisier-but-legitimate domains entirely.
Threshold Effects on Corpus Statistics
Let's look at how different threshold choices affect corpus composition in a concrete way.
import numpy as np
rng = np.random.default_rng(0)
n_docs = 100_000
# Simulate document corpus with quality scores and text properties
quality_scores = rng.beta(2.5, 4.0, size=n_docs) # Most docs are low quality
doc_lengths = rng.lognormal(mean=6.5, sigma=1.0, size=n_docs).astype(int)
domain_labels = rng.choice(
["news", "wiki", "forums", "spam", "boilerplate"],
size=n_docs,
p=[0.15, 0.10, 0.25, 0.30, 0.20],
)
# Higher quality scores correlate with certain domains
domain_score_boost = {
"news": 0.25,
"wiki": 0.30,
"forums": 0.05,
"spam": -0.20,
"boilerplate": -0.25,
}
for i, domain in enumerate(domain_labels):
quality_scores[i] = np.clip(
quality_scores[i] + domain_score_boost[domain], 0, 1
)
thresholds = [0.2, 0.35, 0.5, 0.65, 0.8] Threshold Accept% Docs Kept Mean Length Domain Distribution (news/wiki/forums/spam/bplate)
----------------------------------------------------------------------------------------------------
0.20 68.2% 68,174 1103 22.1% / 14.5% / 33.6% / 19.6% / 10.3%
0.35 48.8% 48,828 1103 29.8% / 20.1% / 33.2% / 11.6% / 5.3%
0.50 30.4% 30,379 1103 37.3% / 27.1% / 29.0% / 4.8% / 1.7%
0.65 15.5% 15,505 1091 42.9% / 35.2% / 21.0% / 0.9% / 0.1%
0.80 6.2% 6,175 1101 45.8% / 43.7% / 10.5% / 0.0% / 0.0%As the threshold increases, the acceptance rate drops and the domain composition shifts away from spam and boilerplate toward news and Wikipedia content. This is exactly the desired behavior. However, note that even at the strictest threshold, some spam still passes through, and some legitimate forum content is excluded. No threshold completely separates the classes.
Notice also how the accepted corpus becomes less diverse as the threshold rises. High-threshold filtering produces a corpus concentrated in formal, structured text (news, encyclopedia entries) and underrepresents informal but legitimate content (forums, personal blogs). This can affect the model's downstream capabilities: models trained only on formal text may struggle with casual conversational queries, colloquial language, or the writing styles of communities whose text disproportionately appears in lower-scored domains.
Visualizing Threshold Effects on Domain Composition
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
domains = ["news", "wiki", "forums", "spam", "boilerplate"]
domain_colors = {
"news": theme_color("#4c72b0"),
"wiki": theme_color("#55a868"),
"forums": theme_color("#c44e52"),
"spam": theme_color("#dd8452"),
"boilerplate": theme_color("#8172b3"),
}
threshold_vals = np.linspace(0.1, 0.9, 50)
domain_fracs_by_thresh = {d: [] for d in domains}
accept_rates = []
for thresh in threshold_vals:
mask = quality_scores >= thresh
kept = mask.sum()
accept_rates.append(100 * kept / n_docs)
for d in domains:
frac = (domain_labels[mask] == d).sum() / kept if kept > 0 else 0
domain_fracs_by_thresh[d].append(frac)
The right panel makes the cost of aggressive filtering concrete: moving from a threshold of 0.3 to 0.7 cuts the accepted corpus to a fraction of its former size. The left panel shows what you gain and lose in detail. The gains are obvious: less spam and boilerplate. The losses are subtler: you also lose forum content that includes useful informal language patterns, colloquialisms, and practical knowledge that does not appear in formal sources. A model trained on the high-threshold corpus will produce more formal text by default and will have worse calibration on casual queries.
This visualization also reveals an important property of corpus composition: the relative fractions of domains shift even when you do not explicitly target them. A quality filter calibrated entirely on spam versus clean text will still disproportionately remove forum content because forum text happens to share surface properties with lower-quality content. If forum coverage matters for your downstream use case, you need either a domain-stratified filter or a post-filtering resampling step that restores target domain proportions.