Quality Filtering › Combining Filtering Methods
Language AI HandbookAll chapters Quality Filtering › Combining Filtering MethodsOne section · Sections in this chapter8 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
No single filtering method is complete on its own. Production pipelines chain multiple filters in sequence, with each stage removing different failure modes. The order of stages reflects a cost-accuracy tradeoff: cheap methods run first to reduce the volume reaching expensive methods, while more sophisticated methods handle the harder cases that simple rules cannot resolve.
The typical ordering of a production filtering pipeline proceeds through four stages. Language identification runs first. Documents classified as not being in the target language are removed immediately, and this alone eliminates a large fraction of raw web data for any target language. Even for a multilingual corpus, this stage assigns language confidence scores and routes documents to language-specific downstream processing.
Heuristic filters run next because they are fast (microseconds per document) and remove obvious garbage (symbol-heavy text, duplicated lines, too-short documents) before more expensive methods are applied. A heuristic stage that processes a trillion-token corpus in a few hours might prevent the perplexity stage from having to score 30-40% of the raw documents that would have been discarded anyway.
Perplexity filtering runs after heuristics. It is slower than heuristics (requiring n-gram model inference) but faster than neural classifiers. Documents surviving heuristic filtering are scored against a KenLM model and those above a percentile threshold are removed. This stage is particularly effective at catching coherently formatted but linguistically unnatural text: machine-translated boilerplate, auto-generated content, and documents in the wrong register for the target domain.
Classifier-based filtering is the final and most expensive stage. Only documents that pass the previous stages reach the classifier. This keeps inference cost manageable while getting the benefit of a high-capacity classifier for the hard borderline cases that heuristics and perplexity cannot cleanly resolve. The classifier has seen enough of the data distribution from the positive and negative examples to recognize quality patterns that are invisible to simpler methods.
The pipeline design reflects a deliberate cost-accuracy tradeoff. Each stage is more accurate but slower than the previous one. By placing cheap methods first, you ensure that expensive methods only process documents that need careful evaluation.
A Practical Filtering Pipeline
from dataclasses import dataclass
from typing import Callable, List
@dataclass
class FilterResult:
kept: bool
reason: str
class FilterPipeline:
"""Chainable document quality filter."""
def __init__(self):
self.filters: List[tuple[str, Callable[[str], bool]]] = []
def add_filter(self, name: str, fn: Callable[[str], bool]):
"""Add a filter. Function returns True if document should be KEPT."""
self.filters.append((name, fn))
return self
def evaluate(self, text: str) -> FilterResult:
for name, fn in self.filters:
if not fn(text):
return FilterResult(kept=False, reason=name)
return FilterResult(kept=True, reason="passed_all")
def batch_filter(self, texts: list) -> dict:
results = {"kept": [], "rejected": []}
rejection_counts = {}
for text in texts:
result = self.evaluate(text)
if result.kept:
results["kept"].append(text)
else:
results["rejected"].append(text)
rejection_counts[result.reason] = (
rejection_counts.get(result.reason, 0) + 1
)
results["rejection_breakdown"] = rejection_counts
return results
# Build a pipeline with increasing specificity
pipeline = FilterPipeline()
pipeline.add_filter("min_length", lambda t: len(t.split()) >= 30)
pipeline.add_filter("symbol_ratio", lambda t: symbol_ratio(t) < 0.25)
pipeline.add_filter("word_repetition", lambda t: max_word_repetition(t) < 0.5)
pipeline.add_filter("duplicate_lines", lambda t: duplicate_line_ratio(t) < 0.3)
pipeline.add_filter("perplexity_threshold", lambda t: lm.perplexity(t) < 400)
# Test documents of varying quality
test_corpus = [
"Scientists have developed a new battery technology that could double the range of electric vehicles. The lithium-sulfur chemistry offers higher energy density than conventional lithium-ion cells and uses cheaper materials. Field trials are expected to begin next year.",
"buy cheap insurance online best insurance deals lowest rates guaranteed compare insurance quotes now save money on auto home life insurance click here for free quote today",
"The committee reviewed proposals for urban infrastructure development including transportation networks water management and housing density regulations. Several competing approaches were evaluated against long-term sustainability criteria.",
"!!! FREE !!! click now limited time 50% off everything sale ends midnight tonight dont miss this amazing deal buy now pay later no interest",
"Home | Products | Services | About | Contact | Blog | Privacy Policy | Terms | Sitemap\nHome | Products | Services | About | Contact | Blog | Privacy Policy | Terms | Sitemap",
"var x = document.getElementById('main'); if (x) { x.style.display = 'block'; } else { console.log('not found'); } window.onload = function() { init(); };",
"The philosophical tradition stretching from Kant through Hegel to contemporary analytic philosophy grapples with persistent questions about the nature of knowledge, justification, and the relationship between mind and world. These debates inform how we understand scientific practice.",
"weather weather weather forecast forecast today tonight tomorrow weekend extended forecast high low temperatures precipitation probability humidity wind speed direction",
]Pipeline results: 2/8 documents passed all filters (25%) Rejection breakdown: min_length : 5 document(s) word_repetition : 1 document(s) Kept documents: - Scientists have developed a new battery technology that could double the range of electric... - The philosophical tradition stretching from Kant through Hegel to contemporary analytic ph...
The pipeline successfully routes different failure modes to different filters. The word repetition filter catches keyword spam. The duplicate line filter catches navigation boilerplate. The perplexity filter catches language model-detectable incoherence. Documents that are legitimate prose pass through all stages. The rejection breakdown tells you which filter is doing the most work, which helps calibrate the pipeline: if the perplexity filter is removing very few documents after the heuristics, you might be able to drop it and save computation; if it is removing many documents, it is earning its cost.
In a real pipeline at scale, the rejection breakdown is also a diagnostic tool. If your "symbol_ratio" filter is suddenly rejecting 40% of documents in a new crawl batch, that is a signal that something changed in the upstream HTML extraction: perhaps a new template is generating more symbol-heavy artifacts. Monitoring rejection rates by stage over time turns the filtering pipeline into a data quality sensor.
Soft Scoring Versus Hard Filtering
The pipeline above uses hard binary decisions: each filter either keeps or discards a document. An alternative is soft scoring, where each stage assigns a real-valued quality estimate and the final inclusion decision is based on a combined score.
Soft scoring has several advantages. It allows a document that barely fails one filter but strongly passes all others to potentially be included, which a hard pipeline would miss. It enables downstream users of the corpus to apply different quality cuts without reprocessing the raw data. And it provides gradient-like signals that help calibrate thresholds by examining the score distribution.
The practical implementation assigns each document a vector of per-filter scores (heuristic score, perplexity score, classifier score) and combines them with a learned or hand-tuned weighting. The Dolma toolkit (Allen AI's open-source data curation pipeline) supports this soft-scoring approach, storing per-document quality attributes alongside the filtered text so that users can apply different thresholds for different downstream applications.