Quality Filtering › Summary

Quality filtering removes low-value text from web-scraped corpora to improve model training efficiency and output quality. The three main approaches differ in cost, accuracy, and failure mode:

• Heuristic filters: Fast rule-based filters that catch obvious garbage including symbol-heavy text, keyword repetition, boilerplate lines, and too-short documents. They cost microseconds per document and scale to petabytes. The limitations are brittleness (rules designed for one domain or language may fail on others) and potential demographic bias in the rules themselves.
• Perplexity filtering: Scores documents against a reference language model trained on known-good text. Documents with high perplexity (surprising to the model) are flagged for removal. More accurate than heuristics for detecting incoherent or non-target-domain text. Uses percentile-based thresholds for robustness across different reference models. The main limitations are domain sensitivity (the reference model's training distribution determines what gets high perplexity) and the circular bias risk when the reference corpus is itself demographically skewed.
• Classifier-based filtering: Trains a binary quality classifier from labeled examples. More flexible and often more accurate than perplexity-based approaches. Can use logistic regression on character n-gram features (fast, interpretable) or neural classifiers with multiple input signals (more accurate). The main limitations are the cost of labeled data and inference, and the risk of inheriting whatever biases exist in the labeling process.

Threshold selection involves tradeoffs between precision (fewer false passes) and recall (fewer false rejections). Downstream evaluation, percentile-based thresholds, acceptance rate monitoring, and manual inspection of samples are the key tools. Domain-stratified filtering and soft scoring extend the basic pipeline to handle heterogeneous corpora.

Real pipelines chain all three approaches in order of increasing cost. Each stage removes different failure modes, and careful threshold calibration determines the final quality-diversity tradeoff. The key limitation is that "quality" is defined relative to a reference corpus, making filters potentially biased against non-standard language and specialized domains. Documenting these choices, measuring their demographic effects, and auditing filtered corpora for unexpected biases is part of responsible data curation practice. The models that result from this pipeline will reflect the filtering decisions made here: every threshold, every heuristic, and every reference corpus choice shapes what the model learns and what communities it serves well.