Quality Filtering › Visualizing Filter Precision and Recall
Language AI HandbookAll chapters Quality Filtering › Visualizing Filter Precision and RecallOne section · Sections in this chapter6 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The fundamental constraint of quality filtering is the precision-recall tradeoff. Precision measures what fraction of the documents you keep are high quality. Recall measures what fraction of the high-quality documents you successfully retain. A perfect filter would have both precision and recall equal to 1.0, but in practice the two values trade off against each other through the threshold parameter.
When you raise the threshold (require a higher quality score to pass), precision goes up (fewer garbage documents pass) but recall goes down (more legitimate documents fail). When you lower the threshold, recall goes up but precision goes down. The right operating point on this curve depends on the relative costs of the two errors.
In data curation, the cost of low recall depends on how much data you have. If you have an enormous raw corpus, you can afford to be strict and discard ambiguous documents. If you are targeting a niche domain with limited available text, discarding borderline documents could leave you with a corpus that is too small for effective training. The cost of low precision depends on how much the garbage hurts. For general-purpose pretraining, some noise may be tolerable. For safety-critical fine-tuning, even a small fraction of problematic content can have outsized effects.
import numpy as np
from sklearn.metrics import precision_recall_curve
# Generate synthetic quality scores for a large corpus
rng = np.random.default_rng(42)
n_docs = 10000
true_quality = rng.binomial(1, 0.35, size=n_docs) # 35% high quality
# Simulate classifier scores: high-quality docs tend to score higher
scores_hq = rng.beta(a=4.0, b=2.0, size=true_quality.sum())
scores_lq = rng.beta(a=2.0, b=5.0, size=(1 - true_quality).sum())
all_scores = np.empty(n_docs)
all_scores[true_quality == 1] = scores_hq
all_scores[true_quality == 0] = scores_lq
precision, recall, thresholds = precision_recall_curve(true_quality, all_scores)
The curve illustrates the fundamental tradeoff. Requiring high precision means setting a strict threshold, which means filtering out more documents including some legitimate ones (low recall). Requiring high recall means setting a lenient threshold, accepting more low-quality documents (lower precision). The right operating point depends on how much high-quality data you have available. If you have an abundance of data, you can afford to be strict. If your target domain is data-sparse, you may need to accept more noise.
The "balanced" operating point shown at roughly equal precision and recall is not always the right choice, even when the costs of both errors seem symmetric. For language model pretraining, most practitioners find that a recall-biased operating point (say, 0.85 recall, 0.70 precision) is preferable to a precision-biased one at the same area under the curve, because the model can partially compensate for noisy training data through its own averaging during pretraining, but it cannot compensate for data it never saw.