Data Mixing › Quality Weighting

Domain labels are coarse. Within a single domain like "web crawl," you will find everything from carefully written encyclopedia articles to spam-filled link farms. Quality weighting addresses this heterogeneity by assigning individual documents or source sites different sampling probabilities based on estimated quality. While domain proportions determine how much weight each broad category receives, quality weighting determines which specific documents within each category get sampled most frequently.

The motivation here is efficiency. If 80% of your web crawl is low-quality noise, sampling uniformly from it means most of your training budget goes toward learning patterns that are not worth learning. Quality weighting lets you concentrate training on the informative 20%. In the limit, sufficiently aggressive quality filtering can produce a smaller, denser corpus that outperforms a much larger unfiltered corpus at the same training compute. The FineWeb dataset from Hugging Face demonstrated this trade-off: careful quality filtering of CommonCrawl produced a dataset where training on the filtered subset outperformed training on the full original despite having far fewer tokens.

Quality is not binary. The useful question is how much to sample from a document, rather than whether to include or exclude it outright. A document that scores 0.6 on a quality scale probably provides some learning signal; discarding it entirely wastes potentially useful data. Treating quality as a continuous weighting factor and adjusting sampling probability smoothly across the quality spectrum is generally more efficient than hard inclusion/exclusion thresholds.

One approach to quality weighting uses a classifier trained to distinguish high-quality text from low-quality text. The classifier assigns each document a score, and documents with higher scores are sampled more frequently.

The training data for such classifiers is typically constructed from contrasting examples. Wikipedia and books are treated as positive examples of high-quality text. Raw web crawl data is treated as negative. A simple binary classifier trained on these examples learns to identify the stylistic and structural features that distinguish clean, informative writing from noise: well-formed sentences, consistent punctuation, appropriate paragraph structure, factual density, absence of navigation menus and boilerplate.

GPT-3 used this approach: the OpenAI team trained a classifier to identify text similar to WebText (a curated collection of web pages linked from Reddit posts that received substantial karma, serving as a proxy for human-judged quality), and then used the classifier scores to filter and weight their CommonCrawl corpus. This single step dramatically improved the effective quality of their web data. The classifier learned to separate the long tail of low-quality web text from the informative content that made up the high-karma-link-target pages.

FineWeb, released by Hugging Face in 2024, takes a more sophisticated approach. Rather than a binary classifier, it applies a cascade of heuristics and learned filters, then evaluates the resulting corpus by training small proxy models and measuring performance on benchmarks. This evaluation-driven approach lets you see the impact of quality filtering choices without committing to a full training run. The FineWeb work also highlighted an important finding: different quality filters are not always additive. Some filters that look good in isolation degrade performance when combined, because they remove documents that carry specific types of useful signal even if those documents score low on some quality dimensions.

One important limitation of classifier-based approaches is that they are bounded by the quality of your positive examples. If your "high quality" training signal comes from Wikipedia and English-language books, your classifier will favor text that resembles Wikipedia and English books. It will down-score technically rigorous but stylistically unusual text, multilingual content, and anything that follows conventions different from your reference corpus. A data science tutorial written in informal, conversational style may score low even though it contains valuable technical content. A legal document in precise but arcane language may be penalized. This is not always a problem, but it is worth keeping in mind when working with diverse data sources, especially when building models intended for non-English languages or specialized technical domains.

An even simpler form of quality weighting assigns weights based on the source of a document rather than its content. Documents from known high-quality sources (academic publishers, major newspapers, authoritative encyclopedias) receive higher weights, while documents from low-quality domains (spam sites, link farms, autogenerated content) receive lower weights or are discarded entirely.

This approach is computationally cheap because it requires only a domain-name lookup, not content analysis. When you are working with a corpus of hundreds of billions of documents, the ability to apply quality weighting without processing each document individually is operationally important. Source-based weighting can be applied as a preprocessing step that drastically reduces the amount of content requiring per-document scoring.

The limitation of source-based weighting is that it applies a single quality score to all content from a given source, even when that source produces variable-quality content. A news site publishes both in-depth investigative journalism and listicle fluff. A university website hosts rigorous research papers alongside undergraduate course syllabi. A programming forum contains expert answers alongside incorrect, confused questions. Source-based weighting cannot distinguish between these. You are averaging quality across a source's entire output, which may wash out the best content from heterogeneous sources.

The C4 dataset used a set of heuristic filters including discarding pages from a blocklist of known low-quality sites. RedPajama constructed per-domain quality scores by estimating the fraction of content from each domain that survived multiple quality filters. This creates a hybrid: the quality score of a domain is derived from content-level filtering, but applied at the domain level for efficiency. Documents from a domain where 90% of content passes filters receive higher base weights than documents from a domain where only 30% of content passes, even before per-document scoring is applied.

A third quality signal uses a small, high-quality language model to score documents. Documents that the model assigns low perplexity to (meaning they are coherent and predictable by that model) are treated as higher quality. Documents that receive high perplexity from the reference model are considered less coherent and may be downweighted or removed.

Perplexity measures how surprised a model is by a given piece of text. A well-formed, coherent sentence follows predictable linguistic patterns, and a language model trained on good text will assign it low perplexity. Fragmented, noisy, or incoherent text violates those patterns and receives high perplexity. The intuition is that text which is surprising even to a model trained on good data is probably not good data itself.

To make this concrete, consider a document containing sentences like: "The algorithm computes optimal solutions using dynamic programming techniques developed by Bellman in the 1950s." A language model trained on scientific and technical writing will assign this low perplexity because the vocabulary, syntax, and topic are all familiar. Now consider a spam document: "Click here!!! Buy now!!! Limited time!!! Amazing deals!!!" A good language model will assign this high perplexity not because the words are unusual but because the pattern is erratic and does not follow the coherent structure the model has learned to expect.

This approach has a subtle bias: it favors text that resembles whatever was used to train the scoring model. If your scoring model was trained on English news articles, it will assign low perplexity to similar text and high perplexity to code, non-English text, or unusual writing styles that might be valuable. Perplexity-based filtering is most effective when combined with other signals and when the scoring model is specifically calibrated for the domains you care about. Using a code-specialized model to score code documents is more appropriate than using a general-purpose English model.

There is also a practical consideration about the size of the scoring model. A scoring model needs to be small enough to process your entire corpus efficiently. Scoring hundreds of billions of documents with a 7-billion-parameter model is feasible but requires careful infrastructure design. Most practitioners use 100-million to 1-billion-parameter models for perplexity-based scoring, accepting some loss of scoring accuracy in exchange for computational tractability.

In practice, effective quality weighting combines multiple signals. A document's final sampling probability might be a product of factors including its domain weight, its classifier score, its source URL reputation, and any hard quality thresholds it survived during the earlier filtering pipeline described in prior chapters.

The combination function matters. Multiplicative combinations, where you multiply all factors together, tend to aggressively discard documents that score poorly on any single dimension, which can over-filter. A document that scores 0.4 on the classifier but 0.9 on URL reputation and 0.8 on perplexity might be acceptable, but multiplicative combination would assign it a combined score of around 0.29. The document gets heavily penalized for one weak signal even though two other signals rate it positively. Additive combinations or learned combination functions are sometimes more robust because they allow high scores on some dimensions to compensate for lower scores on others.

Threshold-based approaches (keep everything above 0.5, discard the rest) are simpler but discard potentially useful intermediate-quality content. A document at 0.49 and a document at 0.5 are nearly identical in quality, but a hard threshold treats them very differently. A continuous weighting scheme that smoothly adjusts sampling probability across the quality spectrum is generally preferable to hard thresholds, because it allows the model to learn from borderline content at reduced frequency rather than wasting that content entirely.

One important practical consideration: quality signals are expensive to compute, and you typically want to compute them once and store them as metadata attached to each document in your corpus. The data mixing system then uses these precomputed scores at training time. This separation between the scoring pipeline and the training pipeline simplifies both: scoring can run asynchronously on whatever hardware is convenient, and training just needs to look up the stored scores and apply the sampling probabilities.