Data Mixing › Domain Proportions

The most fundamental mixing decision is how much weight to assign each broad domain category. Domains are typically coarse groupings like "web," "books," "code," "academic," and "conversational," though practitioners often define finer-grained categories depending on what data they have available. The granularity of domain definition is itself a design choice: coarser categories are easier to reason about and require fewer ablations, while finer categories give you more control but multiply the search space.

The word "proportions" is key here. At any given training step, the model processes a batch of tokens. Those tokens are sampled from the training corpus according to your mixing recipe. If your recipe says 30% web and 20% code, then roughly 30% of tokens in each batch come from web sources and 20% come from code repositories, on average across many batches. The mixing recipe is not about the total size of each corpus but about the probability of sampling from it. This distinction matters enormously: a corpus with 3 trillion tokens does not automatically deserve 3x more weight than a corpus with 1 trillion tokens. Weight reflects intended contribution to the model's learned behavior, not raw data availability.

A naive approach is to sample uniformly from the combined corpus: each document is equally likely to appear in each training batch regardless of its source. This approach sounds principled, but it produces models that reflect the raw composition of the internet, which is heavily skewed. The web contains vastly more casual social media text than scientific publications. If you sample proportionally to corpus size, your model will have seen millions of Reddit comments for every medical journal article.

Representation is only one part of the problem. Domain composition also affects generalization. A model trained mostly on web text will struggle to follow the precise, structured reasoning expected in code or mathematics, even if it has technically seen some examples of those domains. Conversely, a model trained with an unrealistically high proportion of academic text may generate formal, stilted prose when asked for a conversational reply. The domain mixture shapes the model's default mode of expression, its tolerance for ambiguity, and its sensitivity to precise technical language.

The practical manifestation of this is well documented. The original GPT models, trained heavily on web text, were noticeably better at producing fluent prose than at solving multi-step math problems or writing correct code. Early code generation systems required dedicated code-heavy training data. The progression from GPT-3 to Codex, a code-specialized variant, involved dramatically increasing the fraction of Python and other programming language data, showing just how directly domain composition maps to capability. Adding code data does more than improve code generation: it also improves multi-step reasoning in general, because code imposes a discipline of formal, step-by-step logic that transfers to other domains.

The insight that makes domain mixing work is that you can systematically boost or suppress domains to achieve the capability profile you want. This is analogous to class reweighting in supervised learning, except you are reweighting entire source distributions rather than individual labels. And just as class reweighting in supervised learning has principled solutions (inverse frequency weighting, cost-sensitive learning), data mixing has developed its own set of principled approaches over the past few years.

Before going further, it helps to have a clear mental model of what a training run consumes. Modern large language models train on between one trillion and ten trillion tokens. Each training step processes a batch of several thousand tokens. The total number of training steps times the batch size gives you the total token budget. When you specify a mixing recipe, you are specifying how that budget gets allocated across your data pools.

For a concrete example: suppose you are training a 7 billion parameter model on 2 trillion tokens, with a batch size of 4 million tokens per step. You will take 500,000 training steps. If your recipe assigns 20% to code, then across those 500,000 steps, roughly 400 billion tokens will come from code sources. Whether your code corpus has enough distinct tokens to sustain that without excessive repetition depends on its size. If your code pool has only 100 billion distinct tokens, you will see each token an average of 4 times during training. Whether that is a problem depends on the domain and the repetition tolerance, which we discuss in the oversampling section below.

This framing helps you reason about the practical feasibility of your recipe. Before running any ablations, you should calculate the implied oversampling ratio for each domain under your proposed recipe and flag any that look problematic. A recipe that works on paper may be impossible to execute without unacceptable repetition.

Several large-scale training efforts have published their domain compositions, giving a useful starting point for practitioners.

The Llama 2 training dataset used approximately 89% web text (CommonCrawl and related sources), 8% code from GitHub, and smaller fractions of books, Wikipedia, and other high-quality sources. The model's strong general-purpose performance despite this web-heavy mixture reflects the scale and quality of the filtered web data used. GPT-3 weighted its mixture toward filtered web data (CommonCrawl), with books at about 16%, WebText at 22%, and Wikipedia at 3%. These choices reflected the team's judgment about what capabilities mattered most for general-purpose text generation. The Chinchilla paper from DeepMind, while primarily focused on compute-optimal scaling laws, noted the importance of domain composition but did not publish full mixing weights.

The Falcon models, from the Technology Innovation Institute, trained on RefinedWeb, a carefully filtered subset of CommonCrawl, and found that a high-quality web corpus alone could match models trained on more diverse mixtures at equivalent scale. This finding is important: it suggests that quality filtering within a domain can sometimes substitute for domain diversification. A very clean, carefully filtered web corpus can deliver similar capabilities to a more diverse mixture, as long as the filtering is aggressive enough.

Dolma, the open dataset used to train OLMo, published detailed breakdowns: approximately 38% Common Crawl (post-filtering), 26% The Stack (code), 11% C4, 11% Reddit (filtered), 7% peS2o (scientific papers), and other sources making up the remainder. These proportions reflect deliberate choices about what capabilities the OLMo team wanted the model to emphasize. The 26% code allocation is high for a general-purpose model, showing the team's view that code data contributes broadly to reasoning abilities.

The Mistral and Mixtral models from Mistral AI have published limited information about their training data but are widely believed to use a heavy code fraction, consistent with their strong performance on coding and reasoning benchmarks. Phi-1 and its successors from Microsoft went even further, showing that small models trained almost entirely on high-quality synthetic and curated data can outperform much larger models trained on raw web text. These results challenge the assumption that larger corpora are always better and point toward the importance of data quality over quantity.

These published recipes should not be treated as ground truth. They reflect the goals, compute budgets, and data quality decisions of specific projects. A model intended primarily for coding assistance would weight code much more heavily. A model intended for biomedical applications would boost scientific literature. Use published recipes as baselines to understand the range of reasonable choices, not as universal answers. The right recipe for your model depends on your intended capabilities, your available data, and your compute budget.

When you have a small but high-quality domain (say, books or peer-reviewed papers), you will often want to oversample it. If your total corpus is 2 trillion tokens and your book data is 20 billion tokens, proportional sampling means the model sees book text only 1% of the time. To give books a 10% effective weight, you need to repeat the book data roughly 10 times over training.

Oversampling has diminishing returns. Repeating data exposes the model to the same examples many times, which can cause it to memorize rather than generalize. Practical experience across multiple large training runs suggests that repeating high-quality data up to 2-4 times produces meaningful gains, while oversampling beyond 10x tends to hurt. This creates a ceiling on how much you can compensate for data scarcity by upweighting.

Why exactly does high oversampling hurt? The explanation involves what the model's loss is measuring. Early in training, seeing the same text again provides useful gradient signal because the model has not yet learned to predict it well. But once the model has seen a passage several times, its loss on that passage approaches zero: it has effectively memorized the token sequence. The gradient signal from memorized examples becomes very small, and the model gains little from seeing them again. Worse, those memorized patterns may begin to act as spurious shortcuts that hurt generalization to new text.

To see where the ceiling comes from more concretely, consider how many times each token in domain $d$ will be seen during training. If domain $d$ has $N_d$ available tokens and you want it to contribute fraction $w_d$ of a total training budget of $T$ tokens, the expected number of times each token is seen is:

$$\text{repeats}_d = \frac{w_d \cdot T}{N_d}$$

where:

• $w_d$: the target weight for domain $d$, a value between 0 and 1 with all domain weights summing to 1
• $T$: the total training token budget for the full run
• $N_d$: the number of distinct tokens available in domain $d$

When $\text{repeats}_d > 1$, each token is seen more than once on average, meaning the data is oversampled. When $\text{repeats}_d < 1$, the model sees only a fraction of the available tokens, meaning the data is undersampled. Keeping $\text{repeats}_d$ below 4 for most high-quality domains is a reasonable empirical heuristic, though more data-scarce domains may tolerate higher repetition if the quality is very high and the corpus is short enough to be largely memorized without harm.

The ceiling creates a practical upper bound on how much you can compensate for small domain size through upweighting. If your books corpus has 20 billion tokens and you want to train for 2 trillion total tokens with books at 15% weight, you are oversampling books roughly 15x. Most practitioners would reduce the books weight to something compatible with 3-4x oversampling, or accept that books will be underweighted relative to their quality because the corpus is simply too small to sustain higher weight without problematic repetition.

There is an important nuance here. Not all oversampling is equally harmful. Data diversity within a domain modulates how much repetition hurts. A books corpus with 80 billion tokens drawn from hundreds of thousands of distinct books is more tolerant of oversampling than a corpus where the same few thousand books dominate. Within-domain diversity means the model sees different examples each time even if aggregate corpus size stays the same. This is one reason why practitioners care about deduplication within domains: removing near-duplicate documents effectively increases diversity and raises the oversampling ceiling.

Domain proportions interact with each other in non-trivial ways. Adding more code data does not just improve coding performance: it typically also improves mathematical reasoning, logical deduction, and instruction following. Adding more high-quality web text does not just improve general language: it can also improve factual grounding because web text contains references to real-world entities and events. These spillover effects mean that the optimal mixing recipe is not a simple sum of domain-specific contributions.

Understanding these interactions requires controlled ablations. The standard approach is to hold all domains constant except one, vary that domain's weight across a range of values, and measure the impact on all capability dimensions. Repeating this for each domain gives you a sensitivity profile: how much does each capability change when you add or remove a given domain? Domains with large spillover effects are worth weighting more heavily than their primary-capability contribution alone would suggest.

Code is perhaps the best-documented case of cross-domain spillover. Multiple studies have shown that training on code, even for models that will not be used for coding tasks, improves performance on multi-step reasoning benchmarks, mathematical problem solving, and structured generation tasks. The hypothesis is that code imposes constraints that other domains do not: programs must be syntactically valid, variables must be defined before use, and function calls must match their signatures. Learning to model this kind of structured, rule-governed text trains the model to apply similar discipline in other contexts.