Data Mixing › Optimal Mixing

"Optimal" mixing depends on what you are optimizing for, and that target is not always obvious. A model optimized for coding performance will sacrifice some general language quality. A model optimized for factual accuracy will see less conversational text and may produce stiffer responses. The optimal mixing recipe is never optimal in all dimensions simultaneously; it represents a set of tradeoffs chosen by the practitioners who design it. Framing the question correctly, as "optimal for what?" rather than "optimal in general," is the first step toward making principled mixing decisions.

One philosophy is to mix data in proportion to how much each source contributes to the capabilities you want. This sounds circular but is operationalized through benchmark decomposition: run ablations that train models with one domain removed at a time, measure the capability drop on each benchmark, and use those drops to set domain weights. Domains that cause large capability drops when removed should receive more weight; domains with negligible impact can be reduced.

For example, if removing code data causes a large drop on programming benchmarks and a smaller drop on general language understanding, you know that code data is important for coding but has spillover benefits for general reasoning. You can use these marginal contribution estimates to find a weight that delivers acceptable coding performance without sacrificing too much on other tasks. The relative magnitudes of the drops across domains give you a gradient that points toward a better recipe.

The ablation approach reveals direct contributions and spillover effects. Code data often improves mathematical and logical reasoning beyond coding tasks, because code requires formal, step-by-step reasoning that transfers to other domains. Scientific papers improve factual grounding beyond scientific question answering because the writing style trains the model to make precise, verifiable claims. Understanding these transfer effects helps you make more informed weight decisions. A domain that has high spillover is often worth weighting more heavily than its benchmark-specific contribution alone would suggest.

One challenge with leave-one-out ablations is that they capture marginal effects at zero weight (what happens when you go from some weight to zero), which may not accurately predict the effect of smaller weight changes around your current recipe. A domain that shows large effect when completely removed might show a smaller marginal effect when you just reduce its weight by 5%. More granular ablation designs that vary each domain's weight over several levels give you a richer picture of the model's sensitivity, at the cost of running more experiments.

When you have a specific target task or application in mind, you can use direct performance on that task as your optimization objective. If you are building a coding assistant, you care primarily about performance on coding benchmarks. If you are building a medical question-answering system, you care primarily about performance on medical factual accuracy tasks. You can design your mixing recipe explicitly to maximize performance on the tasks you care about, using proxy model ablations to evaluate each candidate recipe on those specific tasks.

This approach naturally incorporates the intended use case into the data curation process. It is the most direct path from "what do I want the model to do" to "what data mixture achieves that." The risk is that optimizing too narrowly for your target tasks may degrade performance on tasks you did not explicitly include in your evaluation suite. A model tuned entirely for medical question answering may lose the general language flexibility needed for the instruction-following and explanation tasks that come up in real medical deployments. Using a target-task-focused objective works best when you are building a highly specialized model where the narrow capability range is a feature rather than a limitation.

A complementary view is that diversity itself is a goal. Models trained on homogeneous data, even if that data is very high quality, tend to fail in predictable ways on out-of-distribution inputs. Maintaining a minimum floor of coverage across diverse domains guards against blind spots that are difficult to anticipate in advance.

This manifests practically as constraints on the mixing optimization: no domain should fall below some minimum weight, regardless of what the quality-maximizing or capability-maximizing solution would suggest. The minimum floor values are usually set by heuristic, typically 1 to 5% of total training tokens, rather than formal optimization.

The intuition behind diversity floors is related to the model's ability to handle unexpected inputs at deployment. A model that has never seen certain kinds of text during training will have degraded performance on those inputs in a way that is difficult to predict in advance. By ensuring minimum exposure to each domain, you hedge against unknown unknowns. You do not know exactly what users will ask at deployment. Diversity floors ensure the model has at least encountered a wide range of topics, genres, styles, and registers, even if it has not deeply mastered all of them.

There is also a subtler benefit to diversity: it improves robustness to distribution shift. When training distribution and deployment distribution differ, which they almost always do, models trained on more diverse data tend to generalize better to the deployment distribution because they have been exposed to a wider range of input patterns. A model trained on only web text and code may struggle with the kinds of formal instructions, edge-case phrasings, and unusual query structures that real users produce. Including diverse source types, even at low weight, improves the model's overall robustness.

Both the order in which domains appear during training and the overall proportions matter. A curriculum that starts with high-quality, structured text (books, Wikipedia) and then gradually introduces more varied sources often outperforms random shuffling of the same mixture.

The intuition is that language models benefit from first establishing strong grammatical and factual foundations before being exposed to noisy or stylistically unusual text. Early in training, the model is learning basic language patterns: how sentences are structured, how paragraphs flow, what vocabulary means in context. High-quality text teaches these patterns more efficiently. Later in training, after the basic representations are established, the model can extract useful signal even from noisier sources, because it can filter that signal through the representations it has already built.

If the first billion tokens a model sees are spam or grammatically mangled text, subsequent learning may be harmed more than if those same tokens appear later in training when the model has already formed basic representations. The model's early gradient steps are particularly influential in shaping the feature representations that all subsequent learning builds on. A poorly initialized representation structure is hard to correct later in training, because the model's accumulated knowledge is encoded in terms of those early representations.

The analogy to human education is imperfect but instructive. Children do not learn to read by being handed a dictionary and a noisy social media feed simultaneously. They start with structured, clear text and encounter complexity gradually. There may be a similar principle at work in neural network training, where early gradient steps are particularly influential in shaping the learned representations that all subsequent learning builds on. The analogy breaks down at some point (children are not language models), but the basic intuition that early training has disproportionate influence on final representations is consistent with empirical findings.

This is not universally accepted. Some practitioners argue that shuffling reduces overfitting to the early-seen distribution and that curriculum effects are overstated or not reproducible across different training setups. The evidence is mixed, and the right choice likely depends on the specific domains involved and the scale of training. Curriculum effects may be more significant at smaller scales where the model has less capacity to learn representations from noisy data, and may diminish at very large scale where the model eventually sees enough data to correct early impressions.

The Chinchilla paper established that for a given compute budget, there exists an optimal balance between model size and training tokens. Data mixing adds another dimension to this optimization: for a given compute budget and model size, there is also an optimal allocation of those training tokens across domains.

Recent work by researchers at EleutherAI and elsewhere has explored whether Chinchilla-like scaling laws can be extended to cover domain mixing. The challenge is that the loss decomposition across domains is coupled in complex ways: improving performance on one domain may change the model's representations in ways that help or hurt other domains. The gradient updates from code training interact with the representations built from book training. A joint scaling law that accounts for both compute budget and domain composition would be extremely useful, but has not yet been derived in a fully satisfactory form. The problem is harder than single-domain scaling because the interaction effects are hard to characterize analytically.

A pragmatic approach is to treat the compute-optimal model size and training token count derived from Chinchilla scaling as fixed constraints, and then use DoReMi or proxy model ablations to find the best mixing ratio within that envelope. This decouples the two optimization problems and makes both more tractable. You use scaling laws to determine how big your model should be and how long to train, then use data mixing experiments to determine the best composition of that training budget.

All strategies discussed so far assume a fixed mixing recipe throughout training. Dynamic mixing adjusts proportions during the training run, typically based on how the model's performance evolves. The flexibility to change proportions during training opens up strategies that are impossible with fixed recipes, but it also introduces new risks and engineering complexity.

One simple variant is two-phase training: use a diversified mixture for most of training, then switch to a quality-focused mixture for a final fine-tuning phase. This approach appears in several large model training recipes, where a quality-heavy final phase (sometimes called "cooldown" or "annealing") is applied to improve instruction-following and factual accuracy. The idea is that broad diversity is valuable early in training when the model is building general representations, while quality focus is more valuable late in training when the model is refining its knowledge and aligning its outputs with human preferences. The final phase effectively specializes the model's behavior without discarding the broad capabilities built earlier.

A more sophisticated variant continuously monitors benchmark performance on held-out data and adjusts domain weights in response, essentially running a real-time version of the DoReMi algorithm throughout training. As the model improves on easy domains, weights on those domains are automatically reduced. As the model stagnates on hard domains, weights on those domains increase. This requires infrastructure to evaluate the model periodically during training, which adds engineering overhead but can improve final model quality by keeping the training distribution well-matched to the model's current state.

Dynamic mixing raises an important technical challenge: stability. When you shift domain weights during training, the data distribution the model is learning from changes suddenly. This distributional shift can cause training instability, particularly if the shift is large or abrupt. The optimizer's internal state (momentum in Adam, for example) reflects the gradient statistics of the previous distribution, and sudden distribution changes can make that accumulated state misleading. Practitioners typically apply gradual transitions rather than hard switches, linearly interpolating between the old and new mixing recipe over some number of training steps to give the optimizer time to adapt.

The learning rate schedule interacts with dynamic mixing in ways that are not fully understood. Many training recipes use cosine decay or linear decay of the learning rate, calibrated under the assumption of a fixed data distribution. When the data distribution changes partway through training, the previously calibrated schedule may be suboptimal for the new distribution. This is an active area of research, and current practice relies more on engineering intuition than on principled guidance.