Data Mixing › Limitations and Practical Considerations

Data mixing can strongly affect a model, but it comes with practical limitations that matter for real training runs. Understanding where the approach is reliable and where it can mislead you is as important as knowing how to implement it.

The proxy model assumption is the most significant limitation. The claim that mixing rankings transfer across scales holds in aggregate across many published experiments, but individual exceptions exist. A mixture that looks good at 1 billion parameters may behave differently at 70 billion if certain domains interact with scale in non-obvious ways. Some capabilities, particularly the emergent ones that appear suddenly at larger scales, simply cannot be observed at proxy scale at all. The safest practice is to validate proxy results with at least one mid-scale run, say 10 to 20 billion parameters, before committing to a full training budget. This intermediate validation step costs more than proxy-only experiments but substantially reduces the risk of arriving at full scale with a suboptimal recipe.

Benchmark overfitting is another real risk. When you tune your data mixture to maximize scores on a specific set of benchmarks, you are implicitly optimizing for the capabilities those benchmarks measure. If your benchmark suite has blind spots, and all suites do, you may be trading unknown capabilities for better scores on the tests you are running. This is especially concerning if the benchmarks are publicly available and your training data includes documents that discuss or reference the benchmark tasks. A model that has seen discussions of BIG-Bench tasks during pretraining may perform better on those tasks not because of general capability but because of specific exposure. Using a diverse, held-out evaluation suite that is not publicly associated with your project helps reduce but cannot eliminate this risk.

Data mixing decisions made early in a project are expensive to revise. Changing your mixing recipe after training has started is disruptive: the model has developed internal representations tuned to the original distribution, and abruptly changing that distribution can cause instability or unlearning. This creates strong pressure to get the recipe right before large-scale training begins, which in turn justifies the investment in proxy experiments and ablation studies. The cost of a few extra proxy runs early in a project is tiny compared to the cost of a failed full-scale run caused by a preventable mixing error.

The interaction between data mixing and other training decisions is underexplored. Learning rate schedules, batch size, gradient clipping thresholds, and weight decay all interact with the data distribution in ways that are not well understood. Optimizer hyperparameters tuned for one mixing recipe may be suboptimal for another. Running mixing ablations under the exact hyperparameter configuration you plan to use for full training is important but expensive. In practice, most teams run mixing ablations under a standard configuration and then tune optimizer hyperparameters separately, accepting that the optimal mixing recipe and the optimal optimizer hyperparameters may not be exactly co-optimized.

The relationship between pretraining data mixing and subsequent fine-tuning is another important practical consideration. Data mixing during pretraining creates a foundation that all subsequent stages build on. A model with weak code understanding from pretraining will struggle to improve at coding through instruction tuning alone, because instruction tuning cannot easily create representations that were never built during pretraining. The same applies to mathematical reasoning, scientific factual grounding, and multilingual ability. Each of these capabilities needs to be developed during pretraining, before instruction tuning can refine and direct them. Practitioners sometimes underestimate this dependency and are surprised when instruction tuning cannot fix capability gaps left by poor pretraining data mixing.

Finally, the practical difficulty of data mixing is compounded by data availability uncertainty. You do not always know in advance exactly how much data you will have in each domain, or what quality distribution that data will have after filtering. Your recipe may be designed for 500 billion tokens of filtered web text, but if the filtering pipeline removes more content than expected, you may end up with 300 billion. Having contingency recipes for different corpus sizes, and monitoring corpus composition as the data pipeline runs, is important for maintaining control over the final training distribution.