Data Mixing › Summary

Data mixing turns a raw collection of training corpora into a structured learning curriculum. The decisions made here propagate forward through the entire model development process, shaping every capability the final model will have. Getting these decisions right requires understanding both the principled frameworks developed by the research community and the practical constraints of real data pipelines.

The key ideas from this chapter are:

• Domain proportions determine how often the model encounters each type of content. Proportional sampling by corpus size is almost never the right answer because corpus sizes do not reflect the relative value of different domains. Carefully curated recipes that deliberately over-represent high-quality, capability-relevant sources consistently outperform naive proportional baselines.

• Quality weighting allows you to boost high-quality sources within a domain, either through trained classifiers, source-based heuristics, or perplexity scores from small reference models. Combining multiple quality signals with a continuous weighting scheme is more robust than applying hard thresholds.

• Oversampling can compensate for data scarcity in high-value domains, but has diminishing returns beyond roughly 2 to 4 times repetition. Beyond that threshold, memorization begins to outweigh generalization, and gradient signal from repeated examples degrades toward zero.

• Proxy model ablations make it feasible to explore the mixing space without committing full compute. The key assumption is that relative performance rankings transfer across model scales, which holds in aggregate but has individual exceptions. Intermediate-scale validation reduces the risk of scale-transfer failures.

• DoReMi formalizes the mixing optimization as minimax distributionally robust optimization, automatically finding weights that minimize worst-case performance across domains through an exponentiated gradient update. It shifts mixing from a manual art toward a principled algorithmic process.

• Dynamic mixing and curriculum effects can further improve over fixed recipes by adapting domain weights to the model's evolving performance profile, at the cost of additional infrastructure and the risk of training instability during distribution transitions.

• Cross-domain spillover means that some domains contribute capabilities beyond their primary domain. Code data improves general reasoning. Scientific writing improves factual grounding. Understanding these interactions helps you build more efficient recipes that use spillover effects rather than treating each domain independently.

The next chapter covers synthetic data generation, which adds a new dimension to the mixing problem: rather than just mixing existing sources, you can generate new training examples to fill capability gaps that no existing corpus adequately covers.