Data Mixing › Data Mixing

Training a large language model from scratch involves more than assembling a massive corpus and pressing go. One of the most consequential decisions you will make is how to mix data from different sources: how much web text versus code, how much scientific writing versus social media, whether to oversample high-quality sources and downsample noisy ones. The mixture you choose shapes what your model knows, how it reasons, and where it fails. Get it wrong, and no amount of compute will fix the resulting gaps.

Data mixing combines empirical science with engineering judgment. Unlike many machine learning hyperparameters, there is no clean analytical formula that tells you the optimal proportions. The problem space is vast, the feedback loop is expensive (you need to train models to see results), and the right answer depends on what you want the model to do. Yet researchers have accumulated enough evidence over the past few years that we now have principled frameworks for thinking about the problem, and a set of algorithmic tools for searching it systematically.

Think of the challenge this way: the internet is not a balanced, curated library. It contains orders-of-magnitude more casual conversation than rigorous technical writing. More marketing copy than scientific analysis. More low-effort content farms than peer-reviewed research. A language model trained on this raw distribution learns to be fluent at what the internet does most: produce casual, chatty, moderately-accurate prose. That may not be what you need. Data mixing is the mechanism for correcting this imbalance.

The stakes are higher than they might first appear. Decisions made at the data mixing stage lock in capability trade-offs that are difficult or impossible to reverse through later fine-tuning. A model that spent most of its pretraining compute reading web crawl will have deeply embedded representations shaped by that distribution. Subsequent instruction tuning can adapt the model's behavior on the surface, but it cannot easily create capabilities that were never built during pretraining. Code generation ability, rigorous multi-step mathematical reasoning, scientific factual grounding, formal writing style: all of these emerge from pretraining, not fine-tuning. Data mixing is where you decide which of these capabilities your model will develop.

This chapter builds directly on the curation pipeline covered in earlier parts of this handbook. By this point, you have raw text that has been crawled from the web (Part XX, chapters 1 through 2), deduplicated using MinHash (chapters 4 through 5), filtered for quality and toxicity (chapters 6 through 7), and scrubbed for PII (chapter 8). What remains is a collection of domain-specific pools, each with different sizes, quality distributions, and relevance profiles. The question now is: how do you sample from these pools when constructing training batches?