Causal Language Modeling › Training Data for CLM

The quality and scale of training data fundamentally shapes what a causal language model learns. Modern LLMs are trained on datasets containing trillions of tokens, carefully curated from diverse sources. Understanding where that data comes from, how it is processed, and what tradeoffs different sources introduce gives you a much clearer picture of why different models behave the way they do.

Common training data sources include:

• Web crawls: Common Crawl and C4, plus similar filtered web scrapes, provide broad coverage of internet text. Heavy filtering removes spam and duplicates, along with low-quality content.
• Books: Project Gutenberg, Books3, and licensed book corpora provide long-form, well-edited text that teaches narrative structure and coherent reasoning.
• Code: GitHub, Stack Overflow, and code documentation help models understand programming languages and technical reasoning.
• Scientific literature: Papers from arXiv and PubMed, as well as Semantic Scholar, provide technical depth and formal reasoning.
• Curated datasets: Wikipedia, news articles, and human-written examples balance quality with scale.

Data quality matters enormously. A model trained on Reddit comments writes like Reddit. A model trained on textbooks writes like textbooks. The mixture of sources directly influences the model's capabilities and style, as well as its failure modes. This is why organizations release detailed data cards alongside their models, and why replicating a model's behavior often requires replicating its data mixture.

Deduplication is critical: repeated text causes models to memorize rather than generalize. Modern pipelines use MinHash, exact substring matching, or embedding-based deduplication to remove near-duplicate documents. Without deduplication, a model may see certain passages thousands of times during training, learning to recite them verbatim rather than learning the underlying patterns. The Pile and RedPajama datasets, for example, both describe extensive deduplication pipelines as a first-class concern, not an afterthought.

Raw text cannot be fed directly to a language model. The first step is tokenization: converting text into a sequence of integer token IDs drawn from a fixed vocabulary. Modern models use subword tokenization schemes like Byte Pair Encoding (BPE) or SentencePiece. These methods split rare words into smaller pieces while keeping common words intact. The word "tokenization" might become "token", "ization" as two subword units, while "the" remains a single token. This balances vocabulary size against coverage: a vocabulary of around 50,000 to 100,000 tokens is standard, large enough to represent most content efficiently but small enough for the output projection layer to remain manageable.

After tokenization, documents are typically concatenated into long streams and then chunked into fixed-length windows, often 2,048 or 4,096 tokens. This ensures the model sees context that spans document boundaries, which helps it learn transition patterns and avoids the artificial constraint of always resetting context at document edges. The resulting data loader simply iterates through these fixed-length windows during training, which is highly efficient for GPU throughput.

Not all data is created equal for training. Raw web data contains enormous amounts of low-value content: spam, boilerplate HTML, generated text, near-duplicate articles. High-quality sources like curated books and peer-reviewed papers improve reasoning and factual accuracy but are available in smaller quantities. Training on only high-quality data produces models that write elegantly but may lack breadth. Training on only web data produces models with broad knowledge but inconsistent quality.

The practical answer is careful mixture weighting. Models like LLaMA 3 weight code-heavy data higher than raw web crawl because code trains logical and structured reasoning that transfers broadly. Wikipedia and books typically receive upsampling, meaning the model sees them multiple times per training run despite being a small fraction of the raw data volume. Getting this mixture right is an active area of research, and small changes in data weights can produce measurable differences in downstream benchmark performance.