LLaMA Architecture › Training Data
Language AI HandbookAll chapters LLaMA Architecture › Training DataOne section · Sections in this chapter5 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
LLaMA's training data strategy prioritizes quality and public availability over scale. While GPT-3 trained on 300 billion tokens and later models reached trillions, LLaMA's original training used 1.4 trillion tokens, carefully curated from high-quality sources. This data efficiency enabled strong performance without proprietary datasets.
LLaMA Training Data Sources ================================================================================ Source Percentage Tokens Description -------------------------------------------------------------------------------- CommonCrawl 67.0% 0.94T Filtered web text with perplexity scoring C4 15.0% 0.21T Colossal Clean Crawled Corpus GitHub 4.5% 63B Public code repositories Wikipedia 4.5% 63B 20 languages, encyclopedic text Books 4.5% 63B Project Gutenberg and Books3 ArXiv 2.5% 35B Scientific papers (LaTeX stripped) StackExchange 2.0% 28B Technical Q&A from 28 sites -------------------------------------------------------------------------------- Total 100.0% 1.40T
Data Quality Over Quantity
The CommonCrawl data, which forms the bulk of training, underwent aggressive filtering. The team used a classifier trained to distinguish Wikipedia articles (high quality) from random web pages (lower quality). Pages scoring below a threshold were discarded. This perplexity-based filtering removed roughly 60% of raw web data, prioritizing text that resembles well-written content.
Deduplication further improved data quality. Duplicate content, common on the web, provides no new information while consuming training compute. The team used near-duplicate detection to remove repeated content both within and across sources.
The reasoning behind aggressive filtering is subtle but important. When you train a language model, every token you show it is a teaching example. Showing the model low-quality text teaches it patterns of poor writing and unreliable text. Showing it high-quality text teaches it well-structured reasoning, clear exposition, and factually accurate statements. The model has no way to distinguish "I should learn this" from "I should discard this" during training. Everything contributes to its learned distribution. This means the composition of training data directly shapes the model's output style and reliability, not just its knowledge base.
The decision to curate aggressively had a measurable effect. Researchers who later trained models on unfiltered CommonCrawl versus filtered CommonCrawl observed consistent differences in output quality and factual consistency. The filtering pipeline is as much a part of LLaMA's recipe as the architectural choices, though it receives less attention in discussions of the model.
Tokenization
LLaMA uses a SentencePiece tokenizer with byte-pair encoding (BPE). The vocabulary contains 32,000 tokens, including special tokens for unknown words and sequence boundaries. Critically, the tokenizer splits all numbers into individual digits, preventing the model from memorizing specific numbers as single tokens.
# Demonstrating number handling in LLaMA tokenization
# LLaMA splits numbers into individual digits
example_tokenizations = {
"2023": ["2", "0", "2", "3"], # Each digit separate
"3.14159": [
"3",
".",
"1",
"4",
"1",
"5",
"9",
], # Digits and decimal separate
"The model has 7B parameters": [
"The",
"model",
"has",
"7",
"B",
"parameters",
],
}LLaMA Tokenization Examples (conceptual) ============================================================ Input: '2023' Tokens: ['2', '0', '2', '3'] Count: 4 tokens Input: '3.14159' Tokens: ['3', '.', '1', '4', '1', '5', '9'] Count: 7 tokens Input: 'The model has 7B parameters' Tokens: ['The', 'model', 'has', '7', 'B', 'parameters'] Count: 6 tokens
This digit-level tokenization helps with arithmetic and numerical reasoning. Instead of treating "2023" as an opaque symbol, the model sees the constituent digits and can learn operations over them.
To understand why this matters, consider how a model that tokenizes numbers as single units would handle arithmetic. If "1024" is one token and "512" is another token, the model can only answer "what is 1024 / 2?" correctly if it saw that exact question during training. It has no way to generalize, because there is no compositional structure to exploit. When digits are split, the model can learn that the digit "1" in a particular position represents a particular value, that subtraction involves borrowing across digit boundaries, and so on. The token-level structure matches the mathematical structure of numbers, letting generalization.
Training Duration and Epochs
Unlike many large models that train for a single epoch over their data, LLaMA trained for multiple epochs on the smaller, higher-quality datasets while limiting exposure to filtered web data:
This strategy amplifies the influence of high-quality data. A Wikipedia article seen 2.5 times contributes more to the model's learned representations than a random web page seen once. The approach recognizes that not all tokens are equally valuable for learning.
The multi-epoch strategy for high-quality sources runs counter to conventional wisdom that language models should see each training example only once to avoid memorization. For very large, diverse datasets, this guidance is approximately correct. But when a source like Wikipedia contains only a few billion tokens, limiting it to one epoch means the model encounters those high-value patterns only briefly relative to the billions of web tokens. Upsampling allows the model to learn more thoroughly from text that exemplifies the behavior we want: clear prose, accurate facts, and coherent argumentation. The trade-off is some risk of overfitting to the upsampled sources, but at the scale of billions of tokens per source, that risk is minimal.