Data-Constrained Scaling › The Data Bottleneck
Language AI HandbookAll chapters Data-Constrained Scaling › The Data BottleneckOne section · Sections in this chapter2 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To understand the data constraint, consider the scale of modern training corpora. The Pile, a widely-used open dataset, contains approximately 825 GB of text, yielding roughly 300 billion tokens. Common Crawl, the largest web scrape, provides about 3 trillion tokens after deduplication and filtering. These numbers sound enormous until you apply Chinchilla-optimal ratios.
A 1 trillion parameter model, following Chinchilla guidelines, would need approximately 20 trillion training tokens. That is roughly seven times the entire deduplicated Common Crawl. A 10 trillion parameter model would require 200 trillion tokens, far exceeding all text ever produced by humanity.
The point at which compute-optimal training would require more unique, high-quality tokens than are practically available. Estimates suggest this wall may be reached between 2026 and 2032 for frontier models, depending on data quality requirements.
Villalobos et al. (2022) systematically estimated when various data sources would be exhausted. Their analysis revealed a sobering timeline:
- High-quality text (books, academic papers, curated web content): Exhausted by 2026-2028 at current scaling rates
- All web text: Exhausted by 2030-2032
- Multimodal data (images, video): Provides more runway but with different utility
This is already a practical concern. Training runs for GPT-4 and similar models already approach or exceed the high-quality data ceiling. The question is no longer whether we'll hit the data wall, but how to train effectively when we do.