Scaling Frontiers › The Data Wall
Language AI HandbookAll chapters Scaling Frontiers › The Data WallOne section · Sections in this chapter4 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Of all the constraints on continued scaling, the data wall is arguably the most immediate and concrete. Unlike compute limits, which require extrapolation across orders of magnitude, or statistical floors, which are theoretically motivated, the data wall is something we can measure directly.
The Finite Internet
The most commonly used training corpora for language models are built from web crawls. Common Crawl, the largest such dataset, contains petabytes of raw HTML representing billions of web pages crawled over many years. After deduplication, quality filtering, and language selection, the usable English text amounts to something in the range of several trillion tokens.
Current compute-optimal training prescriptions require training tokens to scale roughly proportionally to model size. A model with 70 billion parameters, following Chinchilla-optimal ratios, should be trained on roughly 1.4 trillion tokens. A model ten times larger would require 14 trillion tokens. We are already at the point where training frontier models requires a substantial fraction of all high-quality text that exists on the internet.
This creates a fundamental problem: text is produced at a finite rate. Scientific papers, books, code repositories, and high-quality web content grow at a rate of perhaps a few billion tokens per month across all sources combined. A single training run of a large model can consume years of this production in one pass. We cannot simply wait for more data.
Quality Versus Quantity
Not all tokens are equal. The quality of training data has a large impact on model capability, sometimes comparable to orders-of-magnitude differences in dataset size. High-quality data, defined roughly as content written by knowledgeable humans with clear communicative intent, is far scarcer than raw text volume suggests.
A web crawl contains many sources of noise:
- Boilerplate text from templated web pages (navigation menus, legal disclaimers, cookie notices)
- Spam, SEO-optimized content, and machine-generated filler
- Duplicate content republished across thousands of sites
- Low-information text that carries tokens but not knowledge
After aggressive quality filtering, the effective size of even massive web crawls shrinks substantially. The FineWeb dataset from Hugging Face applied rigorous quality filters to Common Crawl and found that only a small fraction of raw data survived as text that met the quality criteria.
This means the data wall hits sooner than raw token counts suggest. The supply of high-quality natural language from human-produced sources may be exhausted within one to two more generations of model scaling, assuming continued exponential growth in compute.
Synthetic Data as a Potential Solution
One response to the data wall is to generate synthetic training data using existing models. This approach has shown real promise in narrow domains. AlphaCode trained on synthetically generated competitive programming problems. Mathematical reasoning data has been augmented with step-by-step solutions generated by larger models. Instruction-following data, as used in RLHF and instruction tuning, is routinely synthetically generated.
However, training models on their own outputs raises a deep theoretical concern: model collapse. When a model trains on data generated by a model of similar or lesser capability, it tends to reinforce the distribution of the generator rather than the distribution of reality. Errors and biases in the generator propagate and potentially amplify in the student.
Model collapse refers to the degradation of model quality that occurs when models are trained predominantly on data generated by other models. The trained model converges to a narrowed distribution that reinforces the systematic biases and errors of its data sources, rather than the diversity and accuracy of human-produced content.
Recent theoretical work has formalized this intuition. Under certain assumptions, training iteratively on model-generated data causes the learned distribution to drift away from the true data distribution at a rate proportional to the fraction of synthetic data in each training corpus. The collapse is not immediate but compounds over successive training iterations.
This does not rule out synthetic data as a tool. But it means that synthetic data works best as an augmentation of, not a replacement for, human-produced content. The most promising approaches use models to produce content in domains where human data is sparse (such as specialized scientific reasoning or multi-step mathematical proofs) rather than replacing general language data.
Data Efficiency and Curriculum Learning
An alternative response to the data wall is to use available data more efficiently. Standard language model training processes each token once or at most a small number of times. This is already a significant departure from human learning, where important concepts are encountered repeatedly in diverse contexts across years of education.
Curriculum learning approaches attempt to improve data efficiency by ordering examples from simple to complex, mimicking a structured educational progression. While early attempts at curriculum learning for language models showed inconsistent results, more recent work has found that careful data ordering and mixing can improve sample efficiency, particularly for reasoning-intensive tasks.
Repeated training on high-quality subsets is another avenue. The Phi family of models from Microsoft demonstrated that smaller models trained exclusively on high-quality educational content could achieve performance comparable to much larger models trained on generic web data. This suggests that data quality improvements, rather than quantity increases, may be a productive direction when the supply of new high-quality data is constrained.
import numpy as np
# Illustrate data efficiency: same compute, different data quality
# We model effective performance as a function of compute and data quality
def effective_performance(compute_budget, quality_factor, base_exponent=0.10):
"""
Simplified model: performance improves with compute and data quality.
quality_factor > 1.0 means higher quality data (more efficient).
Effective tokens = quality_factor * raw_tokens.
"""
effective_compute = compute_budget * quality_factor
loss = 4.0 * effective_compute ** (-base_exponent)
return loss
compute_budgets = np.logspace(21, 25, 100) # 10^21 to 10^25 FLOPs
# Different data quality scenarios
low_quality = [
effective_performance(c, quality_factor=0.5) for c in compute_budgets
]
baseline = [
effective_performance(c, quality_factor=1.0) for c in compute_budgets
]
high_quality = [
effective_performance(c, quality_factor=3.0) for c in compute_budgets
]
very_high_quality = [
effective_performance(c, quality_factor=8.0) for c in compute_budgets
]
Data quality acts as an effective compute multiplier. A training corpus with three times the information density per token behaves like a compute budget three times larger, shifting the scaling curve downward without requiring additional hardware.