Data-Constrained Scaling › Data Repetition Effects

When unique data runs out, the natural response is to repeat existing data, training on the same examples multiple times across different epochs. But repetition is not free. Each additional pass through the same data yields diminishing returns, and excessive repetition can actively harm model quality. To understand why, consider what a neural network learns during training and how that learning changes when the network has already seen an example.

Muennighoff et al. (2023) conducted the first systematic study of data repetition in language model scaling, training models ranging from 25 million to 8.7 billion parameters with varying degrees of repetition. Their findings confirmed what practitioners suspected:

Repeated data is worth less than unique data. A model trained on 100 billion unique tokens significantly outperforms one trained on 25 billion tokens repeated four times, even though both see the same total token count. The gap widens as repetition increases. This result challenges a naive view where tokens are interchangeable units of learning signal. Instead, the novelty of information matters significantly.

Returns diminish rapidly. The first repetition (seeing data twice) captures perhaps 70-80% of the value of unique data. The fourth repetition might capture only 30-40%. Beyond 8-16 repetitions, additional passes contribute almost nothing to final performance. This pattern reflects how gradient-based learning works: early passes make large updates that capture the broad structure of the data, while later passes can only make small refinements to already-learned patterns.

Larger models suffer more from repetition. A 175 million parameter model tolerates 16 epochs with modest degradation. An 8 billion parameter model shows measurable harm after just 4 epochs. This makes intuitive sense: larger models have more capacity to memorize, and they extract more information per pass, leaving less to learn on subsequent passes. A small model might need multiple exposures to fully capture a complex pattern, while a large model grasps it on the first pass, making subsequent passes redundant.

Researchers formalize repetition effects through what we might call an "epoch penalty": a multiplier that discounts the effective value of repeated tokens. If unique tokens have a value of 1.0, the $n$-th repetition of a token has value $f(n) < 1$.

Before diving into the mathematical formulation, let's build intuition for what we're trying to capture. Imagine reading a textbook chapter for the first time. You absorb enormous amounts of new information. Reading it a second time still helps: you catch details you missed, reinforce your understanding, and make new connections. By the fifth reading, you're mostly confirming what you already know. By the twentieth reading, you're gaining almost nothing. The mathematical framework below captures exactly this diminishing pattern.

The functional form of this penalty follows an exponential saturation pattern (despite common references to "power law" behavior in scaling contexts, the repetition penalty specifically uses exponential decay). Let $D$ be the number of unique tokens and $R$ be the number of epochs (repetitions). The effective number of tokens $D_{\text{eff}}$ is lower than the naive product $D \times R$:

$$D_{\text{eff}} = D \times g(R)$$

where:

• $D_{\text{eff}}$: the effective number of training tokens after accounting for repetition penalty
• $D$: the number of unique tokens in the dataset
• $g(R)$: a function that captures the diminishing returns from repetition, mapping epochs to an effective multiplier

The key insight here is that $g(R)$ grows more slowly than $R$ itself. If repetition had no penalty, we would have $g(R) = R$, and effective tokens would simply equal total tokens processed. The entire purpose of introducing $g(R)$ is to quantify how much value we lose to repetition.

Muennighoff et al. found that $g(R)$ is well-approximated by:

$$g(R) = \frac{1 - e^{-\alpha R}}{1 - e^{-\alpha}}$$

where:

• $R$: the number of epochs (times the data is repeated)
• $\alpha$: a decay constant controlling how quickly returns diminish (empirically fitted to approximately 0.1)
• $e^{-\alpha R}$: exponential decay term that approaches zero as repetitions increase
• $e^{-\alpha}$: the single-epoch decay factor, used to normalize the function so that $g(1) = 1$

Let's unpack why this particular mathematical form makes sense. The numerator $1 - e^{-\alpha R}$ is a classic saturation curve. It starts at zero when $R=0$ and approaches 1 as $R$ grows large. This captures the intuition that no matter how many times you repeat the data, you can only extract a finite amount of information from it. The denominator $1 - e^{-\alpha}$ is simply a normalization constant that ensures $g(1) = 1$, meaning one pass through the data has full value.

The parameter $\alpha$ controls the rate of saturation. When $\alpha$ is small (as the empirically fitted value of 0.1 suggests), the saturation happens slowly. You can repeat data many times before hitting severe diminishing returns. If $\alpha$ were larger, returns would diminish more quickly, and even modest repetition would waste computational resources.

This functional form has two useful properties. First, $g(1) = 1$, meaning there is no penalty for seeing data once. A single pass through the data contributes its full value. Second, $g(R)$ asymptotes to a finite limit as $R \to \infty$, meaning there is a maximum effective dataset size regardless of how many times you repeat the data. Specifically, as $R$ grows without bound, the exponential term $e^{-\alpha R}$ vanishes, and $g(R)$ approaches $1/(1 - e^{-\alpha})$. For $\alpha = 0.1$, this asymptotic limit is approximately 10.5, meaning that infinite repetition of your data is equivalent to having roughly 10.5 times as much unique data. This asymptotic behavior captures the intuition that you cannot extract infinite value from finite data through repetition alone.

Incorporating repetition effects into the Chinchilla framework requires modifying the loss function. This modification is an important conceptual shift. We move from treating all tokens as equivalent to recognizing that the information content of tokens depends on whether we've seen them before.

Recall that the basic Chinchilla loss takes the form:

$$L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E$$

where:

• $L(N, D)$: the expected loss (cross-entropy) for a model with $N$ parameters trained on $D$ tokens
• $N$: the number of model parameters
• $D$: the number of training tokens
• $A, B$: empirically fitted scaling coefficients
• $\alpha, \beta$: exponents controlling how loss decreases with scale (approximately 0.34 and 0.28 respectively)
• $E$: irreducible loss representing the entropy of natural language (the theoretical minimum loss even with infinite model size and data)

The formula captures a fundamental tradeoff: the first term $A/N^\alpha$ represents the capacity limitation of the model (reducible by adding more parameters), while the second term $B/D^\beta$ represents the data limitation (reducible by adding more training tokens). The irreducible loss $E$ represents patterns in language that are inherently unpredictable. Understanding this decomposition is essential because data-constrained scaling fundamentally alters how we can address the data limitation term.

Think of the first term as "how much the model could learn but lacks the capacity to store," and the second term as "how much the model could store but has not yet seen." In the standard Chinchilla framework, we can freely trade off between these by adjusting model size and data quantity. But when data is constrained, the second term becomes partially frozen, and we can only reduce it so much before running out of unique information.

When data is repeated, we replace $D$ with $D_{\text{eff}}$:

$$L(N, D, R) = \frac{A}{N^\alpha} + \frac{B}{D_{\text{eff}}^\beta} + E$$

where:

• $L(N, D, R)$: the expected loss for a model with $N$ parameters trained on $D$ unique tokens for $R$ epochs
• $D_{\text{eff}}$: the effective token count after accounting for repetition penalty, computed as $D \times g(R)$
• $N, A, B, \alpha, \beta, E$: retain their meanings from the standard Chinchilla loss above

This substitution changes the loss curve. In the original formulation, increasing $D$ always improves loss (with diminishing returns governed by $\beta$). In the modified formulation, increasing epochs $R$ improves $D_{\text{eff}}$, but the improvement saturates. Eventually, the marginal gain in effective data from additional epochs becomes negligible, while the computational cost continues to grow linearly.

This modification has measurable implications for compute-optimal training. Under the original Chinchilla law, doubling compute should roughly double both $N$ and $D$. Under data-constrained scaling, once $D$ is fixed at the available unique data, additional compute should go primarily toward larger models with more repetitions. But we must be aware that the marginal value of additional training decreases. The key tradeoff becomes: given fixed unique data $D$, how should we allocate compute between model size $N$ and number of epochs $R$? The answer depends on solving for the minimum of $L(N, D, R)$ subject to the compute constraint $C = 6NRD$ (approximately 6 FLOPs per parameter per token per epoch).

This optimization problem no longer has the elegant closed-form solution of the original Chinchilla analysis. Instead, practitioners must numerically search over possible allocations, as we demonstrate in the code implementation section below.