Training Stability › Data Quality and Its Effect on Stability

The role of data quality in training stability is substantial but often underemphasized relative to architectural and optimizer choices. This section addresses the data-centric view of stability.

Large language model pretraining relies on web-scraped corpora containing text from billions of web pages. These corpora are extraordinarily diverse, which is a feature for coverage. But they also contain a long tail of document types that produce unusually high loss during training.

Documents that are problematic from a stability perspective tend to have one or more of these properties:

• High repetition: Documents consisting of repeated phrases or paragraphs produce high-entropy gradients because the model cannot rely on context to predict the next token. The gradient is dominated by all-or-nothing predictions.
• Encoding artifacts: Documents with non-UTF-8 encoding that has been force-converted produce token sequences that look like random byte sequences. The cross-entropy loss for random sequences is much higher than for natural language, producing proportionally large gradients.
• Code with many special characters: Dense code (especially assembly, compiled bytecode, or minified JavaScript) has very different token statistics from natural language, producing high loss if the model was mostly trained on natural language.
• Template-generated text: Spam, boilerplate, and SEO-optimized text often contain long runs of repeated content with very predictable structure, which the model over-learns quickly and then shows very high loss on the small distinctive tokens.

The practical fix is to filter these document types during data preprocessing, before they enter the training corpus. The most effective filtering signals are:

• Perplexity under a small reference language model (very high perplexity indicates unusual text)
• Repetition ratio (fraction of n-grams that appear more than once in the same document)
• Character diversity (very low character diversity indicates encoding artifacts or repeated text)

Curriculum learning is the practice of ordering training examples from easier to harder, rather than presenting them in random order. From a stability perspective, curriculum learning is appealing because it ensures that the model encounters well-represented, high-frequency patterns first, building a strong foundation before seeing outlier data.

The empirical evidence for curriculum learning improving stability is suggestive but mixed. Some large-scale training experiments have found that starting with documents of moderate complexity and gradually introducing outlier documents reduces early training instability. Others have found no significant benefit, perhaps because the gradient clipping and warmup techniques already handle the outlier batches adequately.

A more targeted version of curriculum learning, sometimes called "data pacing," focuses specifically on the outlier tail rather than full ordering. The idea is to hold back the top 5-10% highest-loss documents for the second half of training, when the model is better calibrated and can learn more reliably from difficult examples. This approach has the advantage of being easy to implement (filter documents by initial loss under a small model) and addresses specifically the data-driven cause of stability problems.

A powerful but computationally expensive technique is dynamic data filtering: computing the loss for each document in a candidate batch before applying the gradient, and discarding documents whose loss exceeds a threshold. This ensures that no single document with extreme loss corrupts the gradient estimate.

The computational cost is one forward pass per batch to compute per-document losses. For large models, this doubles the per-step compute. But for models where stability is critical and compute is available, it can substantially reduce spike frequency. Some practitioners implement a softer version: re-weight documents by the inverse of their loss (so high-loss documents contribute less gradient signal) rather than discarding them entirely, preserving some learning signal from difficult documents.