Hyperparameter Selection › Troubleshooting Hyperparameter Problems
Language AI HandbookAll chapters Hyperparameter Selection › Troubleshooting Hyperparameter ProblemsOne section · Sections in this chapter11 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Even with careful selection, training runs sometimes go wrong. Diagnosing whether a problem is hyperparameter-related versus data-related versus hardware-related is a skill that comes with experience. Here are the most common failure patterns and their likely causes.
Training loss diverges early (within the first 5% of steps). This almost always indicates a learning rate that is too large. The loss may spike suddenly or oscillate with increasing amplitude. Reduce the learning rate by 3-5x. If the problem persists even at very low learning rates, check for data corruption (NaN values in the training data, or sequences with degenerate token distributions) and verify that gradient clipping is enabled.
Training loss decreases but plateaus far above expected scaling law predictions. This indicates a learning rate that is too small, an inappropriate schedule, or data quality issues. If gradient norms are consistently near zero, the learning rate is likely too small. If gradient norms are healthy but the loss plateau is unexpectedly high, the data may contain too many duplicates or low-quality examples that dilute the signal.
Training is stable but validation loss begins increasing after many steps. This is overfitting, most common when the training dataset is small relative to model size or training duration. Increase weight decay, reduce the number of training steps, or augment the dataset. For fine-tuning scenarios specifically, also consider reducing the learning rate.
Loss spikes periodically during mid-training. Periodic spikes are often caused by gradient outliers from specific training examples (corrupted documents, very long sequences that overflow attention), or by numerical precision issues with mixed-precision training. Setting gradient clipping to a lower threshold (0.5 instead of 1.0) and checking the data pipeline for anomalies usually resolves this.
The proxy model's optimal learning rate does not work at the target scale. If you are not using muP, this is expected: you need to scale the learning rate by the width ratio. If you are using muP, verify that the muP initialization and learning rate scaling rules were applied correctly to all parameter types, including the output projection and embeddings, which are easy to miss.