Cross-Entropy Loss › Limitations and Impact

Cross-entropy drives modern language model training, but understanding its limitations helps you interpret evaluation results and design better systems. Recognizing what cross-entropy cannot measure is as important as understanding what it can measure.

Cross-entropy is fundamentally a statistical measure that cares only about probability assignment, not semantic content. A model that predicts "The feline rested" instead of "The cat sat" receives the same penalty as predicting a completely unrelated word, even though the first substitution preserves meaning while the second destroys it. Both errors result in identical cross-entropy if the predicted probabilities are the same, despite the first being a reasonable paraphrase and the second being nonsensical.

This limitation becomes most apparent when evaluating generative models. A model that produces a perfect paraphrase of the expected output gets penalized as if it had generated gibberish, because cross-entropy compares against a single reference answer. This is the basic tension in language model evaluation: we train models to maximize probability of specific observed sequences, but we evaluate (and ultimately care about) much broader notions of quality including coherence, factual accuracy, style, and helpfulness.

This limitation motivates the sequence-level evaluation metrics we'll explore in upcoming chapters on BLEU Score and ROUGE Scores, which measure n-gram overlap with reference texts to capture semantic similarity. However, these metrics introduce their own biases, favoring conservative, extractive generation over creative paraphrasing. Neither cross-entropy nor n-gram overlap perfectly captures the fine-grained quality of generated text.

Minimizing cross-entropy does not guarantee well-calibrated probabilities. A model can achieve low cross-entropy while being overconfident on correct predictions and underconfident on errors. This manifests in generation when sampling strategies like Nucleus Sampling rely on probability rankings that may not reflect true likelihoods. A model might assign 99% probability to its top choice even when that choice is wrong half the time, leading to overconfident generation of hallucinations or factually incorrect text.

Calibration errors often appear as systematic biases tied to training data patterns. If the training corpus contains many confident declarative statements and few expressions of uncertainty, the model may learn to be overconfident in its outputs regardless of the actual uncertainty of a given prediction. This is particularly problematic for factual questions: a model may output false facts with the same high confidence as true facts, because the training objective never required distinguishing confident knowledge from uncertain speculation.

Techniques like temperature scaling and label smoothing address calibration to some degree, but the basic disconnect between cross-entropy optimization and calibrated uncertainty remains a challenge, particularly for safety-necessary applications where probability estimates inform decision-making. A medical diagnosis system or legal analysis tool must provide well-calibrated confidence estimates, not just low cross-entropy on training text.

Cross-entropy averages uniformly across positions, treating the first token of a document identically to the thousandth. However, language modeling difficulty varies dramatically with position:

• Initial tokens: High entropy, little context to constrain predictions. The model must predict the first word of a document with essentially no prior information.
• Middle tokens: Lower entropy, rich preceding context. The model can rely on established topic and grammatical structure.
• Long-range dependencies: Cross-entropy may underweight rare but necessary long-range dependencies that require maintaining coherence across paragraphs. A document-level consistency error might affect only a few tokens but represents a serious quality failure.

Modern architectures like those discussed in Long Context sections address this through position interpolation and attention mechanisms, but the uniform averaging of cross-entropy loss does not explicitly prioritize maintaining coherence over arbitrary distances. The loss function treats a pronoun resolution error at position 500 as equally important as a common word prediction error at position 5, even though the former may indicate a more serious failure of understanding.

An important practical consequence is that cross-entropy measured on the first 512 tokens of documents may not reflect model quality on long documents. A model that achieves low perplexity on short contexts may still struggle with long-range coherence, narrative consistency, or multi-step reasoning across extended passages. This limitation motivates dedicated evaluation benchmarks that specifically test long-context understanding, rather than relying solely on standard perplexity metrics that average over all positions equally.

Cross-entropy is computed against the specific tokens observed in training data, which means it depends heavily on the particular tokenization, normalization, and formatting choices applied to the corpus. Two models trained on different tokenizations of the same underlying text may achieve very different cross-entropy scores while capturing equivalent linguistic knowledge. A model trained on text with aggressive lowercasing and punctuation removal will achieve lower cross-entropy on similarly preprocessed test data, but may generate outputs that look different from those of a model trained on the original casing and punctuation.

This dependence also means that cross-entropy conflates learning linguistic patterns with learning the specific stylistic conventions of the training corpus. A model trained primarily on formal text will have lower cross-entropy on formal text than on informal text, even if the formal and informal corpora represent the same underlying language at the semantic level. This is one reason why domain adaptation remains important: fine-tuning on in-domain text reduces cross-entropy for that domain by learning domain-specific vocabulary, facts, and stylistic conventions.

During training, we optimize cross-entropy on next-token prediction. During evaluation for generation tasks, we care about sequence quality, coherence, and factual accuracy. This creates a mismatch between the training objective and downstream utility. We train models to predict the next token, but we evaluate them on their ability to generate helpful, harmless, and honest responses.

This gap becomes stark in instruction-following scenarios. A model trained on web text achieves low cross-entropy by predicting how web text continues, but web text rarely contains the kind of helpful, well-structured responses to explicit questions that we want from an assistant. The model learns the statistical patterns of its training distribution, but that distribution may not align well with the target use case. This explains why instruction fine-tuning and RLHF are necessary: they bridge the gap between the statistical objective (minimize cross-entropy on training data) and the behavioral objective (produce responses that humans find helpful and trustworthy).

Techniques like Reinforcement Learning from Human Feedback address this gap by optimizing for human preferences rather than pure likelihood, but cross-entropy remains the foundational pre-training objective that endows models with linguistic knowledge before alignment tuning. Understanding this gap explains why models can achieve excellent cross-entropy scores on benchmarks while still creating unsatisfactory outputs in practice, and why fine-tuning on human preferences often degrades perplexity while improving utility. A model that learns to say "I'm not sure about that" instead of confidently generating a plausible-sounding but incorrect answer will have higher cross-entropy (since the training data contains many confident assertions) but better real-world reliability.