Reasoning Limitations › Spurious Correlations

Language models learn by observing statistical patterns in training data. This is simultaneously their greatest strength and a major source of weakness. When the training distribution contains systematic associations between surface features and correct answers, models can achieve high accuracy without understanding the underlying reasoning. The features they learn are not causes but correlates.

The distinction matters in practice. A model that has learned causal relationships will generalize: it will perform well when the surface features it was trained on are absent, because it has internalized the deeper structure. A model that has learned spurious correlations will generalize only as far as those correlations extend. Change the distribution, and performance collapses.

A spurious correlation occurs when a model learns to associate a surface-level pattern with an outcome, and that association holds on the training distribution but fails to generalize. These correlations are called "spurious" because they are not grounded in the actual logic of the problem.

Consider a model trained on a natural language inference (NLI) dataset where the task is to determine whether a hypothesis is entailed by, contradicts, or is neutral with respect to a premise. Researchers discovered that models trained on early NLI benchmarks learned that hypotheses containing the word "not" were frequently contradictions, and hypotheses that were very short tended to be entailed. These patterns held often enough in the training data that a model could achieve strong accuracy by exploiting them, without ever learning to reason about logical relationships. The word "not" is not inherently associated with contradiction. It became so because crowdworkers writing contradictory hypotheses unconsciously introduced negation more often when they were expressing opposite meanings.

Another documented example involves question answering. Models trained on datasets like SQuAD learned that answer spans tend to appear near the question words, that the correct span is rarely the very first or last sentence, and that certain syntactic constructions predict answer boundaries. A model exploiting these cues can perform well without understanding the question or the passage. The association between question words and answer proximity is a byproduct of how humans write documents and questions about them, not a logical necessity. In a different document style or question type, the association breaks down.

The NLP community also documented spurious correlations in sentiment analysis. Models trained on movie reviews learned to associate certain directors, actors, and genres with positive or negative sentiment, not because these entities are inherently good or bad but because the training corpus contained systematically skewed reviews. A model that learned "Christopher Nolan" correlates with positive sentiment will misclassify a negative review of one of his films, not because its sentiment analysis is wrong but because its feature representations are wrong.

What makes spurious correlations particularly insidious is that they often cannot be detected from standard test set performance alone. If the test set was sampled from the same distribution as the training set, the spurious correlations that are present in training will also be present in testing. The model's accuracy looks excellent. Only when you construct a test set that deliberately breaks the spurious associations does the gap become visible.

The mechanism is straightforward. Gradient descent optimizes the model to minimize loss on the training data. If surface features are correlated with labels in the training set, the model will use them, because they reduce loss during training. The model has no way to distinguish a spurious correlation from a causal relationship based on the training signal alone. Both look the same from the perspective of the loss function.

This is sometimes described as the model learning to solve the right problem for the wrong reasons. From the model's perspective, there is no wrong reason. Every bit of information that reduces loss is useful, regardless of whether it reflects understanding.

This problem is compounded by dataset construction practices. Many NLP benchmarks were created by crowdworkers who, intentionally or not, introduced systematic patterns. If workers writing NLI hypotheses unconsciously introduce more negations when writing contradictions, the resulting dataset contains a spurious feature. Models trained on it learn the feature, not the logic. This is not a failure of the model. It is a failure of the data generation process.

The phenomenon has a name in causal inference: confounding. The surface feature and the correct label are both caused by a common factor (the type of logical relationship being expressed), which creates a correlation between them that is not itself causal. Removing the confound by controlling for the underlying logical relationship reveals that the surface feature has no real predictive value. Language models, trained purely on observational data, cannot distinguish correlation from causation. They learn everything that is correlated with the labels, confounds and causally relevant features alike.

The magnitude of this problem was demonstrated by "adversarial" NLI datasets like NLI-Stress and HANS (Heuristic Analysis for NLI Systems). These datasets were specifically designed to include examples where common heuristics would fail. Models that achieved 85-90% accuracy on standard NLI benchmarks dropped to near-chance performance (33%) on HANS, because HANS systematically targeted the spurious correlations those models had learned. A model that was thought to understand logical entailment turned out to have learned lexical overlap heuristics. The standard benchmark had not distinguished between these two explanations of performance.

Large language models are not immune to spurious correlations, despite their scale and pretraining on diverse data. They simply acquire them from a much broader and subtler distribution of patterns in the pretraining corpus.

LLMs have been shown to associate specific answer formats with specific question templates, to prefer answer choices that superficially resemble the question wording, and to be influenced by the order of answer choices in multiple-choice settings. These are not reasoning strategies. They are statistical regularities that happened to be useful in the training distribution.

Researchers have also found that LLMs exhibit "sycophantic" biases: they tend to agree with assertions made in the prompt, especially when the prompt provides a confident framing. If you ask a model to evaluate whether a claim is true, and you present the claim as an authoritative statement rather than a question, the model is more likely to agree regardless of the claim's actual truth value. The spurious correlation here is between confident presentation and correctness: in much of the training data, confidently asserted statements are more likely to be true. The model has learned this association and applies it even when it is misleading.

Similarly, LLMs show biases in how they complete reasoning tasks based on the phrasing of the question. Asking "Why is X true?" implicitly presupposes X is true, and models tend to construct supporting arguments rather than questioning the presupposition. Asking "Is X true, and why?" is structurally identical from a reasoning perspective but triggers a more evaluative response. The difference is a spurious correlation between question framing and the type of response expected, learned from thousands of similar examples in training.