Reasoning Limitations › Robustness Challenges

Robust reasoning would produce consistent outputs when the semantic content of a problem is preserved but its surface form is changed. LLMs fail this criterion in multiple ways, and these failures are not marginal. They reveal that model outputs are sensitive to features that should be irrelevant.

The concept of robustness borrows from engineering, where a robust system continues to function correctly under perturbations that do not change the fundamental problem. A robust bridge remains standing in wind and temperature changes. A robust algorithm produces the same output for semantically equivalent inputs. By this standard, LLMs are not robust: they are highly sensitive to surface features that carry no semantic content.

This sensitivity has practical consequences that go beyond academic interest. Deployed LLMs are queried by users who phrase questions in many different ways. If the model's performance is sensitive to phrasing, then user experience is inconsistent and model behavior is difficult to predict. Prompt engineers spend substantial effort finding phrasing that triggers reliable model behavior, not because different phrasings communicate different intent but because different phrasings activate different learned patterns.

Two questions with the same meaning should receive the same answer. This is a basic requirement for robust reasoning. LLMs violate it regularly. Paraphrasing a question, changing active to passive voice, or using synonyms can change the model's output, sometimes substantially.

Consider a factual question: "Who wrote the novel 1984?" versus "1984 was written by whom?" versus "What author published 1984?" These three questions have the same semantic content. A model with a stable internal representation of the fact "George Orwell wrote 1984" should answer all three identically. In practice, researchers have found that LLMs can give different answers to factual questions depending on phrasing, especially for less common facts or when the phrasing is unusual.

The sensitivity to paraphrase extends to instruction following. Providing the same instruction in different surface forms can produce qualitatively different model behaviors. This is problematic for any application where the precise phrasing of prompts matters for output quality, since users cannot know which phrasing is "optimal" and the optimal phrasing may shift across model versions. A user who has carefully engineered a prompt for one version of a model may find that the same prompt performs differently after a model update, requiring the engineering effort to begin again.

The mechanism behind paraphrase sensitivity is the same as the mechanism behind spurious correlations. The model has learned associations between specific phrasings and response patterns. "Who wrote X?" may trigger a different learned pattern than "X was written by whom?" even if the underlying fact being queried is the same. The model's response depends on which patterns its learned representations activate, and that depends on surface form.

Irrelevant information included in a problem can significantly disrupt LLM reasoning. Adding a sentence to a math problem that is related to the problem's topic but irrelevant to its solution can reduce accuracy substantially. This was demonstrated in the Grade School Math (GSM) benchmark, where adding a single irrelevant sentence to each problem caused significant accuracy drops in several models.

This sensitivity to distractors suggests that LLMs are not isolating the relevant information and reasoning exclusively from it. They are influenced by the total distribution of content in the context, even when that content is logically irrelevant to the question.

The distractor effect is strongest when the distractor contains numbers or quantitative information similar to those in the problem. A math problem that asks about calculating the area of a rectangle might be disrupted more by a sentence mentioning an irrelevant numerical value than by a sentence about an unrelated topic. This suggests that the model's reasoning process integrates all numerical content in the context rather than identifying which numbers are relevant to the question.

The downward trend with each added distractor reveals a fundamental property of how LLMs process context. Unlike a human solver who can identify which information is relevant and set the rest aside, the LLM's attention mechanism distributes attention across all tokens in the context. Distractors compete for attention budget with the relevant problem content, potentially drawing the model away from the steps needed to solve the problem correctly.

The format of the prompt itself, independent of its content, influences LLM outputs. The number of few-shot examples, whether examples are ordered from easy to hard or randomly, whether a question is framed as "solve" versus "explain and solve," and even the presence of whitespace can affect outputs. This sensitivity has been extensively documented and has practical implications: prompt engineering is often not about communicating intent more clearly but about finding the surface-level patterns that trigger the model's best-performing behavior for a task type.

A particularly well-studied case is the effect of few-shot example quality and ordering. Research has shown that providing the same few-shot examples in different orders can change accuracy by 10 or more percentage points. The last few-shot example before the test question has a disproportionate influence on the model's output, a recency bias that reflects the sequential nature of autoregressive generation. This means that carefully selected few-shot examples can be undermined by poor ordering, and the best-performing configuration varies unpredictably across tasks and models.

The sensitivity extends to the phrasing of instructions. "Solve this step by step" and "Let's think step by step" have been shown to produce different outputs even though they convey the same directive. The phrase "Let's think step by step" was specifically identified as an effective chain-of-thought trigger in the original zero-shot chain-of-thought paper (Kojima et al., 2022). The effectiveness of this phrase reflects the statistical regularities in training data: this phrasing was associated with careful, step-by-step reasoning in the documents the model was trained on. Alternative phrasings that mean the same thing to a human reader may not activate the same learned behavior.

A particularly revealing robustness test is consistency checking: asking the same question in different forms and testing whether the answers are mutually consistent. "Is Paris north of Rome?" and "Is Rome south of Paris?" should receive the same answer. In practice, LLMs often give inconsistent answers to logically equivalent questions, especially when the surface form differs substantially.

This inconsistency is difficult to reconcile with a model that knows a fact or has reasoned systematically through a problem. A model that truly knows the spatial relationship between Paris and Rome should answer both questions the same way. Inconsistent answers reveal that the model is not consulting a stable internal representation of the fact but is generating responses based on the statistical associations triggered by each specific query.

The consistency problem extends to multi-step reasoning. If you ask a model to solve a problem and then ask a follow-up question that should be directly answerable from the solution it just provided, the model may give an inconsistent answer. "What is 15 times 23?" might yield "345." The follow-up "Is 345 the correct product of 15 and 23?" might yield "No, 15 times 23 is 350." This is not an arithmetic error in the second response alone; it is a consistency failure. The model generated a number in the first response and then, when asked to evaluate it, generated a different number without recognizing the conflict.