Research Directions › Open Problems in Language AI

The following problems represent areas where current methods fall short in ways that matter for real applications. They are not merely performance gaps that scale will close; they reflect architectural, data, or training limitations that require new ideas.

The distinction between "problems that scale will solve" and "problems that require new ideas" is important and often contested. The scaling hypothesis suggests that many apparent limitations of current models are simply the result of insufficient parameters, training data, or compute, and will disappear as these quantities increase. There is empirical support for this view in some domains: models that could not pass grade-school math tests at 7B parameters pass them at 70B, and models that struggled with basic coding tasks at one scale handle significantly harder tasks at a larger scale. But the same scaling that closes some gaps seems to leave others stubbornly open, and understanding which problems fall into which category is itself a major research question.

Large language models are trained to predict text. They are not trained to distinguish between things they know with evidence and things they are confabulating from patterns. The result is hallucination: the generation of plausible-sounding but incorrect information presented with the same surface confidence as accurate information.

Hallucination is not a single phenomenon. Researchers distinguish between several types, each with different characteristics and likely different causes. Factual hallucination produces claims that contradict verifiable facts, such as a model asserting that a historical figure was born in the wrong city or that a scientific paper reached conclusions it did not reach. Faithfulness hallucination produces summaries or answers that contradict the source document provided in context, even when the correct answer is directly stated in the passage the model was given. This second type is particularly troubling because it reveals that models do not always faithfully use context even when it is immediately available. Reasoning hallucination produces conclusions that do not follow from the stated premises, where the individual steps look plausible but the chain of inference contains a subtle error.

Each type likely has different causes and requires different interventions. Factual hallucination may stem from the model's inability to reliably distinguish high-confidence from low-confidence knowledge stored in its weights. Faithfulness hallucination may reflect attention mechanisms that do not always prioritize relevant context over memorized associations. Reasoning hallucination may reflect the absence of an explicit verification step during generation.

Retrieval-augmented generation (RAG), which we covered in detail in the RAG chapter, addresses the knowledge access problem by giving models external memory. But RAG does not fully solve hallucination: models still hallucinate even when the relevant passage is in context. In controlled experiments where researchers confirmed that the correct answer appeared verbatim in the provided context, significant fractions of model responses still contained inaccuracies. The core problem is that the model's generation process does not natively separate "I found this in the provided sources" from "I am generating from prior training." Making this distinction architectural rather than behavioral is an active research area.

A related challenge is calibration: models should express uncertainty in proportion to their actual accuracy. A model that hedges with "I am not certain, but I believe..." when it is likely to be wrong, and states facts without hedging when it is likely to be right, is substantially more useful than one that expresses uniform confidence in everything it says. Current models are often miscalibrated in both directions: they hedge on questions they should answer confidently, and express confidence on questions where they are frequently wrong. Calibration is particularly poor in domains far from the training distribution, in specialized technical domains, and on questions about recent events after the model's knowledge cutoff. The deep challenge here is that a model cannot directly inspect its own confidence; it can only learn to produce calibrated language from training signals that reward appropriate hedging.

Human learners generalize in a particular way: having learned the meaning of "compose a sonnet" and "translate into French," they can execute "translate this sonnet into French" without needing to see that specific combination during training. This compositional generalization, the ability to combine known operations and concepts in novel arrangements, is natural for humans but inconsistently achieved by language models.

Standard benchmarks often evaluate models on distributions similar to their training data. When researchers construct tests that require combining concepts in unfamiliar combinations, model performance drops substantially, even on models that achieve near-human scores on standard evaluations. The SCAN, COGS, and CFQ benchmarks were designed specifically to probe systematic generalization, and they reveal consistent weaknesses. A model might correctly follow the instruction "jump twice" and correctly follow "walk and look," but fail on "jump twice and look," even though this combination requires nothing beyond what the model has already demonstrated.

This failure connects to a deeper question about what language models learn. Do they acquire abstract rules about how concepts combine, the kind of structure that would allow arbitrary recombination of known elements? Or do they learn surface statistics that happen to generalize within the distribution of their training data, but break down when the combination of concepts is new? The evidence suggests the answer is closer to the latter than researchers would like. Models often perform as if they have memorized a very large set of (input, output) mappings and are interpolating between them, rather than applying compositional rules that would generalize to any input expressible in the grammar of the task.

Making compositional generalization reliable requires either fundamentally different architectures that encode compositional structure explicitly, training objectives that specifically reward compositional behavior, or data regimes that force models to learn composition rather than interpolation. None of these directions has yet produced a reliable solution that matches human-level systematic generalization across a broad range of task types.

As we discussed in the reasoning chapters, chain-of-thought prompting substantially improves model performance on problems that require multiple inference steps. But this improvement is brittle in ways that matter practically. Models fail on problems that require small changes to the reasoning chain, commit systematic errors in specific arithmetic and logical domains, and sometimes produce reasoning traces that appear sound but contain subtle errors that standard evaluation does not catch.

The brittleness of chain-of-thought reasoning is well-documented. A model that correctly solves a multi-step word problem may fail on the same problem with different surface features: different numbers, different character names, or a slightly different phrasing that changes the logical structure of the problem without changing its difficulty. This suggests that models are pattern-matching against surface features of common problem types rather than consistently applying the underlying reasoning procedure.

The deepest version of this problem is the gap between process and outcome. A model that produces a correct final answer via a flawed reasoning path is not reliably correct on related problems: the correct answer in that instance may have been reached by coincidence or by the model's statistical priors, not by the stated reasoning. Evaluating the correctness of the reasoning process as well as the final answer, requires either human evaluation, which is expensive and slow, or a separate verifier model, which introduces its own accuracy limitations. Process reward models, covered in the advanced training chapters, are one proposed solution, but they require reasoning-step-level supervision that is costly to obtain at scale.

A further complication is that models can learn to produce reasoning traces that satisfy verifiers without performing the reasoning those traces describe. Just as a student might write out the steps of a mathematical proof while following a memorized template rather than understanding why each step follows, a language model can learn to produce text that looks like valid reasoning without the computation reflecting valid reasoning steps. Distinguishing valid from simulated reasoning is one of the hardest problems in current interpretability research.

Modern language models support context windows of hundreds of thousands of tokens. But having a large context window does not guarantee faithful use of long contexts. Research has documented the "lost in the middle" phenomenon: information placed in the middle of a long context is retrieved less reliably than information at the beginning or end, even when the context fits comfortably within the model's window.

The magnitude of this effect is substantial. In controlled experiments where a single factual question could be answered from a specific passage within a long document, models showed significantly lower accuracy when the relevant passage was in the middle of the document compared to the start or end. The degradation was not gradual but showed a characteristic U-shape: high performance for early and late positions, substantially lower performance for middle positions, even with passages that the model could easily answer from if presented alone.

This suggests that models have learned to pay disproportionate attention to certain positions, a learned bias from pretraining on documents where key information often appears at the start or end. Many natural documents do have this structure: introductions summarize key points, conclusions recapitulate them, and supporting evidence sits in between. A model that has internalized this structure will be poorly suited to tasks that require extracting information from arbitrary positions in long inputs.

Addressing this requires either data curation changes during pretraining to include documents where key information appears in non-prominent positions, architectural changes to how position interacts with attention, or both. Some recent work on positional encodings specifically addresses the locality bias that causes models to disproportionately attend to nearby tokens, but general solutions that maintain faithful recall at arbitrary positions across very long contexts remain an open problem.

Long-context faithfulness also raises evaluation challenges. Standard benchmarks test short-context capabilities. Evaluating whether a model faithfully processes a 100,000-token document requires task designs that specifically test information from the middle of the document, against distractor information elsewhere in the same document. Needle-in-a-haystack evaluations, where a specific fact is embedded in a long irrelevant context and the model is asked to retrieve it, are a common approach, but they measure a narrow capability that may not generalize to more complex long-context reasoning tasks.

The benchmark evaluation problem is arguably meta-level, but it shapes every other research direction. A model that scores highly on a benchmark is not necessarily solving the underlying task; it may be exploiting patterns in how the benchmark was constructed, benefiting from data contamination, or optimizing toward the specific format of the evaluation without generalizing to related tasks.

Data contamination is a well-documented example. If a model's pretraining corpus contains examples from a benchmark's test set, the model's score on that benchmark reflects memorization rather than generalization. Given the size of modern pretraining datasets, which may contain hundreds of billions or trillions of tokens from diverse web sources, and the number of benchmarks in circulation, contamination is nearly impossible to rule out entirely without full dataset transparency. Even approximate contamination, where the model has seen paraphrased versions of test questions, can inflate scores in ways that are difficult to measure.

Benchmark saturation is a related issue. Once models achieve near-ceiling performance on a benchmark, it stops differentiating capabilities. The GLUE benchmark was saturated within two years of its introduction. Its successor SuperGLUE was saturated within another two. The field has shifted to progressively more difficult benchmarks, but the cycle repeats: today's hardest benchmarks become tomorrow's solved problems, and it is not always clear whether the improvement reflects general capability gains or optimization toward the benchmark's specific format. A model that achieves superhuman scores on MMLU has not demonstrated superhuman knowledge; it has demonstrated the ability to select correct answers in the multiple-choice format that MMLU uses, which may involve very different skills than producing correct answers without options provided.

The deeper problem is that we do not have a satisfying theory of what benchmarks measure. A benchmark score is a statistic about model behavior on a specific distribution of inputs, in a specific format, evaluated by a specific protocol. Generalizing from that statistic to claims about model capability requires assumptions about the relationship between benchmark performance and real-world task performance that are difficult to validate. Developing evaluation methodology that produces claims with clearly characterized scope and limitations is one of the most important open problems in the field.