Autoregressive Generation › Limitations and Considerations

Autoregressive generation, while powerful, comes with inherent limitations that affect both capability and deployment. These limitations shape what language models can and cannot do, and they inform the engineering decisions made when building systems on top of them.

The sequential nature of generation creates a basic latency floor. No matter how fast each token can be generated, creating 100 tokens requires at least 100 sequential steps. This contrasts with encoding, where an entire sequence can be processed in parallel. For applications requiring very long outputs, this sequential bottleneck becomes significant. A 1000-token response at 50 tokens-per-second takes 20 seconds, no matter how many GPUs you throw at the problem, because the steps cannot be parallelized. Techniques like speculative decoding and parallel draft trees attempt to mitigate this, but the basic constraint remains: each token must be decided before the next one can be chosen.

The left-to-right commitment structure creates a related problem: errors early in generation compound downstream. If the model generates an incorrect premise in the third token, everything that follows will be conditioned on that error. The model cannot go back and revise; it can only attempt to recover within the constraints of what it has already committed to. This makes autoregressive generation fragile in ways that human writing is not: a human writer can revise a passage, while an autoregressive model is locked into its sequential choices. This limitation motivates techniques like chain-of-thought prompting (giving the model space to "think out loud" before committing to a final answer) and best-of-N generation (running the model multiple times and selecting the best output), but neither fully solves the basic problem.

KV cache memory grows linearly with sequence length, which can become problematic for very long contexts. A 7B parameter model with 32 layers and a 128K context window might require tens of gigabytes just for the cache. This has motivated research into cache compression techniques like sliding window attention, sparse attention patterns, and cache eviction strategies. Some systems maintain only a fixed-size cache of the most recent tokens, trading off the ability to attend to distant context for bounded memory usage. The choice between "full context" and "bounded context" is not purely technical: it affects what kinds of tasks the model can perform. Code generation that must reference a function defined 50,000 tokens earlier requires full context; a customer service chatbot that only needs the last few turns of conversation can work fine with a bounded cache.

The quality of generated text depends heavily on decoding strategy, and there is no universally best strategy. Greedy decoding, while deterministic and fast, often produces repetitive or generic outputs because it always selects the locally optimal token without considering how that choice constrains future options. A greedy decoder might commit to "The quick brown fox" even when a more interesting completion was available, because each individual word in that phrase was slightly more probable than its alternatives. Temperature sampling introduces diversity but can also produce incoherent text at high temperatures, where low-probability tokens are selected too often. Beam search explores multiple hypotheses simultaneously but tends toward generic, high-probability sequences and can be worse than sampling for creative tasks. Finding the right balance for each application requires experimentation, and the optimal strategy differs by domain: factual question answering, creative writing, code generation, and mathematical reasoning each have different optimal settings.

Finally, autoregressive generation has no mechanism for self-correction or planning. The model generates token by token without any explicit ability to evaluate whether the current trajectory is heading toward a good outcome. Humans writing a long document can pause, re-read, and restructure; autoregressive generation has no such facility. Research into "process reward models" and "tree-of-thought" prompting attempts to add higher-level planning on top of the base autoregressive mechanism, but these require additional inference compute and are not always practical at serving scale. Understanding this limitation helps calibrate expectations: autoregressive models are extraordinary pattern completers, but they are not planners, and tasks that require careful forward-looking reasoning expose this limitation.