N-gram Language Models › Limitations and Impact

N-gram models face fundamental limitations that explain why they were eventually supplanted by neural approaches, but also historical significance that earns them careful study.

The most severe limitation is data sparsity. Most possible n-grams never appear in training data, even in corpora of billions of words. This is not a problem that more data fully solves: as n-gram order increases, the number of unseen combinations grows exponentially. Even with 1 trillion words of training text, trigrams for technical or domain-specific language will have coverage gaps. The LDC English Gigaword corpus, a standard benchmark with about 4.3 billion words, covers only a fraction of possible English trigrams. A model trained on the full web, with hundreds of billions of words, still encounters unseen trigrams regularly.

The sparsity problem has a deeper mathematical explanation. The true distribution of language assigns non-zero probability to an enormous number of grammatically valid sentences, most of which will never appear in any finite training corpus. Any model trained by counting observations in finite data fundamentally underestimates the variety of natural language.

The Markov assumption is a second major limitation. Real language has long-range dependencies that exceed any practical context window. Consider the sentence "The cat that the dog had been chasing for three blocks finally escaped." The verb "escaped" agrees with "cat," but the subject and verb are separated by 10 words. A trigram model sees only "blocks finally escaped" and cannot make that connection. More subtly, pronoun resolution ("She arrived late. The professor was angry with her."), discourse coherence, and topic tracking all require understanding relationships across sentences, not just consecutive words.

The history of NLP is partly a history of attempts to overcome this limitation. Long short-term memory networks (LSTMs) extended the context window by learning to remember relevant information from many steps back. Transformer models addressed it more directly by computing attention over the entire sequence simultaneously, allowing any word to directly access any other word regardless of distance.

N-gram models also fail to generalize across similar words. The model treats "cat," "feline," and "kitty" as entirely unrelated symbols. If "the feline sat" was never seen in training, it gets zero probability, even though "the cat sat" appeared many times. Knowledge learned about one word does not transfer to synonyms or semantically related words.

This is the word representation problem: n-gram models use discrete symbols with no notion of similarity or meaning. The solution, word embeddings, was not directly invented to fix n-gram models, but the intuition that similar words should have similar representations emerged partly from recognizing this gap. We discuss word embeddings extensively in later chapters.

Despite these limitations, n-gram models enabled important applications before neural networks took over, and they did so at scales that were practical with 1990s and 2000s hardware.

Speech recognition. Google Voice, Apple Siri (in its early forms), and every voice interface from that era used trigram language models to disambiguate acoustically similar phrases. The acoustic model provides probability distributions over phonemes; the language model rescores those hypotheses to favor plausible word sequences. The combination of a strong acoustic model and a good language model was the dominant paradigm for about two decades.

Statistical machine translation. Systems like Google Translate's early versions used phrase-based statistical MT, which generated candidate translations using phrase tables and scored their fluency using a trigram language model. The language model ensured that the output sounded like natural target-language text even when the phrase-by-phrase translation produced awkward word orders.

Spelling correction. Typing "teh" gets corrected to "the" partly because a character-level language model knows "teh" is not a word, and partly because a word-level language model knows which correction makes the surrounding sequence more probable.

Autocomplete. Early autocomplete systems in smartphones and text editors predicted likely next words using bigram or trigram language models. The experience of a phone typing "on my way" after "I am" emerged from exactly the kind of conditional probability estimation we implemented in this chapter.

The concepts introduced by n-gram models remain central to modern language modeling: the chain rule decomposition, the tension between context length and data sparsity, maximum likelihood estimation from counts, perplexity as an evaluation metric, and temperature-based sampling. Understanding n-grams is not just historical background. It provides the conceptual foundation for the neural approaches covered throughout the rest of this handbook. The chain rule is how neural language models are also trained and evaluated. Perplexity is the standard metric by which GPT-4 and its contemporaries are compared on held-out benchmarks. Temperature sampling, which we implemented here in a few lines, is the same parameter you adjust when calling the OpenAI API.

The contrast with neural language models is sharp and instructive. Consider what a neural language model does differently:

Parameter sharing across contexts. An n-gram model stores separate probabilities for every context it has seen. A neural model learns a shared function from context to probability, which means patterns learned in one context generalize to similar contexts. If the model learns that "the fluffy cat" likely continues with verbs about cat behavior, it can generalize this to "the fluffy kitten" without having seen that trigram.

Dense representations. Neural models map words to dense vectors in a continuous space, where similar words have similar vectors. This allows semantic generalization: information about "cat" contributes to predictions about "kitten" because their vector representations are nearby.

Unlimited context. Transformer-based language models attend to the entire context window simultaneously, up to thousands or tens of thousands of tokens. There is no fixed context length and no exponential blowup in parameters as context grows. The attention mechanism handles variable-length dependencies directly.

Scalability. Neural language models can use the same compute hardware (GPUs and TPUs) as other deep learning work. Training efficiency, along with data requirements, has driven the rise of neural approaches.

The transition from n-gram models to neural models was not sudden. For years, researchers used neural language models as components within larger hybrid systems that also included n-gram components. The neural models handled long-range dependencies while the n-gram models provided reliable low-order estimates. It was only after the Transformer architecture and large-scale pretraining made pure neural approaches dominate on every benchmark that n-gram models were fully retired from production use.