N-grams › Limitations and Trade-offs

N-grams improve on bag of words by preserving local word order and capturing multi-word phrases as single units. But they carry significant limitations that motivate the development of richer representations.

The most fundamental limitation is the fixed context window. An n-gram of size $n$ captures exactly $n$ consecutive words, no more and no less. Any relationship between words separated by more than $n - 1$ positions is invisible to the model. The phrase "if she studies consistently throughout the semester, then her exam results improve" carries a long-range correlation between "if" and "then" that spans many words. No practical n-gram can capture this. Language models that use trigrams model context three words at a time; the rest of the sentence history is discarded by the Markov approximation.

The vocabulary explosion is a fundamental challenge. Moving from bigrams to trigrams multiplies the number of possible n-gram types by the vocabulary size. For a 50,000-word vocabulary, that means going from 2.5 billion possible bigrams to 125 trillion possible trigrams. The observed fraction in any corpus is tiny, making most of the probability space inaccessible and forcing heavy reliance on smoothing. Smoothing helps but cannot fully solve the zero-count problem; it only redistributes probability mass according to heuristics that may not reflect the true distribution.

No semantic understanding is perhaps the most limiting aspect of n-gram representations. "Excellent movie" and "great film" share no n-grams despite expressing the same sentiment. A synonym substitution produces a completely different n-gram fingerprint. This means that n-gram models cannot generalize across synonymous expressions, different phrasings of the same meaning, or across languages. Every surface form is treated as a completely distinct symbol, with no connection to any other symbol that happens to mean the same thing.

Data requirements grow exponentially with n. A reliable bigram model might require millions of words of training data; a trigram model might need tens of millions; 5-gram models used in machine translation systems were trained on billions of words. This makes high-order n-gram models inaccessible to researchers and applications that do not have access to massive corpora.

The sensitivity to tokenization is also important in practice. Whether "New York" is one token or two, whether contractions are split, whether punctuation is attached to words or separated: all of these decisions change the n-gram representation significantly. N-gram models can be brittle when the tokenization of training data differs from the tokenization of test data, because even a slight mismatch propagates through every n-gram that spans a tokenization boundary.

Despite these limitations, n-grams occupied the center of NLP for decades because they are simple and fast, with interpretable features that often work surprisingly well. They require no deep learning infrastructure, no GPU, and no careful architecture design. A trigram language model trained on a large corpus achieves reasonable performance on speech recognition and machine translation with nothing but counting and smoothing.