N-grams › N-grams
Language AI HandbookAll chapters N-grams › N-gramsOne section · Sections in this chapter1 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Language does not work word by word. The meaning of any given word almost always depends on what surrounds it. When you read "I need to book a flight," the word "book" is a verb. When you read "hand me that book," it is a noun. You understood the difference not from the word itself but from the words nearby. This dependence on local context is one of the most fundamental properties of natural language, and n-grams are the simplest tool that captures it.
As we saw in the previous chapter on bag of words, representing a document as a multiset of individual word counts discards every piece of word-order information. The sentence "The dog bit the man" and the sentence "The man bit the dog" produce identical bag-of-words vectors, even though they describe opposite events. For many classification tasks, word-order loss is acceptable and the simplicity of bag of words is worth the trade-off. But as soon as meaning depends on sequence, we need a richer representation.
N-grams offer a direct extension of bag of words: instead of treating single words as atomic units, we treat fixed-length sequences of consecutive words as atomic units. An n-gram is a contiguous sequence of items drawn from a text, where items can be words, characters, syllables, or any other tokens. When we work with words, we call them word n-grams. When we work with characters, we call them character n-grams. The special cases have familiar names: a 1-gram is a unigram (the same as standard bag of words), a 2-gram is a bigram, and a 3-gram is a trigram.
The intuition is straightforward: phrases like "machine learning," "New York," "not good," and "highly recommend" carry meaning as units that their individual words cannot convey in isolation. "Machine" and "learning" are common words on their own, but together they signal a technical domain. "Not" and "good" are individually neutral in many contexts, but as a bigram they express sentiment that would be lost if you only counted the word "good." N-grams capture this by including multi-word sequences in the vocabulary.
This chapter explores how n-grams preserve local word order, why vocabulary size grows explosively with n, and how the statistical properties of n-grams follow predictable mathematical patterns. You will learn to extract n-grams efficiently, understand when character n-grams outperform word n-grams, see how skip-grams relax the strict adjacency requirement, and understand how n-gram models assign probabilities to text sequences. By the end you will have a complete picture of what n-grams can do, what they cannot do, and why they remain relevant even in the age of large language models.
The theoretical foundations of n-gram modeling trace back to Claude Shannon's 1948 paper "A Mathematical Theory of Communication." Shannon used character-level n-gram statistics to estimate the entropy of English text. This shows that consecutive characters are statistically dependent. He showed that letter sequences follow predictable patterns by asking human subjects to guess the next character in a sequence: the more characters they had seen, the better they could guess, up to a point. This experiment captured the essence of what n-gram language models formalize mathematically. Shannon's work directly inspired Frederick Jelinek and colleagues at IBM in the 1970s and 1980s, who built the first large-scale n-gram language models for speech recognition. Their trigram models, trained on tens of millions of words, achieved dramatically better word error rates than earlier systems. The same statistical approach dominated speech recognition for nearly three decades before neural networks overtook it in the early 2010s.