Shannon's N-gram Model (1948) › The Anatomy of an N-gram

An n-gram is, at its most basic, a contiguous sequence of n items from a text. When working with language, these items are typically words, though they can also be characters, syllables, or other linguistic units. The beauty of n-grams lies in their conceptual simplicity. A unigram is a single word considered in isolation. A bigram captures two consecutive words. A trigram extends this to three words in sequence. The principle generalizes naturally: a 4-gram contains four consecutive words, a 5-gram contains five, and so forth.

Consider the sentence "the quick brown fox jumps." The bigrams extracted from this text would be "the quick," "quick brown," "brown fox," and "fox jumps." Each bigram represents a local context, a tiny window into how words combine in natural language. The trigrams would be "the quick brown," "quick brown fox," and "brown fox jumps," providing slightly more context with each additional word.

Shannon recognized that language exhibits strong statistical regularities at this level of analysis. Language is not a random process where any word can follow any other with equal probability. Instead, certain word combinations appear far more frequently than others, and these patterns of co-occurrence carry substantial information about the structure and meaning of language. After the word "peanut," English speakers overwhelmingly favor "butter" as the next word over alternatives like "giraffe" or "telescope." This isn't a hard grammatical rule, all three continuations are grammatically valid, but it reflects powerful statistical tendencies learned from exposure to real language use.

To demonstrate the predictability inherent in language, Shannon conducted a series of elegant experiments. He would show people a partial sentence and ask them to guess the next letter or word. What he discovered was striking: given enough context, people could make remarkably accurate predictions. This predictability varied across different parts of the language. Common phrases and grammatical constructions were highly predictable, while unexpected word choices or rare combinations contained more surprise. This relationship between predictability and information content became a cornerstone of information theory. The less predictable something is, the more information it carries when it occurs. A rare, surprising word tells you more about the specific message than a common, expected one.