Shannon's N-gram Model (1948) › From Observation to Prediction: How N-gram Models Work

The operational principle underlying n-gram models is elegantly straightforward. To predict what word comes next in a sequence, examine the preceding few words and consult the statistical patterns learned from a large corpus of text. If you encounter the sequence "peanut butter and," the model looks for all instances in its training data where "peanut butter and" appeared, then calculates the probability distribution over the words that followed. In English text, "jelly" will appear with high frequency in this context, making it the most probable continuation.

N-gram models operate purely through co-occurrence statistics, not through any deep understanding of meaning or grammar. The model doesn't "know" that peanut butter and jelly form a classic sandwich combination, or that this reflects culinary traditions in certain cultures. It simply encodes the empirical fact that these words appear together frequently in written English. This statistical approach has both advantages and limitations, advantages in that it requires no hand-crafted rules or linguistic knowledge, learning patterns directly from data, and limitations in that it cannot reason about meaning, context beyond the immediate n-gram window, or conceptual relationships between words.

The mathematical framework that makes n-gram models tractable is the Markov assumption. Named after the Russian mathematician Andrey Markov who studied stochastic processes in the early 20th century, this assumption states that the probability of the next word depends only on a fixed number of preceding words, not on the entire history of the sequence. A bigram model assumes the next word depends only on the immediately previous word. A trigram model looks back two words. This simplification is clearly an approximation, language exhibits dependencies that span across sentences and paragraphs, but it makes the mathematics manageable and the models trainable with realistic amounts of data.

The practical impact of n-gram models on language technology cannot be overstated. They became the foundational building blocks for virtually every major application of computational linguistics in the late 20th century. In language modeling, n-grams provide a principled way to estimate how probable any given sentence is in a language, enabling applications from autocomplete suggestions to grammar checking. When your email client suggests the next word as you type, there's likely an n-gram model working behind the scenes. In machine translation, n-gram models helped systems choose natural-sounding word order in the target language, distinguishing between multiple grammatically correct translations by preferring those that matched typical usage patterns. In speech recognition, they served as the language model component that helped systems resolve acoustic ambiguities, deciding whether an ambiguous sound sequence more likely represents "recognize speech" or "wreck a nice beach" based on which phrase appears more commonly in the training corpus.