N-gram Language Models › N-gram Language Models

What makes some word sequences sound natural while others feel wrong? "The cat sat on the mat" flows effortlessly, but "mat the on sat cat the" is gibberish even though it contains exactly the same words. Native speakers intuitively know which sequences are probable, even for sentences they have never heard before. N-gram language models capture this intuition mathematically, assigning probabilities to sequences of words based on patterns learned from large text collections.

Language models are foundational to natural language processing. They power spell checkers that suggest "their" over "thier," autocomplete systems that predict your next word, and speech recognizers that disambiguate "recognize speech" from "wreck a nice beach." Before neural networks dominated the field, n-gram models were the workhorses of language modeling. Systems built on trigrams and simple smoothing techniques ran at the heart of Google's search products, Microsoft's Bing, and virtually every production voice interface from the 1990s through the early 2010s. Understanding them provides essential intuition for the modern approaches covered throughout this handbook.

As we saw in the previous chapter on N-grams, sequences of adjacent words reveal statistical patterns in text. This chapter takes that idea further: instead of simply counting n-gram frequencies, we use those counts to build a probability model over entire word sequences. You will learn the probability theory that makes n-gram language models work, implement them from scratch, evaluate them rigorously with perplexity, and discover both their power and their fundamental limitations. The next chapter on Smoothing Techniques addresses the central failure mode you will encounter here.