Katz Back-off (1987) › A Concrete Example
History of Language AIAll chapters Katz Back-off (1987) › A Concrete ExampleOne section · Sections in this chapter4 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To see how back-off works in practice, let's trace through a specific example. Suppose we're building a trigram language model and we want to predict what word is likely to follow "peanut butter and." This is the kind of prediction a speech recognition system might need to make when deciding between acoustically similar words, or that an autocomplete system would use when suggesting the next word to a user.
The model first attempts to use the full trigram "peanut butter and." It searches through its training data for instances of this exact three-word sequence. In our hypothetical example, suppose this trigram never appeared in the training corpus, or appeared so rarely that the model's threshold declares the data insufficient. The count is zero or near-zero, so the model cannot make a reliable prediction based on this specific context.
The model then backs off to the bigram "butter and," using only the last two words as context. This shorter sequence appears more frequently. Suppose the training data contains five instances of "butter and," and among these five instances, "jelly" appeared three times and "cheese" appeared twice. A simple maximum likelihood estimate would give us a conditional probability of 3/5 or 0.6 for "jelly" following "butter and."
However, Katz back-off doesn't simply use this raw probability. Instead, it applies a discount factor α that adjusts the estimate. The discounted probability becomes α × 0.6, where α is less than one. The remaining probability mass, (1-α) × P(jelly), comes from backing off to even shorter contexts or the base frequency of "jelly" in the entire corpus. This mathematical adjustment ensures the probability estimate accounts for the uncertainty inherent in using a shorter context than originally intended.
The result is a reasonable probability estimate for "jelly" following "peanut butter and," even though we never encountered this exact trigram in training. The model has gracefully degraded from its preferred level of specificity to a broader but more reliable context, and it has adjusted its confidence accordingly.