Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

N-gram Language Models › Smoothing: A Preview

Michael BrenndoerferMarch 23, 20253 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 8N-gram Language Models › Smoothing: A Preview

Reading section 12 of 15 · Smoothing: A Preview · 3 minSection 12 of 15 · 3 min

Section 12 of 15 · 3 min read

N-gram Language Models › Smoothing: A PreviewOne section · Sections in this chapter12 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The zero probability problem is so central to n-gram language modeling that it demands a direct solution. Before the next chapter covers smoothing techniques in depth, let us preview the core idea.

The fundamental problem is that MLE assigns probability zero to any n-gram not seen in training. But in a finite corpus, many perfectly valid n-grams simply never occurred by chance. The solution is smoothing: redistribute some probability mass from seen n-grams to unseen ones.

Add-One (Laplace) SmoothingLink Copied

The simplest smoothing technique is add-one smoothing (also called Laplace smoothing): pretend we saw every n-gram one extra time. This is equivalent to adding a pseudocount of 1 to every possible n-gram count:

P_{\text{Laplace}}(w_i|w_{i-1}) = \frac{C(w_{i-1}, w_i) + 1}{C(w_{i-1}) + V}

where:

$C(w_{i-1}, w_i)$ : the actual count of the bigram in training
$C(w_{i-1})$ : the actual count of the context word in training
$V$ : the vocabulary size (number of unique words)
Adding 1 to the numerator gives every bigram at least one pseudo-observation
Adding $V$ to the denominator ensures probabilities still sum to 1 (since there are $V$ possible next words, each getting +1)

For our earlier example of "the bird flew over the house," add-one smoothing would assign:

P_{\text{Laplace}}(\text{bird}|\text{the}) = \frac{0 + 1}{C(\text{the}) + V} = \frac{1}{2 + 9} = \frac{1}{11} \approx 0.091

This is small but non-zero, which saves the entire sentence from a zero probability. The approach works in principle but has a known flaw: it allocates far too much probability mass to unseen events. For a vocabulary of 50,000 words, adding $V$ to every context count dramatically dilutes the probability of observed events. More sophisticated smoothing methods like Kneser-Ney interpolation do a much better job of distributing probability to unseen events, and that is the subject of the next chapter.

In[39]:

Code

def laplace_bigram_probability(
    w1, w2, bigram_counts, unigram_counts, vocab_size
):
    """P(w2|w1) with add-one (Laplace) smoothing."""
    bigram = (w1, w2)
    unigram = (w1,)

    bigram_count = bigram_counts.get(bigram, 0)
    unigram_count = unigram_counts.get(unigram, 0)

    return (bigram_count + 1) / (unigram_count + vocab_size)


# Compare MLE vs Laplace on seen and unseen bigrams
vocab_size = len(unigram_counts_marked)
comparison_pairs = [
    ("the", "cat"),  # seen bigram
    ("sat", "on"),  # seen bigram (probability 1.0 under MLE)
    ("the", "bird"),  # unseen in original corpus
    ("cat", "flew"),  # unseen
]

Out[40]:

Console

MLE vs. Laplace Smoothing:
------------------------------------------------------------
Bigram                      MLE P    Laplace P
------------------------------------------------------------
(the, cat) [seen]            0.3750       0.2105
(sat, on) [seen]            1.0000       0.2308
(the, bird) [unseen]            0.0000       0.0526
(cat, flew) [unseen]            0.0000       0.0714

Laplace smoothing solves the zero-probability problem: unseen bigrams now get a small but non-zero probability. However, observe that the Laplace probability for "sat, on" (which appeared with probability 1.0 under MLE) drops dramatically. This is the core tension: any probability mass given to unseen events must come from somewhere, and naive Laplace smoothing takes too much from the seen events. The next chapter shows how to do this redistribution more carefully.