Smoothing Techniques › Laplace (Add-One) Smoothing

The simplest smoothing method is Laplace smoothing, also called add-one smoothing. The idea predates NLP entirely, originating with Pierre-Simon Laplace's "rule of succession," his 18th-century method for estimating the probability of events that haven't been observed yet. Laplace applied it to the question of predicting whether the sun would rise tomorrow given that it had risen every day in recorded history. The same reasoning applies here: if we haven't seen a bigram, that doesn't mean it's impossible, just that our sample was finite.

The intuition is direct: pretend you saw every possible n-gram at least once. To achieve this, add 1 to every n-gram count before normalizing. This ensures no count is zero and therefore no probability is zero.

For a bigram model, the smoothed probability is computed by adding 1 to the raw bigram count and adding $|V|$ to the denominator (to account for all the extra counts across the full vocabulary):

$$P_{\text{Laplace}}(w_i \mid w_{i-1}) = \frac{C(w_{i-1}, w_i) + 1}{C(w_{i-1}) + |V|}$$

where:

• $C(w_{i-1}, w_i)$: raw bigram count in the corpus
• $C(w_{i-1})$: raw unigram count of the context word
• $|V|$: vocabulary size (number of unique words)

We add 1 to the numerator for every possible bigram (including unseen ones), and add $|V|$ to the denominator to normalize correctly. Since there are $|V|$ possible next words and each gets one extra count, the total extra count is $|V|$, which must be reflected in the denominator to preserve the sum-to-one property.

The general form for an n-gram model extends this naturally. To estimate the probability of $w_i$ given an $(n-1)$-word context $w_{i-n+1}^{i-1}$:

$$P_{\text{Laplace}}(w_i \mid w_{i-n+1}^{i-1}) = \frac{C(w_{i-n+1}^{i-1}, w_i) + 1}{C(w_{i-n+1}^{i-1}) + |V|}$$

where $w_{i-n+1}^{i-1}$ denotes the $(n-1)$-word context.

Consider a tiny corpus: "the cat sat on the mat the cat sat." Let's compute some bigram probabilities.

Raw bigram counts:

• C("the cat") = 2
• C("cat sat") = 2
• C("sat on") = 1
• C("on the") = 1
• C("the mat") = 1
• C("mat the") = 1

Raw unigram counts: C("the") = 3, C("cat") = 2, C("sat") = 2, C("on") = 1, C("mat") = 1.

Vocabulary size: $|V| = 5$.

The MLE probability of "cat" given "the" is:

$$P_{\text{MLE}}(\text{cat} \mid \text{the}) = \frac{C(\text{the cat})}{C(\text{the})} = \frac{2}{3} \approx 0.667$$

The Laplace-smoothed probability of the same bigram is:

$$P_{\text{Laplace}}(\text{cat} \mid \text{the}) = \frac{2 + 1}{3 + 5} = \frac{3}{8} = 0.375$$

For an unseen bigram ("the sat"), the two methods give:

$$P_{\text{MLE}}(\text{sat} \mid \text{the}) = 0$$ $$P_{\text{Laplace}}(\text{sat} \mid \text{the}) = \frac{0 + 1}{3 + 5} = \frac{1}{8} = 0.125$$

The zero probability disappears. But notice how much probability mass has shifted: "the cat" dropped from 0.667 to 0.375, nearly cut in half. This turns out to be a significant problem.

Laplace smoothing is intuitive but performs poorly in practice. The reason is that it steals far too much probability from seen n-grams to give to unseen ones.

The amount stolen can be quantified by an adjusted count: the count a bigram effectively acts as after smoothing, relative to the denominator. For a bigram with raw count $c$ and context count $C(w_{i-1})$, the adjusted count is:

$$C^*_{\text{Laplace}}(w_{i-1}, w_i) = \frac{\bigl[C(w_{i-1}, w_i) + 1\bigr] \cdot C(w_{i-1})}{C(w_{i-1}) + |V|}$$

where all symbols are as defined above. This formula converts the smoothed probability back into "effective count" units so you can directly compare how much the raw count was distorted.

For our example, the adjusted count of "the cat" is:

$$C^*_{\text{Laplace}}(\text{the cat}) = \frac{(2+1) \times 3}{3 + 5} = \frac{9}{8} = 1.125$$

So the bigram "the cat" (which occurred twice) is being treated as if it occurred only 1.125 times, a 44% reduction. For a bigram that occurred 100 times, adding 1 barely changes anything. But for a bigram that occurred once or twice, the effect is enormous. Laplace smoothing systematically underestimates frequent n-grams and over-estimates the probability mass to allocate to unseen ones.

Church and Gale (1991) showed empirically that Laplace smoothing allocates far more probability mass to unseen bigrams than empirical evidence justifies. For realistic vocabulary sizes (tens of thousands of words), the $|V|$ term in the denominator is huge, and adding 1 to each count forces massive reallocation regardless of what the data shows. When your vocabulary contains 50,000 words and your context word appeared 1,000 times, the denominator becomes 51,000, reducing every probability by roughly 2%. That might seem small, but applied to counts of 1 or 2, it reduces those bigrams to near-uniform noise.

The core insight from this analysis is that the severity of the discount should depend on the raw count. A bigram seen 1,000 times shouldn't be discounted by the same proportion as one seen twice. Laplace uses a fixed additive adjustment that ignores this completely. This motivates more principled alternatives.