Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

N-grams › N-gram Frequency Distributions and Zipf's Law

Michael BrenndoerferUpdated March 22, 20255 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 7N-grams › N-gram Frequency Distributions and Zipf's Law

Reading section 5 of 16 · N-gram Frequency Distributions and Zipf's Law · 5 minSection 5 of 16 · 5 min

Section 5 of 16 · 5 min read

N-grams › N-gram Frequency Distributions and Zipf's LawOne section · Sections in this chapter5 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Individual words follow Zipf's law: the frequency of a word is inversely proportional to its rank in the frequency table. A small number of words account for most of the occurrences in any large corpus. The word "the" might make up 6 to 7 percent of all words in English text, while thousands of low-frequency words each appear only once or twice.

N-grams follow the same pattern, but in an even more extreme form. The top bigrams (phrases like "of the," "in the," "it is") each appear thousands of times, while the vast majority of observed bigrams appear just once. This extreme concentration at the high-frequency end and long tail at the low-frequency end is a universal property of natural language at every level of granularity.

Zipf's Law

Zipf's law states that in a corpus of natural language, the frequency of any word is inversely proportional to its rank in the frequency table. If the most frequent word appears $f$ times, the second most frequent word appears approximately $f/2$ times, the third appears approximately $f/3$ times, and so on. The law generalizes to:

f(r) \propto \frac{1}{r}

where $f(r)$ is the frequency of the word at rank $r$ . On a log-log plot, a Zipfian distribution appears as a straight line with slope approximately $-1$ .

The practical consequence of Zipf's law for n-grams is severe: most of the n-gram types you will ever observe occur exactly once in any given corpus. These single-occurrence n-grams are called hapax legomena (from Greek: "said only once"). As the n-gram order increases, the hapax ratio climbs sharply, because longer sequences are less likely to repeat.

In[13]:

Code

# Analyze frequency distribution for different n-gram orders
def analyze_frequency_distribution(tokens, n, top_k=10):
    """Compute frequency statistics for n-grams."""
    ngrams = extract_ngrams(tokens, n)
    counts = Counter(ngrams)

    # Get frequency of frequencies
    freq_of_freq = Counter(counts.values())

    # Calculate statistics
    total = len(ngrams)
    unique = len(counts)
    hapax = sum(1 for c in counts.values() if c == 1)  # Words appearing once

    return {
        "total": total,
        "unique": unique,
        "hapax": hapax,
        "hapax_ratio": hapax / unique,
        "top_k": counts.most_common(top_k),
        "freq_of_freq": freq_of_freq,
    }


# Analyze unigrams, bigrams, and trigrams
stats = {n: analyze_frequency_distribution(tokens, n) for n in [1, 2, 3]}

Out[14]:

Console

Frequency Distribution Statistics:
============================================================

Unigrams:
  Total tokens: 158,167
  Unique types: 16,945
  Hapax legomena (appear once): 10,232 (60.4%)

  Top 5 most frequent:
    'the': 5,120
    'to': 5,079
    'and': 4,445
    'of': 4,196
    'a': 3,055

Bigrams:
  Total tokens: 158,166
  Unique types: 81,586
  Hapax legomena (appear once): 65,609 (80.4%)

  Top 5 most frequent:
    'to be': 566
    'of the': 558
    'in the': 441
    'it was': 387
    'she had': 313

Trigrams:
  Total tokens: 158,165
  Unique types: 136,444
  Hapax legomena (appear once): 126,523 (92.7%)

  Top 5 most frequent:
    'i do not': 94
    'i am sure': 75
    'she could not': 65
    'a great deal': 56
    'it would be': 56

The hapax ratio climbs steeply with n. For trigrams, the vast majority appear exactly once in even a reasonably large corpus. This sparsity is the core challenge of n-gram models. When you encounter a trigram in new text that you never saw during training, you have no observed count to draw on. You need a strategy for handling zero-count n-grams, which is the motivation for smoothing (discussed in the next section).

Out[15]:

Visualization

Bar chart showing hapax ratio increasing from about 50% for unigrams to over 80% for trigrams. — Hapax ratio (proportion of n-gram types appearing exactly once) increases sharply with n-gram order in Emma. While roughly half of all unigram types are hapax legomena, this rises to over 80 percent for trigrams. The implication is that most trigrams you would encounter in held-out text never appear in training data, making zero-count handling essential for any trigram-based model.

Out[16]:

Visualization

Log-log plot showing frequency vs rank for unigrams, bigrams, and trigrams, all showing approximately linear relationships. — Zipf''s law holds across n-gram orders in Emma. Each curve shows n-gram frequencies plotted against their rank on log-log axes. All three follow approximately linear relationships, confirming Zipf''s law. The steeper slope for bigrams and trigrams indicates an even more extreme concentration of probability mass in the top-ranked sequences: the most common trigrams appear far more often than a uniform distribution would predict.

The Zipfian distribution means that most of your n-gram model's probability mass is concentrated in a tiny fraction of all possible sequences. The top 100 trigrams in a large corpus might account for a significant fraction of all trigram occurrences, while the bottom 90 percent of trigram types each contribute almost nothing. In practice this creates an asymmetry: the common sequences are very well estimated from data, while the rare ones are estimated poorly or not at all.