Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

N-grams › From Words to Word Sequences

Michael BrenndoerferUpdated March 22, 20254 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 7N-grams › From Words to Word Sequences

Reading section 2 of 16 · From Words to Word Sequences · 4 minSection 2 of 16 · 4 min

Section 2 of 16 · 4 min read

N-grams › From Words to Word SequencesOne section · Sections in this chapter2 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The bag of words model creates a vocabulary of individual words and counts how often each appears in a document. N-grams extend this idea by creating a vocabulary of word sequences and counting how often each sequence appears. This small change has large practical consequences.

Think of it this way: a unigram vocabulary for English has perhaps 50,000 to 100,000 entries, one per word form. A bigram vocabulary for the same corpus has millions of entries, one per pair of consecutive words. A trigram vocabulary has tens of millions. The vocabulary grows exponentially, but each entry carries more information about local context.

To see the difference concretely, consider how different n-gram orders segment the same sentence:

N-gram

An n-gram is a contiguous sequence of n items from a text. When the items are words, we call them word n-grams. When the items are characters, we call them character n-grams. Common special cases include unigrams (n=1), bigrams (n=2), and trigrams (n=3). The general plural "n-grams" refers to any or all of these sequence lengths.

Let's see how the same sentence produces different representations at each n-gram level:

In[3]:

Code

def extract_ngrams(tokens, n):
    """Extract n-grams from a list of tokens."""
    return [tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1)]


# Sample sentence
sentence = "the quick brown fox jumps over the lazy dog"
tokens = sentence.split()

# Extract different n-gram sizes
unigrams = extract_ngrams(tokens, 1)
bigrams = extract_ngrams(tokens, 2)
trigrams = extract_ngrams(tokens, 3)

Out[4]:

Console

Sentence: 'the quick brown fox jumps over the lazy dog'
Tokens: 9

Unigrams (9):
  ['the', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']

Bigrams (8):
  [('the', 'quick'), ('quick', 'brown'), ('brown', 'fox'), ('fox', 'jumps'), ('jumps', 'over'), ('over', 'the'), ('the', 'lazy'), ('lazy', 'dog')]

Trigrams (7):
  [('the', 'quick', 'brown'), ('quick', 'brown', 'fox'), ('brown', 'fox', 'jumps'), ('fox', 'jumps', 'over'), ('jumps', 'over', 'the'), ('over', 'the', 'lazy'), ('the', 'lazy', 'dog')]

Notice the pattern: each sentence with $k$ tokens produces a different number of n-grams depending on the sequence length you choose. The relationship between sentence length, n-gram order, and count is deterministic:

\text{count} = k - n + 1

where:

$k$ : the total number of tokens in the sentence
$n$ : the n-gram size (1 for unigrams, 2 for bigrams, 3 for trigrams, etc.)
$\text{count}$ : the resulting number of n-grams

The key insight behind this formula is that each n-gram starts at a unique position in the token sequence. The very first n-gram starts at position 1, and the last valid starting position is $k - n + 1$ , because that is the last position where we can still find $n$ consecutive tokens. Starting any later would run past the end of the sentence. With 9 tokens, we get 9 unigrams ( $9 - 1 + 1 = 9$ ), 8 bigrams ( $9 - 2 + 1 = 8$ ), and 7 trigrams ( $9 - 3 + 1 = 7$ ). As $n$ increases, we extract fewer n-grams from each document, but each n-gram carries more contextual information. The coverage decreases; the specificity increases.

Worked Example: Tracing Bigrams Through a SentenceLink Copied

Let's trace through a short, meaningful sentence step by step to build intuition for what bigrams represent. Take the sentence "cats eat fish."

After lowercasing and tokenizing, we have tokens: ["cats", "eat", "fish"]. Applying the bigram extraction algorithm:

Position 1: window covers tokens 1 and 2, producing ("cats", "eat")
Position 2: window covers tokens 2 and 3, producing ("eat", "fish")

We stop here because the next position would require a token 4 that does not exist. The final bigram representation is [("cats", "eat"), ("eat", "fish")].

Notice that "eat" appears in both bigrams. Each word (except the first and last) participates in exactly two bigrams: one where it is the second word, and one where it is the first. This overlapping structure means that consecutive n-grams share most of their tokens, which is why they remain sensitive to local order while still providing a sliding-window view of the text.

Now compare this to a different sentence with the same words: "fish eat cats." The tokens are identical but in reverse order, giving bigrams [("fish", "eat"), ("eat", "cats")]. These two sets share no bigrams at all, even though they use exactly the same vocabulary. This is the property that unigrams cannot provide: sensitivity to word order.