N-grams › Bigrams and Trigrams in Practice
Language AI HandbookAll chapters N-grams › Bigrams and Trigrams in PracticeOne section · Sections in this chapter3 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Bigrams are the most commonly used n-grams in applied NLP. They capture immediate word relationships, such as noun-adjective pairs, compound nouns, and common verbal phrases, while keeping vocabulary size manageable. Trigrams provide richer context and are standard in language modeling, where they allow conditioning on a two-word history. Longer n-grams exist but are rarely used as standalone features because the sparsity problem becomes severe.
The most revealing way to understand what bigrams encode is to look at their frequency distributions in real text. Common bigrams cluster around function word combinations like "of the," "in the," and "to be," but the bigrams with the strongest discriminative power for content-based tasks are typically noun phrase patterns and domain-specific collocations:
from collections import Counter
# A small corpus to analyze
corpus = [
"I love machine learning",
"I love deep learning",
"machine learning is powerful",
"deep learning is a subset of machine learning",
"I study machine learning every day",
]
# Tokenize and extract bigrams from entire corpus
all_bigrams = []
for doc in corpus:
tokens = doc.lower().split()
bigrams = extract_ngrams(tokens, 2)
all_bigrams.extend(bigrams)
# Count bigram frequencies
bigram_counts = Counter(all_bigrams)Corpus bigram frequencies: ---------------------------------------- machine learning 4 i love 2 deep learning 2 learning is 2 love machine 1 love deep 1 is powerful 1 is a 1 a subset 1 subset of 1 Total bigram tokens: 21 Unique bigrams: 15
The bigram "machine learning" appears frequently because it is a meaningful phrase in this corpus. This reveals the key insight: common n-grams often correspond to collocations or other meaningful multi-word expressions, including domain-specific phrases. When you add bigrams as features alongside unigrams in a text classifier, you are giving the model explicit access to these multi-word units rather than requiring it to infer their meaning from co-occurring unigrams.
# Compare unigram vs bigram representations
all_unigrams = []
for doc in corpus:
tokens = doc.lower().split()
all_unigrams.extend(tokens)
unigram_counts = Counter(all_unigrams)Unigram frequencies (top 10): learning 6 machine 4 i 3 love 2 deep 2 is 2 powerful 1 a 1 subset 1 of 1 Unique unigrams: 13 Unique bigrams: 15 Vocabulary explosion factor: 1.2x
Even in this tiny corpus, the bigram vocabulary is larger than the unigram vocabulary. This vocabulary explosion is not a flaw; it is a reflection of the combinatorial richness of natural language. The same words can be combined in many different orders and contexts, and each unique sequence carries distinct information. But the growth in vocabulary size has direct practical costs in memory, computational complexity, and the amount of data needed to estimate reliable statistics.
The co-occurrence matrix reveals the structure hidden in bigram counts. The cell at row "machine" and column "learning" shows a high count because "machine learning" is a frequent phrase. Most cells are zero. This reflects the sparsity inherent in natural language: most word pairs never occur together in the same order in any given corpus, even if both words appear individually.