N-grams › Practical Considerations
Language AI HandbookAll chapters N-grams › Practical ConsiderationsOne section · Sections in this chapter11 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When working with n-grams in real applications, several engineering and modeling decisions have significant impact on your results.
Choosing n. Larger n captures more context but creates sparser data. The right choice depends on your task, your corpus size, and your tolerance for sparsity:
- Unigrams (n=1): Baseline that loses all word order, but works surprisingly well for many classification tasks
- Bigrams (n=2): Good balance for most tasks; captures immediate phrase structure while remaining tractable
- Trigrams (n=3): Richer context; standard for language modeling tasks; requires substantially more data
- n greater than 3: Rarely used as standalone features due to extreme sparsity; occasionally useful in specific applications like DNA sequence analysis or stylometry
Vocabulary pruning. You almost always want to remove n-grams that appear too rarely (below some minimum document frequency) or too frequently (above some maximum). Rare n-grams add noise rather than signal: if a trigram appears only once in your training data, your estimate of its probability is unreliable. Very frequent n-grams like "of the" and "in the" appear everywhere and carry almost no discriminative information for tasks like topic classification or sentiment analysis. Pruning reduces vocabulary size, which saves memory and often improves model quality.
Padding. For language modeling applications, you may want to add special start and end tokens to capture sentence boundary statistics:
def extract_padded_ngrams(tokens, n, pad_symbol="<PAD>"):
"""Extract n-grams with padding for boundary awareness."""
padded = [pad_symbol] * (n - 1) + tokens + [pad_symbol] * (n - 1)
return extract_ngrams(padded, n)
sentence_tokens = ["the", "cat", "sat"]
padded_bigrams = extract_padded_ngrams(sentence_tokens, 2)
padded_trigrams = extract_padded_ngrams(sentence_tokens, 3)Padded n-grams capture sentence boundaries:
Tokens: ['the', 'cat', 'sat']
Padded bigrams: [('<PAD>', 'the'), ('the', 'cat'), ('cat', 'sat'), ('sat', '<PAD>')]
Padded trigrams: [('<PAD>', '<PAD>', 'the'), ('<PAD>', 'the', 'cat'), ('the', 'cat', 'sat'), ('cat', 'sat', '<PAD>'), ('sat', '<PAD>', '<PAD>')]The padding approach ensures that the beginning and end of a sentence are treated as a special context rather than being silently ignored. Without padding, the first and last words of a sentence only appear as the second element of starting bigrams or the first element of ending bigrams, which underestimates how often they occur in those boundary positions. With <s> and </s> boundary markers, the model can learn that certain words frequently begin or end sentences.
Memory efficiency. For large corpora, string-based n-gram storage becomes prohibitively expensive. In practice:
- Store n-grams as hashed integer tuples rather than string tuples. The hash reduces memory by roughly 4-8x.
- Use sparse matrix representations (scipy.sparse) when building document-term matrices from n-gram features.
- Apply feature hashing (the "hashing trick") to map n-grams to a fixed-dimensional vector without storing the vocabulary explicitly. This bounds memory at the cost of hash collisions.
- Use count sketches or min-count thresholding to discard rare n-grams during construction rather than after.
Combining n-gram orders. In practice, the best results often come from combining multiple n-gram orders. Scikit-learn's CountVectorizer and TfidfVectorizer accept ngram_range=(1, 2) to include both unigrams and bigrams, or ngram_range=(1, 3) to include unigrams through trigrams. Combining orders gives the model access to both word-level and phrase-level patterns, usually improving performance on classification tasks at the cost of a larger feature space.