Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

N-grams › Using NLTK for N-gram Extraction

Michael BrenndoerferUpdated March 22, 20252 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 7N-grams › Using NLTK for N-gram Extraction

Reading section 10 of 16 · Using NLTK for N-gram Extraction · 2 minSection 10 of 16 · 2 min

Section 10 of 16 · 2 min read

N-grams › Using NLTK for N-gram ExtractionOne section · Sections in this chapter10 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

While we have implemented n-gram extraction from scratch to understand the mechanics, NLTK provides optimized utilities that are more convenient for production use:

In[32]:

Code

from nltk import bigrams, ngrams, trigrams
from nltk.util import everygrams

# NLTK's n-gram functions
text = "the quick brown fox jumps over the lazy dog"
tokens = text.split()

# Extract using NLTK
nltk_bigrams = list(bigrams(tokens))
nltk_trigrams = list(trigrams(tokens))
nltk_ngrams_4 = list(ngrams(tokens, 4))

# everygrams extracts all n-grams from 1 to max_len
all_grams = list(everygrams(tokens, min_len=1, max_len=3))

Out[33]:

Console

NLTK n-gram extraction:

Bigrams: [('the', 'quick'), ('quick', 'brown'), ('brown', 'fox'), ('fox', 'jumps')]...
Trigrams: [('the', 'quick', 'brown'), ('quick', 'brown', 'fox'), ('brown', 'fox', 'jumps')]...
4-grams: [('the', 'quick', 'brown', 'fox'), ('quick', 'brown', 'fox', 'jumps')]...

Everygrams (1-3): 24 total
  Sample: [('the',), ('the', 'quick'), ('the', 'quick', 'brown'), ('quick',), ('quick', 'brown')]...

NLTK also provides FreqDist for counting and analyzing n-gram frequencies:

In[34]:

Code

from nltk import FreqDist

# Count bigram frequencies across a corpus
all_corpus_bigrams = []
for doc in corpus:
    doc_tokens = doc.lower().split()
    all_corpus_bigrams.extend(bigrams(doc_tokens))

bigram_freq = FreqDist(all_corpus_bigrams)

Out[35]:

Console

NLTK FreqDist analysis:
  Total bigrams: 21
  Unique bigrams: 15
  Most common: [(('machine', 'learning'), 4), (('i', 'love'), 2), (('deep', 'learning'), 2), (('learning', 'is'), 2), (('love', 'machine'), 1)]
  Frequency of ('machine', 'learning'): 4

NLTK's FreqDist wraps a Counter with additional NLP-specific methods. The .N() method returns the total number of tokens (sum of all counts), while .B() returns the number of unique types (vocabulary size). These are the type/token distinction applied to n-gram collections, which is important for computing statistics like the type-to-token ratio.