Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

N-gram Language Models › Practical Implementation with NLTK

Michael BrenndoerferMarch 23, 20252 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 8N-gram Language Models › Practical Implementation with NLTK

Reading section 10 of 15 · Practical Implementation with NLTK · 2 minSection 10 of 15 · 2 min

Section 10 of 15 · 2 min read

N-gram Language Models › Practical Implementation with NLTKOne section · Sections in this chapter10 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

NLTK provides a well-tested n-gram language modeling module that handles the bookkeeping of counting, vocabulary management, and probability computation:

In[33]:

Code

import nltk
from nltk.lm import MLE
from nltk.lm.preprocessing import padded_everygram_pipeline

# Download required tokenizer data
try:
    nltk.data.find("tokenizers/punkt_tab")
except LookupError:
    nltk.download("punkt_tab", quiet=True)

# Prepare training sentences as lists of tokens
train_sentences = [
    s.lower().split() for s in training_corpus.strip().split("\n") if s.strip()
]

# padded_everygram_pipeline generates all n-grams up to order n
# and handles <s> / </s> padding automatically
n = 3
train_data, padded_vocab = padded_everygram_pipeline(n, train_sentences)

nltk_model = MLE(n)
nltk_model.fit(train_data, padded_vocab)

Out[34]:

Console

NLTK 3-gram MLE model trained
Vocabulary size: 21

In[35]:

Code

# Query probabilities from the NLTK model
test_contexts = [
    ["the"],
    ["the", "cat"],
    ["sat", "on"],
]

nltk_results = []
for context in test_contexts:
    probs = {}
    for word in nltk_model.vocab:
        try:
            p = nltk_model.score(word, context)
            if p > 0:
                probs[word] = p
        except Exception:
            pass

    top_words = sorted(probs.items(), key=lambda x: -x[1])[:5]
    nltk_results.append((context, top_words))

Out[36]:

Console

NLTK Model Top Predictions:
--------------------------------------------------

Context: ['the']
  P(cat) = 0.300
  P(dog) = 0.250
  P(bird) = 0.150
  P(mat) = 0.100
  P(fish) = 0.050

Context: ['the', 'cat']
  P(</s>) = 0.333
  P(sat) = 0.167
  P(ate) = 0.167
  P(ran) = 0.167
  P(watched) = 0.167

Context: ['sat', 'on']
  P(the) = 1.000

The NLTK predictions match what we expect from our training data. Given ["the"], the model predicts words like "cat," "dog," and "bird" that frequently follow "the." Given ["sat", "on"], the model strongly predicts "the." NLTK's padded_everygram_pipeline handles the boundary token logic automatically. The MLE class implements the same count-based formula we derived earlier.

Key ParametersLink Copied

The key parameters for n-gram language models are:

n (n-gram order): Controls how many words of context the model considers. Unigrams ( $n=1$ ) ignore context entirely. Bigrams ( $n=2$ ) use one word of context. Trigrams ( $n=3$ ) were the production standard before neural models. Higher values require exponentially more training data.
temperature: Controls randomness during text generation. Values below 1.0 make generation more deterministic. Values above 1.0 make generation more diverse. Values between 0.7 and 1.2 work well for most applications.
start and end tokens: The <s> token provides context for sentence-initial words; prepend $n-1$ start tokens for an order- $n$ model. The </s> token allows the model to learn which words end sentences.
vocabulary handling: Case normalization reduces vocabulary size but loses capitalization information. An unknown token <UNK> handles out-of-vocabulary words at inference time. A minimum frequency threshold excludes rare words.