N-gram Language Models › Generating Text from N-gram Models
Language AI HandbookAll chapters N-gram Language Models › Generating Text from N-gram ModelsOne section · Sections in this chapter9 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
N-gram models can generate new text by sampling words according to their conditional probabilities. Starting from the start token, we repeatedly sample the next word given the current context, then advance the context window by one position.
The generation algorithm is simple: maintain a sliding window of the last words (the context), look up all words that have appeared after that context in training, sample from the distribution, add the sampled word to the output and to the context window, and repeat until </s> is generated or a maximum length is reached.
import random
def generate_sentence(model, max_length=20, seed=None):
"""Generate a sentence from an n-gram model by sampling."""
if seed is not None:
random.seed(seed)
toks = ["<s>"] * (model.n - 1) if model.n > 1 else []
for _ in range(max_length):
context = tuple(toks[-(model.n - 1) :]) if model.n > 1 else ()
candidates = []
for ngram, count in model.ngram_counts.items():
if ngram[:-1] == context:
word = ngram[-1]
prob = model.probability(word, context)
candidates.append((word, prob))
if not candidates:
break
words, probs = zip(*candidates)
next_word = random.choices(words, weights=probs, k=1)[0]
if next_word == "</s>":
break
toks.append(next_word)
start = model.n - 1 if model.n > 1 else 0
return " ".join(toks[start:])
generated_bigram = [generate_sentence(bigram_model, seed=i) for i in range(5)]
generated_trigram = [generate_sentence(trigram_model, seed=i) for i in range(5)]Generated sentences (bigram model): -------------------------------------------------- 1. the bird sat on the fish 2. the bird 3. the tree 4. the dog chased the dog sat on the cat ate the dog watched the dog sat on the dog ate 5. the cat Generated sentences (trigram model): -------------------------------------------------- 1. the dog 2. the bird 3. the bird flew over the tree 4. the dog chased the cat sat on the mat 5. the cat
The bigram model generates sentences that reflect local word patterns from the training data. The trigram model maintains coherence over slightly longer distances since it sees two words of context at each step. Both models tend to produce sentences similar to those in the training corpus: this is a feature, not a bug, since the models have learned the statistical patterns of the training text. With a larger and more diverse training corpus, the generated text would reflect the full variety of patterns the model has seen.
Temperature Sampling
We can control the randomness of generation with a temperature parameter . Temperature scaling adjusts the probability distribution over next words before sampling, sharpening it (more deterministic) or flattening it (more exploratory).
The idea is to scale log probabilities by dividing by before re-normalizing:
where:
- : the original probability of word from the language model
- : the temperature parameter, a positive real number typically between 0.1 and 2.0
- : dividing the log probability by amplifies or compresses differences between words
- : converts the scaled log probability back to a positive number for normalization
- The denominator normalizes the result so all probabilities sum to 1
The temperature controls the shape of the distribution:
- (low temperature): Differences in log probability are amplified, sharpening the distribution and concentrating probability on the most likely words.
- : Uses the original probabilities unchanged.
- (high temperature): Differences are compressed, flattening the distribution and giving rare words more chance.
def generate_with_temperature(model, temperature=1.0, max_length=20, seed=None):
"""Generate with temperature-scaled probabilities."""
if seed is not None:
random.seed(seed)
toks = ["<s>"] * (model.n - 1) if model.n > 1 else []
for _ in range(max_length):
context = tuple(toks[-(model.n - 1) :]) if model.n > 1 else ()
candidates = []
for ngram, count in model.ngram_counts.items():
if ngram[:-1] == context:
word = ngram[-1]
prob = model.probability(word, context)
candidates.append((word, prob))
if not candidates:
break
words, probs = zip(*candidates)
if temperature != 1.0:
log_probs = [math.log(p + 1e-10) / temperature for p in probs]
max_log = max(log_probs)
scaled_probs = [math.exp(lp - max_log) for lp in log_probs]
total = sum(scaled_probs)
probs = tuple(p / total for p in scaled_probs)
next_word = random.choices(words, weights=probs, k=1)[0]
if next_word == "</s>":
break
toks.append(next_word)
start = model.n - 1 if model.n > 1 else 0
return " ".join(toks[start:])
temperatures = [0.5, 1.0, 1.5]
temp_samples = {
t: [
generate_with_temperature(bigram_model, temperature=t, seed=i)
for i in range(3)
]
for t in temperatures
}Effect of Temperature on Generation: ============================================================ Temperature = 0.5: 1. the dog 2. the food 3. the bird flew over the dog ate the dog ate the cat Temperature = 1.0: 1. the bird sat on the fish 2. the bird 3. the tree Temperature = 1.5: 1. the bird sat on the fish 2. the bird 3. the branch
At low temperatures, the model gravitates toward the most common patterns, often producing the same output regardless of the random seed because the probability mass is concentrated on a single continuation. At high temperatures, it explores less likely paths and sometimes produces more creative but less coherent output. This temperature parameter appears throughout modern language modeling, including GPT-style systems. The same mathematical operation, scaling log probabilities before softmax, is used identically in transformer decoders.
The plots illustrate how temperature reshapes probability distributions. At T=0.5, the highest-probability word dominates the distribution. At T=1.5, the distribution flattens and rare words have a much better chance of being selected. This trade-off between coherence and diversity reappears in transformer-based language models, where it is controlled by exactly the same mechanism.