Repetition Penalties › N-gram Blocking
Language AI HandbookAll chapters Repetition Penalties › N-gram BlockingOne section · Sections in this chapter6 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The penalties we've explored so far are probabilistic: they reduce the likelihood of repetition without preventing it entirely. N-gram blocking takes a deterministic approach by absolutely forbidding the repetition of specific token sequences. This is a qualitatively different intervention: rather than adjusting the shape of the probability distribution, it removes certain tokens from consideration entirely.
Think of n-gram blocking as a constraint rather than a preference. The probabilistic penalties say "you should probably not repeat this token." N-gram blocking says "you are not allowed to repeat this exact sequence." The difference matters when the model is highly confident: a very strong preference can override a probabilistic penalty, but it cannot override an absolute constraint.
This guarantee comes at a cost. Absolute constraints remove tokens from consideration without evaluating whether their repetition would be problematic in context. A model blocked from repeating "the quick brown" cannot use that phrase again even if it naturally continues the same thought. The question is whether the benefit of the guarantee outweighs the loss of flexibility in specific use cases.
N-gram blocking prevents the model from generating any n-gram (sequence of n consecutive tokens) that has already appeared in the generated text. When the model would complete a forbidden n-gram, that token's probability is set to zero, forcing selection of a different continuation.
How N-gram Blocking Works
The penalties we've explored, repetition and frequency alongside presence penalties, all work by adjusting probabilities. They make repetition less likely but don't prevent it entirely. Sometimes a token's original probability is so high that even after penalization, it remains the most likely choice. For applications where exact repetition would be clearly wrong (legal documents, safety-critical outputs), we need a stronger guarantee.
N-gram blocking provides that guarantee through a fundamentally different mechanism: instead of adjusting probabilities, it eliminates certain tokens from consideration entirely by setting their probability to zero.
What Is an N-gram?
An n-gram is simply a contiguous sequence of n tokens. The terminology comes from computational linguistics:
- A bigram (2-gram) is a sequence of 2 tokens, like ["the", "cat"]
- A trigram (3-gram) is a sequence of 3 tokens, like ["sat", "on", "the"]
- A 4-gram is a sequence of 4 tokens, like ["the", "quick", "brown", "fox"]
N-gram blocking maintains a record of all n-grams that have appeared in the generated text. Before each token is sampled, the algorithm checks: "If I generate this token, will it complete an n-gram that already exists?" If so, that token is forbidden.
The Blocking Mechanism
Consider 3-gram blocking with the sequence "machine learning is powerful. Machine learning is". The current context ends with "Machine learning". If the model generates "is", it would create the trigram ["Machine", "learning", "is"], which already appeared earlier. N-gram blocking detects this and sets the probability of "is" to zero, forcing the model to choose a different continuation.
This is absolute prevention, not probabilistic discouragement. No matter how strongly the model wants to generate "is", it cannot. The token is masked out before sampling occurs. The model must find an alternative that does not complete any previously seen n-gram of length n. This forces a novel local phrase structure and keeps exact phrases from being copied from earlier in the generation.
def get_ngrams(token_ids, n):
"""
Extract all n-grams from a sequence of token IDs.
Args:
token_ids: List of token IDs
n: Size of n-grams to extract
Returns:
Set of n-gram tuples
"""
ngrams = set()
for i in range(len(token_ids) - n + 1):
ngram = tuple(token_ids[i : i + n])
ngrams.add(ngram)
return ngrams
def get_banned_tokens(token_ids, n):
"""
Get tokens that would create a repeated n-gram if generated next.
Args:
token_ids: List of previously generated token IDs
n: Size of n-grams to block
Returns:
Set of token IDs that should not be generated
"""
if len(token_ids) < n - 1:
return set()
# Get existing n-grams
existing_ngrams = get_ngrams(token_ids, n)
# Get the (n-1)-gram that would be completed by the next token
context = tuple(token_ids[-(n - 1) :])
# Find tokens that would complete a repeated n-gram
banned = set()
for ngram in existing_ngrams:
if ngram[:-1] == context:
banned.add(ngram[-1])
return bannedLet's trace through an example:
# Simulate a generation scenario
sentence = "The quick brown fox jumps over the lazy dog and the quick brown"
tokens = tokenizer.encode(sentence)
token_strings = [tokenizer.decode([t]) for t in tokens]Sentence: 'The quick brown fox jumps over the lazy dog and the quick brown' Tokens: ['The', ' quick', ' brown', ' fox', ' jumps', ' over', ' the', ' lazy', ' dog', ' and', ' the', ' quick', ' brown'] Existing 3-grams: ['the', 'quick', 'brown'] ['the', 'lazy', 'dog'] ['and', 'the', 'quick'] ['The', 'quick', 'brown'] ['over', 'the', 'lazy'] ['quick', 'brown', 'fox'] ['dog', 'and', 'the'] ['brown', 'fox', 'jumps'] ['lazy', 'dog', 'and'] ['jumps', 'over', 'the'] ['fox', 'jumps', 'over'] Context ends with: [' quick', ' brown'] Banned next tokens: [' fox']
The output shows all trigrams extracted from the sentence. Since the context ends with "quick brown" and the trigram ["quick", "brown", "fox"] already exists, generating "fox" next would create an exact repetition. N-gram blocking identifies this and adds "fox" to the banned token list, forcing the model to choose a different continuation.
Implementation in Generation
def generate_with_ngram_blocking(model, tokenizer, prompt, n, max_tokens=50):
"""
Generate text with n-gram blocking to prevent exact repetition.
Args:
model: Language model
tokenizer: Tokenizer
prompt: Input prompt string
n: Size of n-grams to block (e.g., 3 for trigram blocking)
max_tokens: Maximum tokens to generate
Returns:
Generated text string
"""
input_ids = tokenizer.encode(prompt, return_tensors="pt")
generated_ids = input_ids[0].tolist()
for _ in range(max_tokens):
with torch.no_grad():
outputs = model(torch.tensor([generated_ids]))
logits = outputs.logits[0, -1, :]
# Get banned tokens
banned_tokens = get_banned_tokens(generated_ids, n)
# Mask out banned tokens
for token_id in banned_tokens:
logits[token_id] = float("-inf")
# Apply temperature and sample
probs = F.softmax(logits / 0.8, dim=0)
next_token = torch.multinomial(probs, num_samples=1).item()
generated_ids.append(next_token)
if next_token == tokenizer.eos_token_id:
break
return tokenizer.decode(generated_ids, skip_special_tokens=True)With 2-gram blocking, the text avoids repeating any two consecutive tokens, which can make the output feel choppy since common phrases like "of the" can only appear once. With 3-gram blocking, the constraint relaxes slightly, allowing more natural flow while still preventing obvious repetition. At 4-gram blocking, only longer repeated phrases are blocked, preserving most natural language patterns while catching more egregious loops.
The practical range of useful n values is 2-5. Below 2 makes no sense (you cannot block unigrams with this approach), and above 5 the blocking is so permissive that it rarely activates. In Hugging Face's generate() function, the default for summarization tasks is often 3 or 4, which catches most copy-paste repetition without over-constraining the output.
Trade-offs of N-gram Blocking
N-gram blocking guarantees no exact phrase repetition but has notable limitations that make it unsuitable as a universal solution. Understanding these limitations helps you decide when the guarantee is worth the constraints.
Rigidity: N-gram blocking cannot distinguish between undesirable repetition and natural language patterns. Phrases like "on the other hand" or "in addition to" might be blocked after first use, even when appropriate. The algorithm has no understanding of semantics: it treats all repeated n-grams as equally problematic. A repeated cliche is blocked just as readily as a repeated argument, even though the latter repetition might be deliberate and rhetorically appropriate.
Local focus: The blocking only prevents exact token-level matches. "The cat sat" and "A cat sat" are different trigrams, so both could appear despite semantic similarity. Two sentences that say the same thing in slightly different words will both be generated without any penalty, even though the semantic repetition might be just as problematic as token-level repetition. This gap between token-level and semantic-level repetition is a fundamental limitation of all the techniques in this chapter: they operate on tokens, not meaning.
Brittleness: The blocking can force awkward workarounds when the model needs to repeat a phrase. If the phrase "neural network" is blocked after first use, the model must avoid it for the rest of the generation, even when it is the most precise term available. The forced alternatives may be less clear or less accurate. In technical writing, where precision of terminology matters, this rigidity can reduce quality significantly.
Smaller n values (2 or 3) are more restrictive but may hurt fluency. Larger values (4 or 5) allow more natural repetition while still preventing obvious loops. As a rule of thumb, 3-gram blocking is the most common choice for general summarization and generation, while 4-gram or 5-gram blocking is better for tasks where natural repetition of short phrases is expected.