Repetition Penalties: Preventing Loops in LLM Generation

Language models have a peculiar tendency to get stuck in loops. Ask GPT to write a story, and without intervention, you might get output like "The cat sat on the mat. The cat sat on the mat. The cat sat on the mat..." This repetitive behavior emerges from a fundamental property of autoregressive generation: at each step, the model selects tokens that maximize likelihood given the context, and if a pattern worked once, the same pattern often has high probability again.

Think of it like a river carving a channel: once water flows a particular path, the channel deepens and that path becomes increasingly easy to follow. In language models, each generated token strengthens the patterns already in the context, gradually making those exact patterns more attractive at future steps. A model generating a list might produce one item, then another that uses the same structure, then another, then another, until it is trapped forever in a loop it cannot escape on its own. The channel has been carved so deep that the water cannot climb out.

Repetition is not always undesirable. Certain phrases naturally repeat in language: "again and again", "more and more", or the rhythmic repetition in poetry and song lyrics. Technical writing legitimately repeats specialized terms: a machine learning paper must say "gradient descent" many times. Legal documents repeat precise phrasings deliberately. But uncontrolled repetition signals a failure mode where the model has collapsed into a degenerate loop, producing text that no human would write intentionally. The challenge is to suppress pathological repetition while preserving the natural repetition that language requires.

This chapter explores four techniques to prevent such loops while preserving natural language patterns. Each operates at a different level of granularity and with a different mathematical mechanism. Understanding all four gives you a toolkit to match the intervention to the specific failure mode you are encountering.

We'll examine three probabilistic approaches that modify the probability distribution during generation. Repetition penalty scales down the logits of previously used tokens. Frequency penalty applies increasingly strong penalties based on how many times each token has appeared. Presence penalty applies a flat penalty to any token that has appeared at all. We'll also explore n-gram blocking, a deterministic approach that prevents exact phrase repetition. Each technique offers different trade-offs between preventing loops and preserving natural repetition.

Before diving into solutions, we should understand why repetition happens. The answer lies in how autoregressive models generate text and how they are trained. This is not a superficial bug but a deep consequence of the architecture and training objective.

During training, language models learn to predict the next token given all previous tokens. They're optimized to assign high probability to tokens that frequently follow specific contexts in the training data. When a particular phrase or structure appears often in training text, the model learns to give it high probability in similar contexts. The model is, in a sense, a compressed summary of what patterns appeared in the training corpus, and repetitive patterns show up strongly in that corpus.

The problem manifests during generation. Suppose the model generates "The results show that" and then produces "the model performs well." Now "The results show that the model performs well" becomes part of the context. If the training data contained similar patterns of presenting multiple results, the phrase "The results show that" might again have high probability, leading to another similar sentence. Each repetition reinforces the pattern in the context, making further repetition even more likely. The model is not doing anything wrong by its own internal logic: it is faithfully predicting what typically follows such contexts. The training data likely contained many documents that followed this structure, so the model learned to associate such contexts with such continuations.

There is also a more subtle contributor: maximum likelihood training creates pressure toward safe, high-probability tokens. During training, the model is penalized for predicting low-probability tokens and rewarded for predicting high-probability ones. This creates a systematic bias toward tokens the model is confident about, and confidence often correlates with familiarity. A token that just appeared is familiar: its embedding has just been activated, similar patterns have just been processed. It is, in a sense, "top of mind" for the model, making it an appealing choice at the next step.

The main point is that repetition is not random noise but a systematic attractor in the probability distribution. Once a phrase is in context, it raises the probability of similar phrases, which when generated raise the probability of even more similar phrases. Left unchecked, this positive feedback loop converges to a fixed point where the model endlessly repeats the same content.

The output may contain repetitive phrases or sentence structures. While sampling strategies like temperature and nucleus sampling add randomness, they don't specifically target repetition. A token that appeared recently still has the same probability as before, so the model can easily select it again. Temperature and sampling truncation change the shape of the distribution but do not specifically push previously used tokens toward lower probability. We need dedicated mechanisms for that.

Repetition often starts subtly and escalates. The model might repeat a word, then a phrase, then entire sentences. This acceleration happens because each repetition adds more of the same pattern to the context, which the model uses to predict the next token. The context becomes increasingly dominated by the repeated content, making continuation of that content more and more likely.

Consider a model generating a list. After producing "First, we need to consider..." it might generate "Second, we need to consider..." and then "Third, we need to consider...". So far, this is reasonable parallel structure. But without intervention, the model might continue with "Fourth, we need to consider..." long after the list should have ended, trapped in a pattern that keeps reinforcing itself. The parallel structure pattern is strong enough in the training data that the model keeps predicting more of it.

The feedback loop has a characteristic shape. Early in generation, repetition probability is roughly at baseline. After a repeated phrase first appears, its probability edges up slightly. The second repetition raises it further. By the third or fourth repetition, the probability can be so high that almost any intervention short of hard blocking is insufficient. This is why the problem is called a "degenerate loop": the model has reached a stable fixed point from which it cannot easily escape through normal sampling variation. Understanding this escalation pattern helps explain why even moderate penalties are effective: they flatten the early growth of repetition probability before the feedback loop has a chance to establish itself.

The feedback loop also helps explain why beam search, a common decoding strategy, is particularly vulnerable to repetition. Beam search maintains multiple high-probability hypotheses and keeps expanding the most likely ones. Since repetition temporarily maximizes local probability, beam search selects into loops very aggressively. This is why n-gram blocking is most commonly associated with beam search in summarization: without hard blocking, beam search almost always collapses into repetitive output.

The most direct approach to preventing repetition modifies the logits for tokens that have already appeared in the generated sequence. The repetition penalty, introduced by Keskar et al. (2019) in the CTRL paper, divides the logits of previously seen tokens by a penalty factor, making them less likely to be selected.

Think of the repetition penalty as a "familiarity tax." Every token that has appeared before faces a handicap when competing for the next position. Fresh tokens, those not yet used, compete on their own merits. Used tokens start at a disadvantage proportional to the strength of the penalty. The effect is to tilt the playing field toward novelty without eliminating previously used tokens entirely.

This approach has an appealing simplicity. It requires only a single parameter, the penalty factor $\theta$, which has a natural interpretation: at $\theta = 1.0$ there is no effect, and higher values impose stronger penalties. Implementation requires only tracking which tokens have appeared, a set that is trivially maintained as generation proceeds. The computational overhead is minimal: one division or multiplication per token in the used-token set per generation step.

To understand how repetition penalty works, we need to trace the path from model output to token selection. When a language model predicts the next token, it doesn't output probabilities directly. Instead, it produces logits: raw, unnormalized scores for each token in the vocabulary. These logits then pass through the softmax function to become probabilities, and finally, we sample from that probability distribution.

This pipeline gives us a natural intervention point. If we want to reduce the probability of certain tokens, we can modify their logits before softmax. The question is: how should we modify them?

The Core InsightLink Copied

Our goal is straightforward: make previously used tokens less likely to appear again. Since softmax converts logits to probabilities, reducing a logit reduces its corresponding probability. But there's a subtlety. Logits can be positive or negative, and the operation that reduces a positive number (division) increases a negative number (makes it closer to zero). We need an operation that pushes logits in the "less likely" direction regardless of their sign.

The solution is a piecewise function that applies different operations based on the sign of the logit. Let $z_i$ be the logit for token $i$ and let $G$ be the set of tokens that have already appeared in the generated sequence. The repetition penalty modifies logits as follows:

where:

• $z_i$: the original logit for token $i$, the raw score from the model before softmax
• $z'_i$: the modified logit after applying the penalty
• $\theta$: the repetition penalty factor, typically in the range $[1.0, 2.0]$
• $G$: the set of token indices that have appeared in the context
• $i \in G$: notation meaning "token $i$ is a member of set $G$" (i.e., token $i$ has appeared before)

The formula reads: for each token, check if it has appeared before. If it has and its logit is positive, divide by the penalty factor. If it has and its logit is zero or negative, multiply by the penalty factor. If it hasn't appeared, leave it unchanged.

Why does this formula make sense? Notice that in both the positive and negative cases, the modification moves the logit away from large positive values. For a positive logit, division reduces the magnitude (e.g., 4.0 becomes 2.0). For a negative logit, multiplication increases the magnitude in the negative direction (e.g., -2.0 becomes -3.0). Both operations reduce the value in the mathematical sense (the result is smaller or more negative), and because softmax is an increasing function of each logit, a smaller logit always maps to a smaller probability. The penalty is thus correctly reducing probability in both cases.

Why the Asymmetric Treatment?Link Copied

This asymmetry might seem arbitrary, but it follows directly from how softmax works. Recall that softmax converts a logit $z_i$ to a probability:

The probability depends on $e^{z_i}$, the exponential of the logit. The exponential function is strictly increasing: larger inputs produce larger outputs. So to reduce $P(i)$, we must reduce $z_i$.

where:

• $P(i)$: the probability assigned to token $i$
• $e^{z_i}$: the exponential of token $i$'s logit, which must be positive
• $\sum_j e^{z_j}$: the sum of exponentials across all tokens in the vocabulary, serving as the normalizing constant

For a positive logit like $z_i = 2.0$, dividing by $\theta = 1.5$ gives $z'_i = 1.33$. Since $1.33 < 2.0$, we've reduced the logit and thus reduced the probability. Division works as intended.

For a negative logit like $z_i = -2.0$, the same division gives $z'_i = -2.0 / 1.5 = -1.33$. But $-1.33 > -2.0$ (it's closer to zero), so we've increased the logit and the probability. Division fails here.

The fix is multiplication. Multiplying $-2.0$ by $\theta = 1.5$ gives $z'_i = -3.0$. Since $-3.0 < -2.0$, we've pushed the logit further negative, reducing the probability. Both operations, division for positive logits and multiplication for negative ones, consistently reduce probability.

Why does this asymmetry arise in the first place? The root cause is that division by a number greater than 1 is not a "reduction" operation for negative numbers. Division by 1.5 takes -2.0 to -1.33, which is closer to 0 and thus larger. Multiplication by the same factor takes -2.0 to -3.0, which is further from 0 and thus smaller. The piecewise formula selects the operation that achieves the desired effect in each case.

The Neutral CaseLink Copied

When $\theta = 1.0$, the penalty has no effect. Dividing by 1 or multiplying by 1 leaves any number unchanged. This gives us a natural "off switch" for the penalty and a baseline for comparison. As $\theta$ increases above 1.0, the penalty strengthens, pushing repeated tokens further toward improbability.

The visualization shows the key insight: for positive logits, division pulls values toward zero (reducing probability), while for negative logits, multiplication pushes values further negative (also reducing probability). Both transformations achieve the same goal through different arithmetic operations. The dashed line at $\theta = 1.0$ shows the identity case where the penalty has no effect, serving as a useful reference for understanding how each higher $\theta$ value moves tokens further from their original probability.

Let's implement the repetition penalty from scratch to see exactly how it works:

The table shows how the repetition penalty redistributes probability mass. Tokens that appeared in the context (marked with *) have their probabilities reduced, while tokens that haven't appeared receive proportionally higher probabilities. Notice that "sat" (token 2), which had the highest original logit, is no longer the most likely token after applying the penalty. The probability mass that was taken from the penalized tokens flows to unpenalized tokens, effectively boosting the chances of less familiar choices.

The repetition penalty parameter requires careful tuning. Too low, and repetition persists. Too high, and the model avoids necessary repetition, producing awkward text that never uses "the" or "a" more than once. Finding the right value is partly a matter of matching the penalty to the type of content being generated.

The challenge is that function words like "the", "a", "is", "and" legitimately appear hundreds of times in any long document. A penalty strong enough to suppress pathological repetition of content words will, if applied uniformly, also heavily penalize these necessary function words. The repetition penalty applies to all tokens equally regardless of their type, which means tuning it always involves a trade-off between suppressing content repetition and preserving grammatical fluency. This limitation motivates the frequency and presence penalties discussed later, which can be calibrated to allow some repetition (of function words) while punishing excessive repetition (of content words).

Common values and their effects:

• $\theta = 1.0$: No penalty applied. Baseline behavior with full repetition potential.
• $\theta = 1.1$ to $1.2$: Light penalty. Gently discourages repetition while allowing natural patterns. Good starting point for most applications.
• $\theta = 1.3$ to $1.5$: Moderate penalty. Noticeably reduces repetition. May affect fluency for texts requiring repeated terms (technical writing, legal documents).
• $\theta > 1.5$: Strong penalty. Significantly suppresses any repeated token. Can produce stilted or unnatural text, especially for longer outputs.

The heatmap reveals an important pattern: the probability reduction depends on both the penalty strength and the token's original standing. Tokens with moderate positive logits (around 1-3) experience the largest percentage reductions because they start with meaningful probability that can be substantially reduced. Very high logits still dominate even after penalty, while very low logits have little probability to lose. This means that the most problematic tokens, the ones the model is most confident about and most likely to repeat, are also the most resistant to moderate penalties. Extremely confident predictions require strong penalties to dislodge, while less confident predictions can be steered with mild ones.

While repetition penalty treats all repeated tokens identically, frequency penalty scales with occurrence count. A token used once receives a small penalty; a token used ten times receives ten times that penalty. This graduated approach allows some repetition while strongly discouraging excessive use of any single token.

Think of frequency penalty as a progressive tax on word usage. The first time you use a word, you pay a small surcharge. The second time, you pay twice that surcharge. By the tenth use, the cumulative surcharge is substantial enough to make the word expensive to use again. Function words that legitimately appear throughout a document incur some cost, but content words that are being pathologically overused incur enormous cumulative cost. The graduated nature of the penalty naturally adapts to the difference between normal and pathological repetition.

This approach is particularly well suited to long-form generation tasks where you want the model to explore a topic broadly rather than drilling into one narrow aspect. When writing a blog post, you want to cover multiple subtopics and use diverse vocabulary. Frequency penalty naturally pushes the model in this direction by making it progressively more costly to return to the same words and phrases that have already appeared.

Repetition penalty has a limitation: it treats a token that appeared once the same as one that appeared fifty times. Both receive the same penalty. But intuitively, we might want to allow occasional repetition (the word "the" naturally appears multiple times in most texts) while strongly discouraging tokens that have become overused. This calls for a penalty that accumulates with each occurrence.

From Binary to Graduated PenaltiesLink Copied

Frequency penalty introduces a simple but powerful shift: instead of asking "has this token appeared?", we ask "how many times has this token appeared?" The penalty then scales proportionally with the answer.

The mechanism is additive rather than multiplicative. For each occurrence of a token, we subtract a fixed amount from its logit. Let $c_i$ denote the number of times token $i$ has appeared in the generated sequence. The frequency penalty modifies logits as:

where:

• $z_i$: the original logit for token $i$
• $z'_i$: the modified logit after applying the penalty
• $\alpha$: the frequency penalty coefficient, typically in the range $[0, 2]$, controlling how strongly each occurrence is penalized
• $c_i$: the count of token $i$ in the generated sequence (0 if the token has not appeared)

The formula subtracts $\alpha$ once for each time the token has appeared. If a token appeared three times and $\alpha = 0.5$, we subtract $0.5 \times 3 = 1.5$ from its logit. Note that if $c_i = 0$, the formula applies no penalty at all, since subtracting zero has no effect. New tokens are left untouched.

Why does this formula make sense? Each appearance of a token is a "vote" that the model has already expressed its preference for that token. The frequency penalty accumulates these votes into a debt that the token must overcome to be chosen again. The more it has been chosen in the past, the heavier the debt, and the harder it must compete against fresher alternatives. This creates a natural tendency toward exploration: once the model has heavily used a token, it becomes progressively less attractive, and the model naturally turns to other options.

Why Additive WorksLink Copied

Unlike repetition penalty, frequency penalty doesn't need to handle positive and negative logits differently. Subtraction always reduces a number, regardless of its sign. Subtracting from a positive logit makes it smaller (or even negative). Subtracting from a negative logit makes it more negative. Both operations reduce the logit and thus reduce the probability after softmax.

This simplicity is a feature. The formula is easy to implement, easy to understand, and produces predictable behavior: each occurrence adds the same "cost" to using that token again. The mathematical mechanism is also more transparent: we know exactly how much the logit has been reduced by looking at the count, whereas the multiplicative mechanism of repetition penalty interacts with the original logit value in a more complex way.

Linear Scaling in ActionLink Copied

The linearity of frequency penalty creates a distinctive pattern. Early occurrences of a token receive mild penalties, allowing natural repetition of common words. But as a token accumulates, the penalty grows relentlessly. Consider $\alpha = 0.5$:

By the tenth occurrence, the token's logit has been reduced by 5.0, a substantial penalty that makes it far less competitive against tokens that haven't been overused. This graduated pressure naturally prevents any single token from dominating the output without harsh early constraints. Notice that a token appearing twice is only twice as penalized as one appearing once, which is quite lenient: the real impact accumulates at higher counts where pathological repetition is more likely to be occurring.

The key difference is apparent: tokens with higher counts receive proportionally stronger penalties. "The", which appeared 5 times, has its logit reduced by 2.5 (5 × 0.5), while "data", which appeared once, only loses 0.5 from its logit. This graduated penalty is particularly effective for suppressing overused words while allowing reasonable repetition of less common terms. Notice that "learning" and "result", which have not appeared at all, receive no penalty and maintain their original probability relative to each other.

Presence penalty takes an even simpler approach: penalize any token that has appeared, regardless of how many times. This binary penalty treats "appeared once" and "appeared twenty times" identically. At first glance this might seem crude compared to the graduated precision of frequency penalty, but it captures a different goal.

Think of presence penalty as an entry tax rather than a per-use tax. The first time a token appears in the generated text, it pays a fixed fee. Subsequent appearances pay nothing extra. This structure is ideal when your primary concern is vocabulary breadth rather than repetition frequency. You want the model to explore as many different words and concepts as possible, with each new token representing a new idea being introduced. Once a concept is on the table, having it mentioned again is less concerning than having entirely new concepts go unmentioned.

This makes presence penalty particularly valuable for brainstorming tasks, where the goal is to generate as many distinct ideas as possible in a limited number of tokens. It also works well for summarization, where you want to ensure coverage of multiple aspects of the source document rather than repeatedly circling around one or two salient points. In both cases, the question is "have I introduced this yet?" rather than "how many times have I used this?", which maps directly to the binary structure of presence penalty.

We've now seen two approaches: repetition penalty treats all repeated tokens equally (ignoring count), while frequency penalty scales with count. Presence penalty takes the opposite extreme from frequency penalty: it also ignores count, but uses the simpler additive mechanism.

The Binary QuestionLink Copied

Presence penalty asks the simplest possible question about each token: "Have you appeared before, yes or no?" If yes, apply a fixed penalty. If no, leave the logit unchanged. The number of previous occurrences is irrelevant.

To express this mathematically, we use an indicator function, a standard notation for encoding yes/no conditions as 1/0 values:

where:

• $z_i$: the original logit for token $i$
• $z'_i$: the modified logit after applying the penalty
• $\beta$: the presence penalty coefficient, controlling how strongly any prior appearance is penalized
• $\mathbb{1}[i \in G]$: the indicator function, which equals 1 if token $i$ has appeared (is in set $G$), and 0 otherwise
• $G$: the set of tokens that have appeared in the generated sequence

Reading the Indicator FunctionLink Copied

The notation $\mathbb{1}[i \in G]$ might look intimidating, but it simply acts like a switch. The condition inside the brackets, "$i \in G$" (token $i$ is in set $G$), is evaluated as true or false. The indicator function converts this to a number:

• If the condition is true (token has appeared): $\mathbb{1}[\text{true}] = 1$
• If the condition is false (token has not appeared): $\mathbb{1}[\text{false}] = 0$

Substituting back into the formula:

• For a token that has appeared: $z'_i = z_i - \beta \cdot 1 = z_i - \beta$
• For a token that hasn't appeared: $z'_i = z_i - \beta \cdot 0 = z_i$

The indicator function elegantly handles both cases in a single equation. In practice, you can implement this without any indicator function machinery: simply iterate over the set of appeared tokens and subtract $\beta$ from each. The indicator notation is useful for mathematical exposition but translates to a simple loop in code.

Why does this formula make sense? Because the logit space is linear in its effect on probability (through the exponentiation in softmax), a fixed subtraction of $\beta$ from a logit translates to dividing the unnormalized probability by $e^\beta$. So presence penalty is equivalent to multiplying all appeared tokens' unnormalized probabilities by a constant factor $e^{-\beta}$, which is less than 1 for any positive $\beta$. This is a clean, uniform discounting of all previously seen tokens.

Presence vs. Frequency: A ComparisonLink Copied

The contrast with frequency penalty is instructive. Frequency penalty applies $\alpha$ for each occurrence, so a token appearing 10 times receives 10 times the penalty of one appearing once. Presence penalty applies $\beta$ exactly once regardless of occurrence count. Whether a token appeared 1 time or 100 times, it receives the same penalty $\beta$.

This makes presence penalty particularly effective for encouraging vocabulary diversity. Once a word has been used, it faces a fixed "tax" on appearing again. The model is pushed to explore alternatives, to find synonyms, to vary its phrasing. It is a blunt instrument compared to frequency penalty's graduated pressure, but for tasks like brainstorming or generating diverse lists, that bluntness is exactly what is needed. The model cannot "amortize" the penalty by spreading uses over time, the way it can with frequency penalty at low counts. The penalty is immediate and permanent from the first use.

The contrast is stark. Presence penalty (dashed lines) jumps to its full value after the first occurrence and stays flat. Frequency penalty (solid lines) starts at zero and grows steadily. For tokens that appear many times, frequency penalty eventually dominates; for tokens that appear just once or twice, presence penalty applies stronger immediate pressure. If you want a token to be discouraged immediately upon first use, presence penalty is the sharper tool. If you want the penalty to grow with overuse rather than applying immediately, frequency penalty is better suited.

Notice how much simpler this implementation is compared to frequency penalty. We do not need to maintain a count dictionary, only a set of token IDs that have appeared. For long generation runs with large vocabularies, this can be a meaningful efficiency advantage, particularly when the set of appeared tokens is substantially smaller than the full vocabulary. The set can be maintained with O(1) average insertion and lookup time, making the penalty application efficient even for very long sequences.

Let's see how the three penalties differ when applied to the same scenario:

The visualization reveals each penalty's character:

• Repetition penalty (blue) uniformly reduces all appeared tokens, regardless of count. "data" (×1) and "the" (×5) are penalized identically, which is the key limitation of this approach.
• Frequency penalty (red) creates graduated reductions: "the" (×5) drops dramatically while "data" (×1) drops minimally. This is the most fine-grained of the three approaches.
• Presence penalty (green) applies equal reduction to all appeared tokens, similar to repetition penalty but using additive rather than multiplicative adjustment. The difference from repetition penalty is visible in tokens with very negative logits, but for the equal-logit scenario here, the two approaches look similar.

The penalties we've explored so far are probabilistic: they reduce the likelihood of repetition without preventing it entirely. N-gram blocking takes a deterministic approach by absolutely forbidding the repetition of specific token sequences. This is a qualitatively different intervention: rather than adjusting the shape of the probability distribution, it removes certain tokens from consideration entirely.

Think of n-gram blocking as a constraint rather than a preference. The probabilistic penalties say "you should probably not repeat this token." N-gram blocking says "you are not allowed to repeat this exact sequence." The difference matters when the model is highly confident: a very strong preference can override a probabilistic penalty, but it cannot override an absolute constraint.

This guarantee comes at a cost. Absolute constraints remove tokens from consideration without evaluating whether their repetition would be problematic in context. A model blocked from repeating "the quick brown" cannot use that phrase again even if it naturally continues the same thought. The question is whether the benefit of the guarantee outweighs the loss of flexibility in specific use cases.

The penalties we've explored, repetition and frequency alongside presence penalties, all work by adjusting probabilities. They make repetition less likely but don't prevent it entirely. Sometimes a token's original probability is so high that even after penalization, it remains the most likely choice. For applications where exact repetition would be clearly wrong (legal documents, safety-critical outputs), we need a stronger guarantee.

N-gram blocking provides that guarantee through a fundamentally different mechanism: instead of adjusting probabilities, it eliminates certain tokens from consideration entirely by setting their probability to zero.

What Is an N-gram?Link Copied

An n-gram is simply a contiguous sequence of n tokens. The terminology comes from computational linguistics:

• A bigram (2-gram) is a sequence of 2 tokens, like ["the", "cat"]
• A trigram (3-gram) is a sequence of 3 tokens, like ["sat", "on", "the"]
• A 4-gram is a sequence of 4 tokens, like ["the", "quick", "brown", "fox"]

N-gram blocking maintains a record of all n-grams that have appeared in the generated text. Before each token is sampled, the algorithm checks: "If I generate this token, will it complete an n-gram that already exists?" If so, that token is forbidden.

The Blocking MechanismLink Copied

Consider 3-gram blocking with the sequence "machine learning is powerful. Machine learning is". The current context ends with "Machine learning". If the model generates "is", it would create the trigram ["Machine", "learning", "is"], which already appeared earlier. N-gram blocking detects this and sets the probability of "is" to zero, forcing the model to choose a different continuation.

This is absolute prevention, not probabilistic discouragement. No matter how strongly the model wants to generate "is", it cannot. The token is masked out before sampling occurs. The model must find an alternative that does not complete any previously seen n-gram of length n. This forces a novel local phrase structure and keeps exact phrases from being copied from earlier in the generation.

The output shows all trigrams extracted from the sentence. Since the context ends with "quick brown" and the trigram ["quick", "brown", "fox"] already exists, generating "fox" next would create an exact repetition. N-gram blocking identifies this and adds "fox" to the banned token list, forcing the model to choose a different continuation.

With 2-gram blocking, the text avoids repeating any two consecutive tokens, which can make the output feel choppy since common phrases like "of the" can only appear once. With 3-gram blocking, the constraint relaxes slightly, allowing more natural flow while still preventing obvious repetition. At 4-gram blocking, only longer repeated phrases are blocked, preserving most natural language patterns while catching more egregious loops.

The practical range of useful n values is 2-5. Below 2 makes no sense (you cannot block unigrams with this approach), and above 5 the blocking is so permissive that it rarely activates. In Hugging Face's generate() function, the default for summarization tasks is often 3 or 4, which catches most copy-paste repetition without over-constraining the output.

N-gram blocking guarantees no exact phrase repetition but has notable limitations that make it unsuitable as a universal solution. Understanding these limitations helps you decide when the guarantee is worth the constraints.

Rigidity: N-gram blocking cannot distinguish between undesirable repetition and natural language patterns. Phrases like "on the other hand" or "in addition to" might be blocked after first use, even when appropriate. The algorithm has no understanding of semantics: it treats all repeated n-grams as equally problematic. A repeated cliche is blocked just as readily as a repeated argument, even though the latter repetition might be deliberate and rhetorically appropriate.

Local focus: The blocking only prevents exact token-level matches. "The cat sat" and "A cat sat" are different trigrams, so both could appear despite semantic similarity. Two sentences that say the same thing in slightly different words will both be generated without any penalty, even though the semantic repetition might be just as problematic as token-level repetition. This gap between token-level and semantic-level repetition is a fundamental limitation of all the techniques in this chapter: they operate on tokens, not meaning.

Brittleness: The blocking can force awkward workarounds when the model needs to repeat a phrase. If the phrase "neural network" is blocked after first use, the model must avoid it for the rest of the generation, even when it is the most precise term available. The forced alternatives may be less clear or less accurate. In technical writing, where precision of terminology matters, this rigidity can reduce quality significantly.

Smaller n values (2 or 3) are more restrictive but may hurt fluency. Larger values (4 or 5) allow more natural repetition while still preventing obvious loops. As a rule of thumb, 3-gram blocking is the most common choice for general summarization and generation, while 4-gram or 5-gram blocking is better for tasks where natural repetition of short phrases is expected.

In practice, repetition penalties work alongside temperature, top-k, and nucleus sampling. The interactions between these components matter for understanding and tuning generation quality. Each technique operates at a different stage of the logit-to-token pipeline, and applying them in the correct order is essential for the intended behavior.

The typical processing order is well-defined and follows a natural logic:

1. Get raw logits from the model's final layer
2. Apply repetition/frequency/presence penalties to adjust logit values for previously used tokens
3. Apply temperature scaling to control distribution sharpness (dividing logits by temperature)
4. Apply top-k or nucleus truncation to limit candidates to the most promising tokens
5. Sample from the resulting distribution

This order is not arbitrary. Repetition penalties should be applied before temperature because the penalty magnitudes are calibrated for logit-scale values. If you apply temperature first (which rescales all logits), the same penalty value would have a different effective strength depending on the temperature used. Applying penalties first ensures consistent behavior regardless of temperature.

Similarly, sampling truncation (top-k or nucleus) should come after penalties because the penalties change which tokens are most competitive. If you truncated first, then applied penalties, you might inadvertently keep penalized tokens in the candidate set while excluding better alternatives. Applying penalties before truncation ensures the truncation operates on the post-penalty distribution, correctly identifying the best candidates after repetition is accounted for.

Comparing the outputs reveals how each penalty type shapes generation. Without penalties, the model may fall into repetitive patterns. Repetition penalty alone provides uniform discouragement of repeated tokens. Frequency penalty creates graduated pressure that builds as tokens accumulate, while presence penalty encourages the model to continuously introduce new vocabulary. The combined approach balances these effects, often producing the most natural-sounding output by gently discouraging repetition without forcing unnatural word choices.

You can also combine penalties with n-gram blocking for a hybrid approach: the probabilistic penalties soften the probability distribution to make repetition less attractive, while n-gram blocking provides a hard guarantee against the most egregious exact repetitions. This combination is particularly effective for tasks like abstractive summarization, where you want rich vocabulary diversity but also need to ensure the summary does not copy entire sentences from the source document.

To solidify understanding, let's trace through a concrete numerical example that shows exactly how frequency penalty modifies the generation decision at one step. We'll use a toy vocabulary of 5 tokens and walk through every calculation.

Suppose we are generating text and have just produced the sequence: "the cat sat on the mat and the cat". The current context window contains these 8 tokens. We track their counts:

• "the": appeared 3 times
• "cat": appeared 2 times
• "sat": appeared 1 time
• "on": appeared 1 time
• "mat": appeared 1 time
• "and": appeared 1 time

The model produces the following raw logits for the next token (we use a tiny vocabulary for clarity):

Without any penalty, softmax would strongly favor "the" and "cat". Let's apply frequency penalty with $\alpha = 0.5$.

Step 1: Compute modified logits. For each token, subtract $\alpha \times c_i$:

• "the": $3.2 - 0.5 \times 3 = 3.2 - 1.5 = 1.7$
• "cat": $2.8 - 0.5 \times 2 = 2.8 - 1.0 = 1.8$
• "sat": $2.1 - 0.5 \times 1 = 2.1 - 0.5 = 1.6$
• "again": $1.5 - 0.5 \times 0 = 1.5$ (no penalty: first use)
• "purred": $1.0 - 0.5 \times 0 = 1.0$ (no penalty: first use)

Step 2: Apply softmax to modified logits. We compute $e^{z'_i}$ for each:

• "the": $e^{1.7} \approx 5.47$
• "cat": $e^{1.8} \approx 6.05$
• "sat": $e^{1.6} \approx 4.95$
• "again": $e^{1.5} \approx 4.48$
• "purred": $e^{1.0} \approx 2.72$

Sum: $5.47 + 6.05 + 4.95 + 4.48 + 2.72 = 23.67$

Probabilities: "the" = 0.231, "cat" = 0.256, "sat" = 0.209, "again" = 0.189, "purred" = 0.115

Step 3: Compare to unpenalized probabilities. With the original logits, the top choices were overwhelmingly "the" and "cat". After frequency penalty, "cat" is still favored but by a much smaller margin, and "again" and "purred" have become viable competitors. The penalty has not eliminated the model's preference but has substantially leveled the playing field, giving novel tokens a real chance at selection.

The key insight from this example is that frequency penalty is most effective when the penalized tokens have modest logit advantages over alternatives. If "the" had a logit of 10.0 rather than 3.2, the same $\alpha = 0.5$ with count 3 would reduce it to 8.5, which would still completely dominate. This is why very confident model predictions can resist even strong penalties, and why n-gram blocking is sometimes necessary as a fallback guarantee.

The choice between penalty types depends on your generation task. Each technique has a domain where it performs best, and using the wrong technique can introduce problems while failing to solve the ones you have.

Repetition Penalty works well as a general-purpose solution. It's the most widely implemented (available in Hugging Face's generate() method) and handles most cases adequately. Use it when you want a simple, effective baseline for reducing repetition, particularly when you're not sure which penalty type is most appropriate. The single-parameter interface makes it easy to tune and reason about. Typical starting value: $\theta = 1.1$ to $1.2$.

Frequency Penalty excels when you want to allow some repetition while preventing excessive use. It's particularly useful for:

• Long-form content where occasional word repetition is acceptable but word overuse is not
• Creative writing where you want natural variation without harsh constraints
• Technical writing where certain terms must repeat but shouldn't dominate

Presence Penalty promotes vocabulary diversity and topic breadth. Consider it for:

• Brainstorming and idea generation where you want many distinct concepts
• Summarization where you want to cover multiple points without redundancy
• Conversational agents that should avoid fixating on specific words

N-gram Blocking provides guaranteed protection against exact repetition. Use it when:

• Exact phrase repetition would be clearly wrong (e.g., safety-critical applications)
• You need deterministic behavior rather than probabilistic reduction
• Other penalties aren't sufficiently preventing loops

For many applications, combining moderate repetition penalty ($\theta \approx 1.1$-$1.2$) with nucleus sampling provides a good balance between preventing loops and maintaining fluent output.

When implementing repetition penalties in generation, these parameters control behavior:

• repetition_penalty (float, typically 1.0-2.0): Multiplicative penalty applied to logits of repeated tokens. A value of 1.0 applies no penalty. Values around 1.1-1.3 provide gentle repetition reduction. Values above 1.5 strongly discourage any repetition.

• frequency_penalty (float, typically 0-2.0): Additive penalty proportional to token occurrence count. Higher values increasingly penalize frequently used tokens. OpenAI's API uses this parameter with typical values of 0-1.0.

• presence_penalty (float, typically 0-2.0): Flat additive penalty for tokens that have appeared at all. Encourages vocabulary diversity. Also used in OpenAI's API with typical values of 0-1.0.

• no_repeat_ngram_size (int): Size of n-grams to block from repeating. In Hugging Face's generate(), setting this to 2 prevents any bigram repetition, 3 prevents trigram repetition, etc. Set to 0 to disable.

• encoder_no_repeat_ngram_size (int): For encoder-decoder models, prevents generating n-grams that appear in the encoder input. Useful for abstractive summarization to avoid copying source phrases.

Repetition penalties, for all their utility, have several important limitations that practitioners need to understand before deploying them in production systems.

The most fundamental limitation is that all these techniques operate on token identity, not semantic content. They can prevent the model from repeating the exact token sequence ["neural", "network"] but cannot prevent semantic repetition: sentences that convey the same information using different words. A model avoiding "neural network" might say "the artificial intelligence system" instead, producing essentially identical meaning with no token overlap. If your actual concern is semantic redundancy rather than surface-level repetition, these token-level techniques address only a symptom of the problem, not the underlying cause. Semantic-level repetition detection would require computing embeddings and measuring semantic similarity, which is computationally much more expensive.

The tuning challenge is also significant. The right parameter values depend heavily on the task, the model, the prompt length, and the expected output length. There are no universal values that work well across all settings. A $\theta = 1.5$ that prevents pathological loops in story generation might produce stilted, awkward text in a domain requiring technical precision. Frequency penalty values that work well for a 200-token response may over-penalize function words in a 2000-token essay. This means practitioners typically need to run empirical evaluations with human raters to find good settings for each deployment context, which requires significant time and resources.

There is also an inherent tension between repetition prevention and coherence. Language models achieve coherence partly through consistent use of terminology and consistent patterns of expression. A model that never repeats a content word may feel incoherent, jumping from synonym to synonym in ways that are hard to follow. Repetition and coherence are not fully separable: some amount of repetition is needed for text to feel like a unified whole rather than a collection of unrelated sentences. Strong penalties that eliminate most repetition can make text feel fragmented, even if it is technically non-repetitive. Finding the balance requires accepting that you cannot fully optimize both goals simultaneously.

Finally, these methods are reactive rather than proactive. They respond to repetition after it begins to occur, rather than preventing the model from entering repetition-prone states in the first place. More sophisticated approaches to this problem include training with diversity-promoting objectives, using contrastive decoding to compare outputs with a smaller "amateur" model, or training with reinforcement learning from human feedback where repetitive outputs receive lower rewards. These methods address the root cause rather than patching the symptom at inference time, but they require access to the training process, which is not always available.

Repetition is a common failure mode in language model generation, emerging from the autoregressive process where patterns reinforce themselves through context accumulation. The feedback loop at the heart of this failure mode is well-understood: each generated token becomes part of the context used to predict the next token, and high-probability patterns in the training data become increasingly attractive attractors that the model can spiral into. Several techniques address this problem by modifying token probabilities based on prior usage.

Key takeaways:

• Repetition penalty divides logits of previously generated tokens by a penalty factor, uniformly reducing their probability regardless of occurrence count. Simple and effective for general use.

• Frequency penalty subtracts from logits proportionally to occurrence count, creating graduated penalties that scale with overuse. Allows natural repetition while strongly penalizing excessive use.

• Presence penalty applies a flat reduction to any token that has appeared, promoting vocabulary diversity. Effective for brainstorming and ensuring broad coverage.

• N-gram blocking deterministically prevents exact phrase repetition by masking tokens that would complete a previously seen n-gram. Provides guaranteed protection but can reduce fluency.

• Order matters: Apply penalties before temperature and sampling truncation. The modified logits then flow through the standard sampling pipeline.

• Context window: Consider whether penalties apply to the full context (including prompt) or only generated tokens. Most implementations penalize tokens anywhere in the sequence, which can affect prompt-relevant words. Some frameworks allow specifying that only the generated portion should be tracked.

• Limitations: All four techniques operate on token identity rather than semantic content, cannot prevent semantic repetition through paraphrase, and require empirical tuning for each deployment context.

The optimal configuration depends heavily on your use case. Start with a moderate repetition penalty ($\theta = 1.1$-$1.2$), observe the output quality, and adjust based on whether you see too much repetition or unnaturally varied vocabulary. For production systems, A/B testing different configurations against human evaluations provides the most reliable guidance. When probabilistic penalties are insufficient, n-gram blocking provides a hard guarantee at the cost of reduced flexibility.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about repetition penalties in language model generation.