Repetition Penalties › Summary

Repetition is a common failure mode in language model generation, emerging from the autoregressive process where patterns reinforce themselves through context accumulation. The feedback loop at the heart of this failure mode is well-understood: each generated token becomes part of the context used to predict the next token, and high-probability patterns in the training data become increasingly attractive attractors that the model can spiral into. Several techniques address this problem by modifying token probabilities based on prior usage.

Key takeaways:

• Repetition penalty divides logits of previously generated tokens by a penalty factor, uniformly reducing their probability regardless of occurrence count. Simple and effective for general use.

• Frequency penalty subtracts from logits proportionally to occurrence count, creating graduated penalties that scale with overuse. Allows natural repetition while strongly penalizing excessive use.

• Presence penalty applies a flat reduction to any token that has appeared, promoting vocabulary diversity. Effective for brainstorming and ensuring broad coverage.

• N-gram blocking deterministically prevents exact phrase repetition by masking tokens that would complete a previously seen n-gram. Provides guaranteed protection but can reduce fluency.

• Order matters: Apply penalties before temperature and sampling truncation. The modified logits then flow through the standard sampling pipeline.

• Context window: Consider whether penalties apply to the full context (including prompt) or only generated tokens. Most implementations penalize tokens anywhere in the sequence, which can affect prompt-relevant words. Some frameworks allow specifying that only the generated portion should be tracked.

• Limitations: All four techniques operate on token identity rather than semantic content, cannot prevent semantic repetition through paraphrase, and require empirical tuning for each deployment context.

The optimal configuration depends heavily on your use case. Start with a moderate repetition penalty ($\theta = 1.1$-$1.2$), observe the output quality, and adjust based on whether you see too much repetition or unnaturally varied vocabulary. For production systems, A/B testing different configurations against human evaluations provides the most reliable guidance. When probabilistic penalties are insufficient, n-gram blocking provides a hard guarantee at the cost of reduced flexibility.