Repetition Penalties › Why Models Repeat Themselves
Language AI HandbookAll chapters Repetition Penalties › Why Models Repeat ThemselvesOne section · Sections in this chapter2 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before diving into solutions, we should understand why repetition happens. The answer lies in how autoregressive models generate text and how they are trained. This is not a superficial bug but a deep consequence of the architecture and training objective.
During training, language models learn to predict the next token given all previous tokens. They're optimized to assign high probability to tokens that frequently follow specific contexts in the training data. When a particular phrase or structure appears often in training text, the model learns to give it high probability in similar contexts. The model is, in a sense, a compressed summary of what patterns appeared in the training corpus, and repetitive patterns show up strongly in that corpus.
The problem manifests during generation. Suppose the model generates "The results show that" and then produces "the model performs well." Now "The results show that the model performs well" becomes part of the context. If the training data contained similar patterns of presenting multiple results, the phrase "The results show that" might again have high probability, leading to another similar sentence. Each repetition reinforces the pattern in the context, making further repetition even more likely. The model is not doing anything wrong by its own internal logic: it is faithfully predicting what typically follows such contexts. The training data likely contained many documents that followed this structure, so the model learned to associate such contexts with such continuations.
There is also a more subtle contributor: maximum likelihood training creates pressure toward safe, high-probability tokens. During training, the model is penalized for predicting low-probability tokens and rewarded for predicting high-probability ones. This creates a systematic bias toward tokens the model is confident about, and confidence often correlates with familiarity. A token that just appeared is familiar: its embedding has just been activated, similar patterns have just been processed. It is, in a sense, "top of mind" for the model, making it an appealing choice at the next step.
The main point is that repetition is not random noise but a systematic attractor in the probability distribution. Once a phrase is in context, it raises the probability of similar phrases, which when generated raise the probability of even more similar phrases. Left unchecked, this positive feedback loop converges to a fixed point where the model endlessly repeats the same content.
def generate_without_penalty(model, tokenizer, prompt, max_tokens=100):
"""Generate text without any repetition penalty."""
input_ids = tokenizer.encode(prompt, return_tensors="pt")
with torch.no_grad():
output = model.generate(
input_ids,
max_new_tokens=max_tokens,
do_sample=True,
temperature=0.8,
top_p=0.9,
repetition_penalty=1.0, # No penalty
pad_token_id=tokenizer.eos_token_id,
)
return tokenizer.decode(output[0], skip_special_tokens=True)The output may contain repetitive phrases or sentence structures. While sampling strategies like temperature and nucleus sampling add randomness, they don't specifically target repetition. A token that appeared recently still has the same probability as before, so the model can easily select it again. Temperature and sampling truncation change the shape of the distribution but do not specifically push previously used tokens toward lower probability. We need dedicated mechanisms for that.
The Feedback Loop
Repetition often starts subtly and escalates. The model might repeat a word, then a phrase, then entire sentences. This acceleration happens because each repetition adds more of the same pattern to the context, which the model uses to predict the next token. The context becomes increasingly dominated by the repeated content, making continuation of that content more and more likely.
Consider a model generating a list. After producing "First, we need to consider..." it might generate "Second, we need to consider..." and then "Third, we need to consider...". So far, this is reasonable parallel structure. But without intervention, the model might continue with "Fourth, we need to consider..." long after the list should have ended, trapped in a pattern that keeps reinforcing itself. The parallel structure pattern is strong enough in the training data that the model keeps predicting more of it.
The feedback loop has a characteristic shape. Early in generation, repetition probability is roughly at baseline. After a repeated phrase first appears, its probability edges up slightly. The second repetition raises it further. By the third or fourth repetition, the probability can be so high that almost any intervention short of hard blocking is insufficient. This is why the problem is called a "degenerate loop": the model has reached a stable fixed point from which it cannot easily escape through normal sampling variation. Understanding this escalation pattern helps explain why even moderate penalties are effective: they flatten the early growth of repetition probability before the feedback loop has a chance to establish itself.
The feedback loop also helps explain why beam search, a common decoding strategy, is particularly vulnerable to repetition. Beam search maintains multiple high-probability hypotheses and keeps expanding the most likely ones. Since repetition temporarily maximizes local probability, beam search selects into loops very aggressively. This is why n-gram blocking is most commonly associated with beam search in summarization: without hard blocking, beam search almost always collapses into repetitive output.