Repetition Penalties › Repetition Penalties

Language models have a peculiar tendency to get stuck in loops. Ask GPT to write a story, and without intervention, you might get output like "The cat sat on the mat. The cat sat on the mat. The cat sat on the mat..." This repetitive behavior emerges from a fundamental property of autoregressive generation: at each step, the model selects tokens that maximize likelihood given the context, and if a pattern worked once, the same pattern often has high probability again.

Think of it like a river carving a channel: once water flows a particular path, the channel deepens and that path becomes increasingly easy to follow. In language models, each generated token strengthens the patterns already in the context, gradually making those exact patterns more attractive at future steps. A model generating a list might produce one item, then another that uses the same structure, then another, then another, until it is trapped forever in a loop it cannot escape on its own. The channel has been carved so deep that the water cannot climb out.

Repetition is not always undesirable. Certain phrases naturally repeat in language: "again and again", "more and more", or the rhythmic repetition in poetry and song lyrics. Technical writing legitimately repeats specialized terms: a machine learning paper must say "gradient descent" many times. Legal documents repeat precise phrasings deliberately. But uncontrolled repetition signals a failure mode where the model has collapsed into a degenerate loop, producing text that no human would write intentionally. The challenge is to suppress pathological repetition while preserving the natural repetition that language requires.

This chapter explores four techniques to prevent such loops while preserving natural language patterns. Each operates at a different level of granularity and with a different mathematical mechanism. Understanding all four gives you a toolkit to match the intervention to the specific failure mode you are encountering.

We'll examine three probabilistic approaches that modify the probability distribution during generation. Repetition penalty scales down the logits of previously used tokens. Frequency penalty applies increasingly strong penalties based on how many times each token has appeared. Presence penalty applies a flat penalty to any token that has appeared at all. We'll also explore n-gram blocking, a deterministic approach that prevents exact phrase repetition. Each technique offers different trade-offs between preventing loops and preserving natural repetition.