Repetition Penalties › Frequency Penalty
Language AI HandbookAll chapters Repetition Penalties › Frequency PenaltyOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
While repetition penalty treats all repeated tokens identically, frequency penalty scales with occurrence count. A token used once receives a small penalty; a token used ten times receives ten times that penalty. This graduated approach allows some repetition while strongly discouraging excessive use of any single token.
Think of frequency penalty as a progressive tax on word usage. The first time you use a word, you pay a small surcharge. The second time, you pay twice that surcharge. By the tenth use, the cumulative surcharge is substantial enough to make the word expensive to use again. Function words that legitimately appear throughout a document incur some cost, but content words that are being pathologically overused incur enormous cumulative cost. The graduated nature of the penalty naturally adapts to the difference between normal and pathological repetition.
This approach is particularly well suited to long-form generation tasks where you want the model to explore a topic broadly rather than drilling into one narrow aspect. When writing a blog post, you want to cover multiple subtopics and use diverse vocabulary. Frequency penalty naturally pushes the model in this direction by making it progressively more costly to return to the same words and phrases that have already appeared.
Frequency penalty subtracts a value from each token's logit proportional to how many times that token has appeared in the generated text. The formula is , where is the penalty strength and is the number of times token has been generated.
Mathematical Formulation
Repetition penalty has a limitation: it treats a token that appeared once the same as one that appeared fifty times. Both receive the same penalty. But intuitively, we might want to allow occasional repetition (the word "the" naturally appears multiple times in most texts) while strongly discouraging tokens that have become overused. This calls for a penalty that accumulates with each occurrence.
From Binary to Graduated Penalties
Frequency penalty introduces a simple but powerful shift: instead of asking "has this token appeared?", we ask "how many times has this token appeared?" The penalty then scales proportionally with the answer.
The mechanism is additive rather than multiplicative. For each occurrence of a token, we subtract a fixed amount from its logit. Let denote the number of times token has appeared in the generated sequence. The frequency penalty modifies logits as:
where:
- : the original logit for token
- : the modified logit after applying the penalty
- : the frequency penalty coefficient, typically in the range , controlling how strongly each occurrence is penalized
- : the count of token in the generated sequence (0 if the token has not appeared)
The formula subtracts once for each time the token has appeared. If a token appeared three times and , we subtract from its logit. Note that if , the formula applies no penalty at all, since subtracting zero has no effect. New tokens are left untouched.
Why does this formula make sense? Each appearance of a token is a "vote" that the model has already expressed its preference for that token. The frequency penalty accumulates these votes into a debt that the token must overcome to be chosen again. The more it has been chosen in the past, the heavier the debt, and the harder it must compete against fresher alternatives. This creates a natural tendency toward exploration: once the model has heavily used a token, it becomes progressively less attractive, and the model naturally turns to other options.
Why Additive Works
Unlike repetition penalty, frequency penalty doesn't need to handle positive and negative logits differently. Subtraction always reduces a number, regardless of its sign. Subtracting from a positive logit makes it smaller (or even negative). Subtracting from a negative logit makes it more negative. Both operations reduce the logit and thus reduce the probability after softmax.
This simplicity is a feature. The formula is easy to implement, easy to understand, and produces predictable behavior: each occurrence adds the same "cost" to using that token again. The mathematical mechanism is also more transparent: we know exactly how much the logit has been reduced by looking at the count, whereas the multiplicative mechanism of repetition penalty interacts with the original logit value in a more complex way.
Linear Scaling in Action
The linearity of frequency penalty creates a distinctive pattern. Early occurrences of a token receive mild penalties, allowing natural repetition of common words. But as a token accumulates, the penalty grows relentlessly. Consider :
| Occurrences | Penalty Applied |
|---|---|
| 0 | 0.0 |
| 1 | 0.5 |
| 2 | 1.0 |
| 5 | 2.5 |
| 10 | 5.0 |
By the tenth occurrence, the token's logit has been reduced by 5.0, a substantial penalty that makes it far less competitive against tokens that haven't been overused. This graduated pressure naturally prevents any single token from dominating the output without harsh early constraints. Notice that a token appearing twice is only twice as penalized as one appearing once, which is quite lenient: the real impact accumulates at higher counts where pathological repetition is more likely to be occurring.
Implementation
def apply_frequency_penalty(logits, token_counts, alpha):
"""
Apply frequency penalty based on token occurrence counts.
Args:
logits: Tensor of shape (vocab_size,) containing raw logits
token_counts: Dict mapping token IDs to their occurrence counts
alpha: Frequency penalty coefficient
Returns:
Modified logits with frequency-based penalties applied
"""
modified_logits = logits.clone()
for token_id, count in token_counts.items():
modified_logits[token_id] = modified_logits[token_id] - alpha * count
return modified_logitsLet's see how frequency penalty differs from repetition penalty:
# Create a scenario where tokens have different occurrence counts
example_logits = torch.tensor([2.0, 2.0, 2.0, 2.0, 2.0])
token_names = ["data", "model", "the", "learning", "result"]
# Token counts: "the" appeared 5 times, "model" appeared 3 times, "data" once
token_counts = {0: 1, 1: 3, 2: 5} # data: 1, model: 3, the: 5
alpha = 0.5 # Frequency penalty coefficient
modified_logits = apply_frequency_penalty(example_logits, token_counts, alpha)Frequency penalty effect (α = 0.5): ----------------------------------------------------------------- Token Count Logit Modified P(orig) P(mod) ----------------------------------------------------------------- data 1 2.00 1.50 0.200 0.208 model 3 2.00 0.50 0.200 0.077 the 5 2.00 -0.50 0.200 0.028 learning 0 2.00 2.00 0.200 0.343 result 0 2.00 2.00 0.200 0.343 -----------------------------------------------------------------
The key difference is apparent: tokens with higher counts receive proportionally stronger penalties. "The", which appeared 5 times, has its logit reduced by 2.5 (5 × 0.5), while "data", which appeared once, only loses 0.5 from its logit. This graduated penalty is particularly effective for suppressing overused words while allowing reasonable repetition of less common terms. Notice that "learning" and "result", which have not appeared at all, receive no penalty and maintain their original probability relative to each other.
Visualizing Frequency Penalty
