Repetition Penalties › Worked Example: Tracing a Penalty Step by Step

To solidify understanding, let's trace through a concrete numerical example that shows exactly how frequency penalty modifies the generation decision at one step. We'll use a toy vocabulary of 5 tokens and walk through every calculation.

Suppose we are generating text and have just produced the sequence: "the cat sat on the mat and the cat". The current context window contains these 8 tokens. We track their counts:

• "the": appeared 3 times
• "cat": appeared 2 times
• "sat": appeared 1 time
• "on": appeared 1 time
• "mat": appeared 1 time
• "and": appeared 1 time

The model produces the following raw logits for the next token (we use a tiny vocabulary for clarity):

Without any penalty, softmax would strongly favor "the" and "cat". Let's apply frequency penalty with $\alpha = 0.5$.

Step 1: Compute modified logits. For each token, subtract $\alpha \times c_i$:

• "the": $3.2 - 0.5 \times 3 = 3.2 - 1.5 = 1.7$
• "cat": $2.8 - 0.5 \times 2 = 2.8 - 1.0 = 1.8$
• "sat": $2.1 - 0.5 \times 1 = 2.1 - 0.5 = 1.6$
• "again": $1.5 - 0.5 \times 0 = 1.5$ (no penalty: first use)
• "purred": $1.0 - 0.5 \times 0 = 1.0$ (no penalty: first use)

Step 2: Apply softmax to modified logits. We compute $e^{z'_i}$ for each:

• "the": $e^{1.7} \approx 5.47$
• "cat": $e^{1.8} \approx 6.05$
• "sat": $e^{1.6} \approx 4.95$
• "again": $e^{1.5} \approx 4.48$
• "purred": $e^{1.0} \approx 2.72$

Sum: $5.47 + 6.05 + 4.95 + 4.48 + 2.72 = 23.67$

Probabilities: "the" = 0.231, "cat" = 0.256, "sat" = 0.209, "again" = 0.189, "purred" = 0.115

Step 3: Compare to unpenalized probabilities. With the original logits, the top choices were overwhelmingly "the" and "cat". After frequency penalty, "cat" is still favored but by a much smaller margin, and "again" and "purred" have become viable competitors. The penalty has not eliminated the model's preference but has substantially leveled the playing field, giving novel tokens a real chance at selection.

The key insight from this example is that frequency penalty is most effective when the penalized tokens have modest logit advantages over alternatives. If "the" had a logit of 10.0 rather than 3.2, the same $\alpha = 0.5$ with count 3 would reduce it to 8.5, which would still completely dominate. This is why very confident model predictions can resist even strong penalties, and why n-gram blocking is sometimes necessary as a fallback guarantee.