Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Repetition Penalties › Presence Penalty

Michael BrenndoerferUpdated August 1, 20259 min read

Language AI HandbookAll chapters

Part 28 of 67GPT Architecture

Chapter 205Repetition Penalties › Presence Penalty

Reading section 5 of 13 · Presence Penalty · 9 minSection 5 of 13 · 9 min

Section 5 of 13 · 9 min read

Repetition Penalties › Presence PenaltyOne section · Sections in this chapter5 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Presence penalty takes an even simpler approach: penalize any token that has appeared, regardless of how many times. This binary penalty treats "appeared once" and "appeared twenty times" identically. At first glance this might seem crude compared to the graduated precision of frequency penalty, but it captures a different goal.

Think of presence penalty as an entry tax rather than a per-use tax. The first time a token appears in the generated text, it pays a fixed fee. Subsequent appearances pay nothing extra. This structure is ideal when your primary concern is vocabulary breadth rather than repetition frequency. You want the model to explore as many different words and concepts as possible, with each new token representing a new idea being introduced. Once a concept is on the table, having it mentioned again is less concerning than having entirely new concepts go unmentioned.

This makes presence penalty particularly valuable for brainstorming tasks, where the goal is to generate as many distinct ideas as possible in a limited number of tokens. It also works well for summarization, where you want to ensure coverage of multiple aspects of the source document rather than repeatedly circling around one or two salient points. In both cases, the question is "have I introduced this yet?" rather than "how many times have I used this?", which maps directly to the binary structure of presence penalty.

Presence Penalty

Presence penalty applies a fixed penalty to the logit of any token that has appeared in the generated text, regardless of frequency. The formula is $z'_i = z_i - \beta \cdot \mathbb{1}[i \in G]$ , where $\beta$ is the penalty strength and $\mathbb{1}[i \in G]$ is an indicator that equals 1 if token $i$ has appeared and 0 otherwise.

Mathematical FormulationLink Copied

We've now seen two approaches: repetition penalty treats all repeated tokens equally (ignoring count), while frequency penalty scales with count. Presence penalty takes the opposite extreme from frequency penalty: it also ignores count, but uses the simpler additive mechanism.

The Binary QuestionLink Copied

Presence penalty asks the simplest possible question about each token: "Have you appeared before, yes or no?" If yes, apply a fixed penalty. If no, leave the logit unchanged. The number of previous occurrences is irrelevant.

To express this mathematically, we use an indicator function, a standard notation for encoding yes/no conditions as 1/0 values:

z'_i = z_i - \beta \cdot \mathbb{1}[i \in G]

where:

$z_i$ : the original logit for token $i$
$z'_i$ : the modified logit after applying the penalty
$\beta$ : the presence penalty coefficient, controlling how strongly any prior appearance is penalized
$\mathbb{1}[i \in G]$ : the indicator function, which equals 1 if token $i$ has appeared (is in set $G$ ), and 0 otherwise
$G$ : the set of tokens that have appeared in the generated sequence

Reading the Indicator FunctionLink Copied

The notation $\mathbb{1}[i \in G]$ might look intimidating, but it simply acts like a switch. The condition inside the brackets, " $i \in G$ " (token $i$ is in set $G$ ), is evaluated as true or false. The indicator function converts this to a number:

If the condition is true (token has appeared): $\mathbb{1}[\text{true}] = 1$
If the condition is false (token has not appeared): $\mathbb{1}[\text{false}] = 0$

Substituting back into the formula:

For a token that has appeared: $z'_i = z_i - \beta \cdot 1 = z_i - \beta$
For a token that hasn't appeared: $z'_i = z_i - \beta \cdot 0 = z_i$

The indicator function elegantly handles both cases in a single equation. In practice, you can implement this without any indicator function machinery: simply iterate over the set of appeared tokens and subtract $\beta$ from each. The indicator notation is useful for mathematical exposition but translates to a simple loop in code.

Why does this formula make sense? Because the logit space is linear in its effect on probability (through the exponentiation in softmax), a fixed subtraction of $\beta$ from a logit translates to dividing the unnormalized probability by $e^\beta$ . So presence penalty is equivalent to multiplying all appeared tokens' unnormalized probabilities by a constant factor $e^{-\beta}$ , which is less than 1 for any positive $\beta$ . This is a clean, uniform discounting of all previously seen tokens.

Presence vs. Frequency: A ComparisonLink Copied

The contrast with frequency penalty is instructive. Frequency penalty applies $\alpha$ for each occurrence, so a token appearing 10 times receives 10 times the penalty of one appearing once. Presence penalty applies $\beta$ exactly once regardless of occurrence count. Whether a token appeared 1 time or 100 times, it receives the same penalty $\beta$ .

This makes presence penalty particularly effective for encouraging vocabulary diversity. Once a word has been used, it faces a fixed "tax" on appearing again. The model is pushed to explore alternatives, to find synonyms, to vary its phrasing. It is a blunt instrument compared to frequency penalty's graduated pressure, but for tasks like brainstorming or generating diverse lists, that bluntness is exactly what is needed. The model cannot "amortize" the penalty by spreading uses over time, the way it can with frequency penalty at low counts. The penalty is immediate and permanent from the first use.

Out[17]:

Visualization

Line plot comparing flat presence penalty curve against linearly increasing frequency penalty curves. — Presence penalty vs frequency penalty as occurrence count increases. Presence penalty (dashed) applies a constant penalty regardless of count, while frequency penalty (solid) grows linearly.

The contrast is stark. Presence penalty (dashed lines) jumps to its full value after the first occurrence and stays flat. Frequency penalty (solid lines) starts at zero and grows steadily. For tokens that appear many times, frequency penalty eventually dominates; for tokens that appear just once or twice, presence penalty applies stronger immediate pressure. If you want a token to be discouraged immediately upon first use, presence penalty is the sharper tool. If you want the penalty to grow with overuse rather than applying immediately, frequency penalty is better suited.

ImplementationLink Copied

In[18]:

Code

def apply_presence_penalty(logits, generated_tokens, beta):
    """
    Apply presence penalty to tokens that have appeared at least once.

    Args:
        logits: Tensor of shape (vocab_size,) containing raw logits
        generated_tokens: Set of token IDs that have appeared
        beta: Presence penalty coefficient

    Returns:
        Modified logits with presence-based penalties applied
    """
    modified_logits = logits.clone()

    for token_id in generated_tokens:
        modified_logits[token_id] = modified_logits[token_id] - beta

    return modified_logits

Notice how much simpler this implementation is compared to frequency penalty. We do not need to maintain a count dictionary, only a set of token IDs that have appeared. For long generation runs with large vocabularies, this can be a meaningful efficiency advantage, particularly when the set of appeared tokens is substantially smaller than the full vocabulary. The set can be maintained with O(1) average insertion and lookup time, making the penalty application efficient even for very long sequences.

Comparing All Three PenaltiesLink Copied

Let's see how the three penalties differ when applied to the same scenario:

In[19]:

Code

# Setup: tokens with varying occurrence counts
logits = torch.tensor([2.0, 2.0, 2.0, 2.0, 2.0, 2.0])
names = ["novel", "data", "the", "model", "is", "good"]

# Counts: novel=0, data=1, the=5, model=3, is=2, good=0
counts = {1: 1, 2: 5, 3: 3, 4: 2}  # data, the, model, is
appeared = {1, 2, 3, 4}  # Set of tokens that appeared

# Apply each penalty type
rep_penalty = 1.5
freq_alpha = 0.3
pres_beta = 1.0

logits_rep = apply_repetition_penalty(logits, appeared, rep_penalty)
logits_freq = apply_frequency_penalty(logits, counts, freq_alpha)
logits_pres = apply_presence_penalty(logits, appeared, pres_beta)

Out[20]:

Console

Comparison of penalty types:
================================================================================
Token     Count   Original   Rep(θ=1.5)  Freq(α=0.3)    Pres(β=1)
================================================================================
novel         0      0.167        0.247        0.255        0.288
data          1      0.167        0.127        0.189        0.106
the           5      0.167        0.127        0.057        0.106
model         3      0.167        0.127        0.104        0.106
is            2      0.167        0.127        0.140        0.106
good          0      0.167        0.247        0.255        0.288
================================================================================

Out[21]:

Visualization

Grouped bar chart comparing original, repetition, frequency, and presence penalty probabilities across six tokens. — Comparison of three penalty types on the same token distribution. Frequency penalty scales with occurrence count, heavily penalizing 'the' (5 occurrences) while lightly penalizing 'data' (1 occurrence).

The visualization reveals each penalty's character:

Repetition penalty (blue) uniformly reduces all appeared tokens, regardless of count. "data" (×1) and "the" (×5) are penalized identically, which is the key limitation of this approach.
Frequency penalty (red) creates graduated reductions: "the" (×5) drops dramatically while "data" (×1) drops minimally. This is the most fine-grained of the three approaches.
Presence penalty (green) applies equal reduction to all appeared tokens, similar to repetition penalty but using additive rather than multiplicative adjustment. The difference from repetition penalty is visible in tokens with very negative logits, but for the equal-logit scenario here, the two approaches look similar.