Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Attention Masking › Padding Masks

Michael BrenndoerferUpdated May 27, 202510 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 93Attention Masking › Padding Masks

Reading section 3 of 13 · Padding Masks · 10 minSection 3 of 13 · 10 min

Section 3 of 13 · 10 min read

Attention Masking › Padding MasksOne section · Sections in this chapter3 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Real-world text comes in varying lengths. "Hello" has one token while "The quick brown fox jumps over the lazy dog" has nine. To process multiple sequences efficiently in a batch, we pad shorter sequences to a common length using a special padding token. Think of padding as blank lines at the bottom of a form: they are there so every form has the same number of lines, but they carry no information and should be ignored when reading the content.

The practical consequence of padding is that every batch contains some positions that should not influence any computation. If we naively apply self-attention without masking, those padding positions will accumulate information from real tokens, and real tokens will waste attention capacity attending to padding positions. Neither interaction is meaningful. A token like "[PAD]" has no semantic content, so attending to it pollutes the output representation with noise.

The severity of this problem scales with the degree of padding in the batch. In a batch where most sequences are short but one is very long, most positions across most sequences are padding. In that scenario, a large fraction of every attention matrix would be wasted on padding interactions. The learned representations would be noisy, training would be less efficient, and the model might learn spurious patterns based on the position of padding tokens rather than the content of real tokens.

In[3]:

Code

# Example batch with varying sequence lengths
sequences = [
    ["The", "cat", "sat"],  # Length 3
    ["Hello", "world"],  # Length 2
    ["A", "quick", "brown", "fox"],  # Length 4
]

# Pad to maximum length
max_len = max(len(seq) for seq in sequences)
pad_token = "[PAD]"

padded_sequences = []
for seq in sequences:
    padded = seq + [pad_token] * (max_len - len(seq))
    padded_sequences.append(padded)

Out[4]:

Console

Padded sequences (max length = 4):
  Sequence 0: ['The', 'cat', 'sat', '[PAD]']
  Sequence 1: ['Hello', 'world', '[PAD]', '[PAD]']
  Sequence 2: ['A', 'quick', 'brown', 'fox']

Padding ensures uniform tensor shapes, but it introduces tokens that should be invisible to the attention mechanism. A padding mask marks which positions contain real tokens versus padding. The mask is a binary indicator: each position either belongs to the real content (marked as "keep") or to the padding area (marked as "block").

In[5]:

Code

import numpy as np


def create_padding_mask(sequences, pad_token="[PAD]"):
    """
    Create a padding mask for a batch of sequences.

    Returns a mask where True indicates positions to KEEP (real tokens)
    and False indicates positions to MASK (padding tokens).
    """
    batch_size = len(sequences)
    seq_len = len(sequences[0])

    # True where we have real tokens, False for padding
    mask = np.array(
        [[token != pad_token for token in seq] for seq in sequences]
    )

    return mask


padding_mask = create_padding_mask(padded_sequences)

Out[6]:

Console

Padding mask (True = real token, False = padding):
  Sequence 0: [np.True_, np.True_, np.True_, np.False_]
             ['The', 'cat', 'sat', '[PAD]']
  Sequence 1: [np.True_, np.True_, np.False_, np.False_]
             ['Hello', 'world', '[PAD]', '[PAD]']
  Sequence 2: [np.True_, np.True_, np.True_, np.True_]
             ['A', 'quick', 'brown', 'fox']

The padding mask is a boolean array where True marks real tokens and False marks padding. To apply this mask to attention scores, we need to convert it into an additive mask. This conversion is the bridge between the "which positions are real" question and the "how much should each position contribute" answer.

Converting to Attention MaskLink Copied

We have a boolean padding mask, but attention needs numerical scores. How do we bridge this gap? The answer lies in understanding how softmax behaves with extreme values.

Softmax converts a vector of arbitrary real numbers into a probability distribution. Every element becomes positive, and all elements sum to one. The function achieves this through exponentiation followed by normalization. The key insight is that softmax is sensitive to relative differences between values, not their absolute magnitudes. When computing softmax, each score is exponentiated, then divided by the sum of all exponentials:

\text{softmax}(z)_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

where:

$z_i$ : the $i$ -th element of the input vector, which is the attention score for one query-key pair
$e^{z_i}$ : the exponential of that score, which converts any real number to a positive value
$\sum_j e^{z_j}$ : the sum of exponentials across all positions, which is the normalizing constant ensuring the outputs sum to one

If one score is extremely negative while others are moderate, its exponential becomes vanishingly small. To illustrate, consider applying softmax to a vector with one large negative value:

\text{softmax}([2.0, 1.5, -10000]) \approx [0.62, 0.38, 0.00]

The third element receives essentially zero weight. This happens because $e^{-10000}$ is astronomically small compared to $e^{2.0}$ and $e^{1.5}$ . In practice, $e^{-10000}$ is so close to zero that floating-point arithmetic rounds it to exactly zero, so the masked position truly disappears from the computation.

Why does this formula make sense? Notice that the exponential function is monotone: larger inputs produce larger outputs. By making one input dramatically smaller than all the others, we guarantee its contribution to the normalization sum is negligible, and its share of the normalized output is negligible too. We are essentially creating a value so small that, in the competition of exponentials, it loses completely.

This gives us our masking strategy: add a large negative value to positions we want to block. We typically use $-10^9$ rather than true infinity to avoid numerical edge cases, though both work in practice. True negative infinity can cause NaN values in edge cases where entire rows are masked (since $0/0$ becomes NaN), whereas $-10^9$ keeps the arithmetic well-defined.

In[7]:

Code

# Demonstrate how mask value affects attention weight

base_scores = np.array([2.0, 1.5, 1.0])  # Three positions with similar scores
mask_values = np.linspace(0, -20, 100)

# For each mask value, compute softmax and track the masked position's weight
masked_weights = []
other_weights_0 = []
other_weights_1 = []

for mv in mask_values:
    scores_with_mask = base_scores.copy()
    scores_with_mask[2] = base_scores[2] + mv  # Apply mask to position 2
    exp_scores = np.exp(scores_with_mask - scores_with_mask.max())
    weights = exp_scores / exp_scores.sum()
    masked_weights.append(weights[2])
    other_weights_0.append(weights[0])
    other_weights_1.append(weights[1])

Out[8]:

Visualization

Line plot showing attention weight vs mask value, with masked position weight dropping to zero as mask becomes negative. — Effect of mask value on attention weight distribution. As the mask value becomes more negative, attention to the masked position drops rapidly toward zero while unmasked positions absorb the redistributed attention. By a mask value of -15, the masked position is effectively invisible.

The plot demonstrates why we use large negative values like $-10^9$ . Even a mask value of $-10$ reduces the masked position's weight to near zero, while $-15$ makes it essentially invisible. The unmasked positions automatically absorb the redistributed attention: when one position drops out of the softmax competition, the remaining positions share that "freed" probability mass. The model does not need to relearn anything; the redistribution happens automatically through the normalization in softmax.

In[9]:

Code

def padding_mask_to_attention_mask(padding_mask):
    """
    Convert padding mask to additive attention mask.

    For self-attention, we need a (batch, seq_len, seq_len) mask
    that masks out attention TO padding positions.

    Args:
        padding_mask: shape (batch, seq_len), True for real tokens

    Returns:
        attention_mask: shape (batch, seq_len, seq_len)
        0.0 for allowed attention, -inf for masked positions
    """
    batch_size, seq_len = padding_mask.shape

    # Expand mask to (batch, 1, seq_len) for broadcasting
    # We want to mask attention TO padding positions (keys/values)
    mask_expanded = padding_mask[:, np.newaxis, :]

    # Broadcast to (batch, seq_len, seq_len)
    attention_mask = np.where(
        mask_expanded,
        0.0,  # Real token: no modification
        -1e9,  # Padding: large negative value
    )

    return attention_mask


attention_mask = padding_mask_to_attention_mask(padding_mask)

Out[10]:

Console

Attention mask shape: (3, 1, 4)

Sequence 0 attention mask (length 3, 1 pad token):
[[ 0.e+00  0.e+00  0.e+00 -1.e+09]]

Sequence 1 attention mask (length 2, 2 pad tokens):
[[ 0.e+00  0.e+00 -1.e+09 -1.e+09]]

The attention mask has shape (batch, seq_len, seq_len). For sequence 0, the last column contains large negative values because position 3 is padding. Every query position will receive near-zero attention weight for position 3 after softmax. Notice that we mask along the key dimension, not the query dimension. The reason is that we want to prevent any query from attending to a padding key, but we do not need to prevent a padding position from being a query: in practice the output at padding positions is simply discarded, so it does not matter what the padding query "attends to."

Visualizing Padding Mask EffectsLink Copied

Let's see how padding masks affect attention weights in practice. We'll create random attention scores and compare the distributions with and without masking.

In[11]:

Code

def softmax(x, axis=-1):
    """Numerically stable softmax."""
    exp_x = np.exp(x - x.max(axis=axis, keepdims=True))
    return exp_x / exp_x.sum(axis=axis, keepdims=True)


# Simulate attention scores for sequence 1 (2 real tokens, 2 padding)
seq_len = 4
scores = np.random.randn(seq_len, seq_len)

# Attention without masking
weights_unmasked = softmax(scores)

# Attention with padding mask for sequence 1
mask = attention_mask[1]  # Sequence 1 has 2 padding tokens
weights_masked = softmax(scores + mask)

Out[12]:

Visualization

Heatmap showing attention weights distributed across all 4 positions. — Attention weights without masking. All four positions receive non-trivial attention, including the two padding tokens in columns 2 and 3, which dilute the signal from real content.

Heatmap showing attention weights concentrated in first 2 columns with zeros in last 2 columns. — Attention weights with padding mask applied. Columns 2 and 3 (padding positions) receive exactly zero weight, and the freed probability mass is redistributed entirely to the real tokens in columns 0 and 1.

The contrast is stark. Without masking, attention flows to all positions including padding. The padding tokens absorb 10-30% of the attention weight at each query position. With the mask applied, padding positions receive exactly 0.00 weight, and the remaining attention redistributes entirely to real tokens. This ensures padding never contaminates token representations.

The redistribution effect is particularly important: the two real tokens now share 100% of the attention weight instead of roughly 50-70%. The model has more focused, meaningful signal to work with. Over millions of training examples, this difference accumulates into substantially cleaner representations.