Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Relative Position Encoding › Visualizing Relative Position Effects

Michael BrenndoerferUpdated June 2, 20254 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 99Relative Position Encoding › Visualizing Relative Position Effects

Reading section 8 of 16 · Visualizing Relative Position Effects · 4 minSection 8 of 16 · 4 min

Section 8 of 16 · 4 min read

Relative Position Encoding › Visualizing Relative Position EffectsOne section · Sections in this chapter8 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's visualize how relative position embeddings influence attention. We'll examine the position term $\mathbf{q}_i \cdot \mathbf{a}^K_{j-i}$ separately from the content term. Isolating these two components lets us see how much of the final attention pattern comes from what tokens mean versus where they are.

In[16]:

Code

# Extract position-only attention biases
def compute_position_biases(Q, rel_embed):
    """
    Compute attention score contributions from relative positions only.
    """
    seq_len, d_k = Q.shape
    biases = np.zeros((seq_len, seq_len))

    for i in range(seq_len):
        for j in range(seq_len):
            rel_pos = j - i
            a_K = rel_embed.get_key_embedding(rel_pos)
            biases[i, j] = np.dot(Q[i], a_K) / np.sqrt(d_k)

    return biases


# Compute position biases for our test case
Q_test = X @ rel_attn.W_q
position_biases = compute_position_biases(Q_test, rel_attn.rel_embed)

# Also compute content-only scores for comparison
K_test = X @ rel_attn.W_k
content_scores = (Q_test @ K_test.T) / np.sqrt(d_k)

Out[17]:

Visualization

Heatmap of content-based attention scores between six positions showing irregular patterns based on embedding similarity. — Content scores computed from query-key dot products alone. These irregular patterns reflect purely semantic compatibility between tokens and carry no distance information.

Heatmap of position-based attention biases showing diagonal band structure reflecting relative distances. — Position biases from the relative position embeddings. Each diagonal stripe corresponds to a fixed offset, revealing that tokens at the same relative distance always receive the same positional treatment regardless of their absolute positions.

The content scores show irregular patterns depending on the input embeddings. The position biases show a distinctive diagonal structure: each anti-diagonal corresponds to positions at a fixed relative distance. For example, the main diagonal (offset 0) shows self-attention bias, while diagonals above and below show biases for looking forward and backward. Notice that in the position biases panel, all cells on the same anti-diagonal have identical values. This is because they share the same relative offset and therefore the same position embedding. This structural constraint is what makes relative encoding powerful: every occurrence of the same distance gets the same treatment.

Let's examine how the combined scores compare to content-only attention:

In[18]:

Code

# Compute attention weights with and without position
def softmax_rows(x):
    exp_x = np.exp(x - x.max(axis=1, keepdims=True))
    return exp_x / exp_x.sum(axis=1, keepdims=True)


content_only_weights = softmax_rows(content_scores)
combined_scores = content_scores + position_biases
combined_weights = softmax_rows(combined_scores)

Out[19]:

Visualization

Heatmap of attention weights using only content similarity between positions. — Content-only attention weights based purely on query-key similarity. The spread of weights across positions reflects only semantic compatibility between tokens.

Heatmap of attention weights combining content and relative position, showing modified attention patterns. — Attention weights combining content and relative position biases. The relative position term shifts weights toward or away from certain distances, introducing a structured distance preference on top of the semantic signal.

The position biases subtly shift attention patterns. In this random example, the differences are modest because the embeddings are random. In trained models, relative position embeddings learn linguistically meaningful patterns: verbs attending strongly to positions 1-2 back (typical subject distance), determiners attending 1-2 forward (to their nouns), and so on. The combined plot differs from a sum of the two individual plots because the softmax nonlinearity mixes them: a moderate positive position bias for a nearby token can dramatically amplify its weight if the content score is also high, while a large positive bias for a distant token with low content score may still produce a negligible final weight.