Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Relative Position Encoding › Learned Position Patterns

Michael BrenndoerferUpdated June 2, 20252 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 99Relative Position Encoding › Learned Position Patterns

Reading section 12 of 16 · Learned Position Patterns · 2 minSection 12 of 16 · 2 min

Section 12 of 16 · 2 min read

Relative Position Encoding › Learned Position PatternsOne section · Sections in this chapter12 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To illustrate what relative position embeddings might learn, let's simulate embeddings that encode common linguistic patterns:

In[24]:

Code

# Simulate learned relative position embeddings
# Pattern: slight preference for immediately adjacent positions

d_k_sim = 4
max_dist_sim = 5

# Create embeddings that encode locality preference
np.random.seed(42)
a_K_sim = np.random.randn(2 * max_dist_sim + 1, d_k_sim) * 0.1

# Add locality bias: positions closer to 0 have higher dot product potential
for idx in range(2 * max_dist_sim + 1):
    offset = idx - max_dist_sim  # Convert back to relative position
    locality_factor = np.exp(-0.5 * (offset**2))  # Gaussian decay
    a_K_sim[idx, 0] += locality_factor  # Boost first dimension

# Create a uniform query that will reveal the position pattern
uniform_query = np.ones(d_k_sim) / np.sqrt(d_k_sim)
position_profile = uniform_query @ a_K_sim.T

Out[25]:

Visualization

Line plot showing position score peaking at offset 0 and decaying symmetrically for more distant positions. — Simulated learned relative position profile showing a locality preference. The Gaussian-shaped curve peaks at offset 0 (self-attention) and decays for more distant positions. Real models learn more asymmetric patterns reflecting directional linguistic dependencies, such as verbs preferring to look back for subjects and forward for objects.

This simulated pattern shows a Gaussian-like preference for nearby positions. Real models learn more complex patterns that capture linguistic structure: determiners attending to following nouns, verbs attending to preceding subjects, and so on. A trained model's position profiles are also typically asymmetric. This reflects the fact that English (and many other languages) has predominantly left-to-right dependency structure. Subjects appear before verbs, modifiers often precede their heads, and complementizers introduce subordinate clauses that follow the main verb. These directional biases show up clearly when you visualize position profiles from models trained on real text.