Learned Position Embeddings › How Position Embeddings Learn
Language AI HandbookAll chapters Learned Position Embeddings › How Position Embeddings LearnOne section · Sections in this chapter4 / 19 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Position embeddings learn through the same backpropagation process as all other parameters. When the model makes a prediction, gradients flow backward through the network, and the position embeddings receive updates that push them toward configurations that reduce the loss. The position embedding table is just a matrix of floats, and each float gets its own gradient computed via the chain rule.
The key question is: what signal drives these updates? Consider a language model predicting the next word. The prediction depends on the current token and its context. Position information affects what constitutes good context: a verb near the beginning of a sentence plays a different role than a verb near the end. When the model consistently makes better predictions by recognizing these positional patterns, the gradients will reinforce the position embeddings that support those patterns. Over many training examples, the embeddings learn to encode whichever aspects of position matter for reducing loss.
But what patterns do position embeddings learn? Research has revealed several consistent findings that appear across different models and architectures.
Position embeddings typically learn to encode absolute position information directly. Nearby positions tend to have similar embeddings, creating a smooth gradient across the sequence. This makes intuitive sense: positions 5 and 6 are more similar in terms of their role in the sequence than positions 5 and 50. A token at position 6 can attend to a token at position 5 with roughly the same relationship as any two adjacent tokens, while a token at position 5 and a token at position 50 are separated by a much wider context gap.
The key insight is that the model does not need to be told that nearby positions should have similar embeddings. This property emerges automatically because the training signal rewards it. When the model uses position embeddings to distinguish nearby versus distant context, it naturally learns that fine-grained positional distinctions (position 5 versus 6) look different from coarse distinctions (position 5 versus 50), and the embeddings reflect this.
Let's simulate what trained position embeddings might look like by creating a toy example with smoothly varying patterns:
def create_trained_position_embeddings(max_len, embed_dim, seed=42):
"""
Create position embeddings that simulate trained patterns.
Real trained embeddings show smooth variation across positions.
We simulate this with a combination of:
- Low-frequency sinusoids (capturing global position)
- Medium-frequency patterns (capturing local structure)
- Some learned noise (capturing task-specific patterns)
"""
np.random.seed(seed)
embeddings = np.zeros((max_len, embed_dim))
positions = np.arange(max_len)
for dim in range(embed_dim):
# Mix of frequencies, simulating what training discovers
freq = 0.1 + 0.5 * (dim / embed_dim) # Lower dims = lower freq
phase = np.random.rand() * 2 * np.pi
# Smooth sinusoidal component
embeddings[:, dim] = 0.5 * np.sin(freq * positions + phase)
# Add some learned variation
embeddings[:, dim] += (
0.2 * np.random.randn(max_len) * np.exp(-0.01 * positions)
)
# Normalize to have reasonable magnitude
embeddings = embeddings / np.sqrt(embed_dim)
return embeddings
trained_embeddings = create_trained_position_embeddings(128, 64)
The visualization shows the characteristic structure of learned position embeddings. Different dimensions capture different aspects of position: some vary slowly across the entire sequence (low-frequency components), while others vary more rapidly (high-frequency components). This multi-scale representation allows the model to detect both global position ("near the start versus near the end") and local structure ("two positions apart"). Notice the similarity to sinusoidal encodings, but with irregular phase offsets and amplitude variations that reflect task-specific patterns rather than a fixed formula.
In practice, the learned embeddings from models like GPT-2 look remarkably similar to what our simulation shows: smooth variations across positions, with different dimensions operating at different frequencies. This convergence suggests that the training objective itself, minimizing next-word prediction loss over large corpora, naturally induces a frequency-based positional structure even without it being designed in.