Learned Position Embeddings › Worked Example: Tracing Position Through a Transformer

To make the mechanism concrete, let's trace exactly what happens to position information when a transformer processes a simple sentence. Consider the sentence "The cat sat." with tokens at positions 0, 1, and 2.

At the input layer, each token is converted to its embedding and the corresponding position embedding is added. The token embedding for "cat" encodes something like "feline animal, common noun." The position embedding for position 1 encodes something like "second position in the sequence, often subject or modifier." Their sum is a single vector that carries both pieces of information.

When the attention layer processes this combined representation, it computes attention scores between all pairs of positions. The attention score between position 0 ("The") and position 1 ("cat") depends partly on their position embeddings. If those embeddings are similar (adjacent positions), the attention mechanism can easily learn to attend to nearby tokens. If they are dissimilar (distant positions), the mechanism must work harder to bridge the distance.

The query and key vectors are linear projections of the combined representation, so they carry positional information mixed with token identity information. A query at position 2 ("sat") asking "what is the subject of this verb?" will produce high attention scores for position 1 ("cat") partly because the position embeddings have been trained to make subject-verb relationships detectable by attention. The gradient signals that drive this learning come from the model's attempts to predict which tokens likely follow which patterns, and position is one of the key distinguishing features.

After attention, the position information propagates forward through the feed-forward layers, becoming increasingly mixed with the token's semantic content. By the final layer, the representation at each position has been enriched by context from other positions, with the original position embedding serving as one component of the total positional signal.

This concrete tracing illustrates a key property: position embeddings don't encode position in isolation. They work together with the attention mechanism to enable position-sensitive pattern matching. The embeddings provide raw positional coordinates, and the attention weights learn to use those coordinates to implement meaningful linguistic operations like subject-verb agreement, modifier-head relationships, and sentence boundary detection.