Position Problem › What We Need from Positional Information

Understanding why position matters helps us identify what properties any positional encoding scheme must have. The ideal solution must satisfy several requirements, and each requirement rules out simple approaches you might first consider.

Each position in a sequence needs a distinct representation. If positions 3 and 7 have identical positional signals, the model cannot distinguish tokens at those positions based on their locations. This requirement seems obvious, but it rules out the simplest possible approaches. Using a constant vector (the same positional signal everywhere) provides no information. Using a random vector assigned at initialization but not varied across positions gives each position a signal, but that signal is arbitrary and meaningless to the model.

The uniqueness requirement means the positional representation must vary systematically with position. The question is how.

Whatever numerical representation we use for position, it must have bounded, well-behaved values throughout. If position 1 is encoded as 1, position 1000 as 1000, and position 1 million as 1 million, the scale differences will overwhelm the semantic content of token embeddings. Neural networks are sensitive to the magnitudes of their inputs. Unbounded positional representations would create training instabilities and force the model to learn very different weight scales to handle early versus late positions in a sequence.

The key insight is that the positional representation should live in the same numerical space as the token embeddings it is being combined with. Token embeddings are typically distributed with mean near zero and standard deviation near one, thanks to embedding initialization schemes and layer normalization. Positional representations should match this scale.

The encoding for position 5 should be the same whether it appears in a 10-token sequence or a 10,000-token sequence. If positional representations change based on sequence length, the model must learn different positional patterns for every possible length it might encounter. This would make it impossible to transfer knowledge from short sequences to long ones, and vice versa.

This requirement rules out a common naive approach: normalizing the position by the sequence length. Position $p$ in a sequence of length $n$ might be represented as $p/n$, giving values between 0 and 1. This is bounded, which is good. But it violates consistency: position 5 in a 10-token sequence gets value 0.5, while position 5 in a 100-token sequence gets value 0.05. The model learns that "0.5" means "halfway through," but the absolute meaning of position 5 changes with sequence length.

Many linguistic relationships depend on relative rather than absolute position. A verb typically appears near its subject and object, regardless of where they fall in the sentence. An adjective typically immediately precedes the noun it modifies. A relative clause immediately follows the noun it modifies. These are all relative positional relationships. The model needs to compute "token A is 3 positions before token B," rather than only "token A is at position 7 and token B is at position 10."

Ideally, the positional encoding should make it easy to determine that position 7 is "2 steps after" position 5, rather than only showing that both are somewhere in the sequence. This is a stronger requirement: each position needs a unique signal, and the relationship between signals needs to encode relative distance in a way the model can exploit.

Models trained on sequences of length 512 may encounter sequences of length 1024 at inference time. The positional encoding should degrade gracefully (or not at all) when extrapolating to positions not seen during training. This requirement is especially important for applications where input length is unpredictable, such as processing documents of arbitrary length.

Fixed mathematical functions (like sinusoids) can extrapolate naturally, because the formula applies to any position, including ones never seen during training. Learned embeddings, by contrast, have no entry in their lookup table for positions beyond the training maximum. Handling this gracefully requires interpolation, extrapolation heuristics, or other architectural choices.

The positional information must integrate with the attention mechanism in a way that allows position to influence attention patterns. If a query cannot distinguish nearby keys from distant keys, positional information is not flowing into the core computation. This requirement means we cannot just append position indices somewhere; the position information must be in a form that the dot-product attention can use.

The standard approach is to combine positional representations with token embeddings before computing QKV representations. This way, when the model computes dot products between queries and keys, those dot products depend on position as well as content.