Sinusoidal Position Encoding › The Position Encoding Formula

Before diving into the mathematics, let's consider what properties we want from a position encoding. We need a function that takes a position index and produces a vector, and this function must satisfy several constraints that, taken together, are quite demanding.

The first constraint is uniqueness. Each position must map to a distinct vector. If positions 5 and 17 produce the same encoding, the model cannot distinguish them, and all the information about order is lost at those positions. This sounds simple but rules out many naive approaches: a constant function is not unique, and a linear function is unique but fails the next constraint.

The second constraint is bounded values. The encoding should not grow unboundedly with position. If position 1000 produces values 1000 times larger than position 1, the position signal would overwhelm the semantic content of the embeddings. The model would spend most of its capacity reasoning about raw position magnitude rather than meaning. We need the encoding values to stay within a fixed range regardless of how long the sequence is.

The third constraint is smooth progression. Nearby positions should have similar encodings. Position 50 should be more similar to position 51 than to position 500, giving the model useful gradient information during training and letting it generalize position-aware patterns across neighboring positions. An encoding that jumps chaotically between positions would be hard to learn from.

The fourth constraint is determinism. The same position should always produce the same encoding, without requiring any learned parameters. This is not strictly necessary for correctness (learned embeddings also satisfy the first three constraints), but it means the encoding requires zero parameters and can represent positions never seen during training.

Sinusoidal functions satisfy all these requirements elegantly. Sine and cosine oscillate smoothly between -1 and 1. This ensures bounded values. Different frequencies distinguish positions at different scales. And the encoding is purely deterministic, computed from a fixed formula. The use of multiple frequencies simultaneously is what makes the encoding both unique and smooth at the same time.

The core idea is to assign each position a unique "fingerprint" using waves of different frequencies. Think of how you might describe your location in a building: you could give the floor number (coarse scale), the room number (medium scale), and your position within the room (fine scale). Together, these scales uniquely identify any location. No single scale alone would suffice: the floor number alone could apply to hundreds of rooms, and the position-within-room alone could apply to rooms on every floor.

For position encoding, we use sine and cosine waves at different frequencies to achieve the same multi-scale identification. Each position $pos$ in the sequence receives a $d$-dimensional vector, where consecutive pairs of dimensions use sine and cosine at the same frequency:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right)$$ $$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)$$

where:

• $pos$: the position index in the sequence (0, 1, 2, ...)
• $i$: the dimension index pair (0, 1, 2, ..., $d/2 - 1$)
• $d$: the total embedding dimension
• $PE_{(pos, 2i)}$: the encoding value at position $pos$, even dimension $2i$
• $PE_{(pos, 2i+1)}$: the encoding value at position $pos$, odd dimension $2i+1$
• $10000$: a base constant that controls the frequency range

Notice that for a fixed position $pos$, as we scan across the $d$ dimensions, we compute sine and cosine of decreasing frequencies. The first two dimensions use the fastest oscillation, and the last two dimensions use the slowest. The encoding contains a full spectrum of waves, from very high frequency to very low frequency, stacked into a single vector.

The key to the formula is the denominator $10000^{2i/d}$. This term controls how fast each dimension pair oscillates as position increases. Let's unpack what happens at different dimension indices.

When $i = 0$ (the first dimension pair), the denominator is $10000^0 = 1$, so we compute $\sin(pos)$ and $\cos(pos)$. This oscillates rapidly: moving from position 0 to position 6 covers roughly one full cycle.

When $i = d/2 - 1$ (the last dimension pair), the denominator is approximately $10000^1 = 10000$, so we compute $\sin(pos/10000)$ and $\cos(pos/10000)$. This oscillates extremely slowly: you need 62,832 positions to complete one full cycle.

The exponent $2i/d$ creates a geometric progression of frequencies. As $i$ increases from 0 to $d/2 - 1$, the exponent increases from 0 to approximately 1, and the denominator grows from 1 to 10000. This exponential scaling ensures that each dimension pair captures position information at a different resolution. The spacing between frequencies is not arbitrary: it is chosen so that the wavelengths span a wide enough range to cover practical sequence lengths while keeping the total number of dimensions manageable.

The choice of 10000 as the base constant is also deliberate. A smaller base would cluster the wavelengths too close together. This provides redundant information in nearby dimensions. A larger base would spread them out but might leave some scale ranges uncovered. With 10000 and typical embedding dimensions of 256 or 512, the wavelengths range from about 6 positions up to 63,000 positions, which comfortably covers sequences of any practical length at the time the paper was written.

Each dimension pair uses both sine and cosine at the same frequency. This pairing is not arbitrary; it serves two distinct purposes that are both essential.

The first purpose is unique identification within a cycle. Sine alone cannot distinguish positions that differ by multiples of $2\pi$. At the first dimension's frequency, positions 0 and 6 have very similar sine values, and positions 0 and approximately 6.28 have identical sine values. But the (sine, cosine) pair at any frequency uniquely identifies a phase angle. Geometrically, as position increases, the (sin, cos) pair traces a circle in 2D space, and every point on that circle corresponds to a unique position within one cycle. The cosine value is 90 degrees out of phase with the sine value, so there is never a point where both are ambiguous simultaneously.

The second purpose is enabling relative position computation. The sine/cosine pairing allows relative positions to be computed through rotation matrices, a property we'll explore in detail later in this chapter. This mathematical structure means the model can potentially learn to attend to relative positions using simple linear operations, which is a powerful inductive bias for sequence modeling. This property is so useful that it directly inspired modern architectures: RoPE (Rotary Position Embedding), used in LLaMA and many other models, is essentially a refined version of this rotation insight, applied directly to the query and key vectors inside attention.