Sinusoidal Position Encoding › Sinusoidal Position Encoding

The transformer's self-attention mechanism is permutation invariant: it produces the same output regardless of token order. Feed a transformer the sentence "The dog bit the man" and "The man bit the dog" and, without any additional signal, it will treat them identically. This is a fundamental problem, because word order is one of the primary carriers of meaning in language. A mechanism that cannot distinguish "not guilty" from "guilty" based on word order is useless for any serious language task.

To fix this, the original "Attention Is All You Need" paper introduced sinusoidal position encoding, an elegant solution that encodes each position as a unique pattern of sine and cosine waves at different frequencies. Rather than learning position representations from data, the encoding is computed from a fixed mathematical formula, requiring zero extra parameters. Every position in a sequence receives a distinct vector that the model can use to reason about order and distance.

This approach is one of the most instructive designs in the entire transformer architecture, because it solves a concrete engineering problem, uniquely identifying positions in a way that supports both absolute and relative reasoning, using pure mathematics rather than learned tables. Understanding sinusoidal encoding deeply will also give you the conceptual foundation for understanding modern alternatives like RoPE and ALiBi, which build directly on the same ideas.

Before we examine the formula, consider why the problem is harder than it looks. You might wonder: why append the position index to the token embedding? Why multiply by the position or add a one-hot vector? Each idea fails at least one property we need. Appending the position number means that position 1000 carries a value 1000 times larger than position 1, which would dominate the semantic embedding entirely. A one-hot vector of length equal to the maximum sequence length would require enormous memory and could not generalize to lengths not seen during training. The sinusoidal approach avoids these failure modes, and the rest of this chapter explains how.