Position Problem › Summary

Self-attention is blind to position because its computation depends only on content, not on where tokens appear in the sequence. This permutation equivariance is a fundamental property of the attention mechanism, arising from the fact that position indices serve only as bookkeeping labels and never appear as values in any arithmetic operation. It is not a bug in any particular implementation; it is built into the mathematical structure of dot-product attention.

Key takeaways from this chapter:

• Permutation equivariance: Self-attention produces the same outputs (reordered) regardless of input order. Shuffling the input shuffles the output identically, because attention weights depend only on query-key compatibility, not on position indices. The equation $f(\pi(\mathbf{X})) = \pi(f(\mathbf{X}))$ captures this precisely.
• Language requires order: Grammatical roles, negation scope, modifier attachment, temporal relationships, and semantic composition all depend on word position. "Dog bites man" and "man bites dog" are opposite events expressed by the same vocabulary, and only position distinguishes them.
• Requirements for positional information: Any solution must provide unique position identification, bounded values, consistency across sequence lengths, accessibility of relative positions, generalization beyond training, and compatibility with attention.
• Encoding vs. embedding: Positional encoding uses fixed mathematical formulas (like sinusoids) that generalize to any position. Positional embedding uses learned lookup tables that adapt to training data but have a fixed maximum length. Both have significant trade-offs.
• Absolute vs. relative: Absolute encoding assigns fixed vectors to positions, requiring the model to infer relative distances. Relative encoding directly captures distances between positions, supporting better generalization but requiring architectural modifications.
• Injection mechanism: The standard approach adds positional vectors to token embeddings before computing queries and keys, combining semantic and positional information in a shared representation. Alternative approaches modify attention scores directly.
• Modern trends: The field has moved from absolute learned embeddings (BERT, GPT) toward relative encoding schemes (T5, RoPE). This reflects the linguistic insight that relative distances matter more than absolute positions for most language tasks.

In the next chapter, we will examine the sinusoidal positional encoding introduced in the original Transformer paper. This elegant mathematical construction uses sine and cosine functions at different frequencies to create unique position vectors that support relative position computation through simple linear operations. You will see exactly how the requirements identified in this chapter shaped the design of that solution.