Position Problem › The Position Problem: Why Self-Attention Is Blind to Word Order

In the previous chapters, we explored self-attention and the query-key-value mechanism in detail. We saw how each token can attend to every other token in a sequence, computing rich contextual representations that capture relationships regardless of distance. A token at position 1 can attend to a token at position 100 just as easily as to its immediate neighbor, which is one of self-attention's most powerful properties. But there is a fundamental limitation hiding in plain sight: self-attention has no notion of order. It treats the input sequence as an unordered bag of tokens, completely ignoring the positions where those tokens appear.

Think of it this way. Imagine you receive a bag of Scrabble tiles spelling out a message. You can see every letter and every pair of letters, but the bag has no concept of left-to-right sequence. You know which letters co-occur with which other letters, but you cannot reconstruct the original order from that information alone. Self-attention faces the same problem. It sees every pair of tokens and computes their compatibility, but the positions those tokens occupy leave no trace in the final representation.

This is not a minor implementation detail: it cuts to the heart of language understanding. Language is not a bag of words. The sentence "Dog bites man" and the sentence "Man bites dog" share exactly the same vocabulary, the same word co-occurrences, and the same pairwise relationships between tokens. Yet they describe opposite events. If a model cannot distinguish word order, it cannot distinguish these sentences. And if it cannot distinguish these sentences, it cannot reason about who is doing what to whom, which is arguably the most basic task in natural language understanding.

The position problem pervades nearly every aspect of language. Subject-verb agreement, pronoun resolution, negation scope, modifier attachment, syntactic parsing, temporal ordering of events, causal relationships between clauses: all of these linguistic phenomena depend critically on where words appear in a sentence, not merely on which words appear. A language model that ignores position is like a reader who sees all the words on a page at once but has no concept of reading left to right. The words are all there, but the structure that gives them meaning is lost.

This chapter explores why position blindness emerges from the mathematics of attention, why word order matters so critically for language understanding across multiple dimensions, and what properties any solution must possess to restore positional awareness to the model. We will set the stage for the various positional encoding schemes that subsequent chapters cover in depth. By the end, you will understand that position matters, why the attention mechanism discards it, and what the design criteria are for getting it back.