Position Problem › A Formal Look at What Breaks Without Position

It is worth being precise about the consequences of permutation equivariance for specific NLP tasks. Different tasks suffer from position blindness in different ways.

Classification tasks are affected but sometimes less severely. If a task asks "does this review express positive or negative sentiment?", the answer often depends more on the presence of positive/negative words than on their order. "Terrible, not at all good" and "good, not at all terrible" have similar bag-of-words representations but different meanings. However, for most documents, sentiment is carried reasonably well by vocabulary, and classification models can sometimes compensate. The failure mode is subtle and appears when negation or ordering changes sentiment.

Sequence labeling tasks are severely affected. Named entity recognition, part-of-speech tagging, and syntactic parsing all require assigning labels to specific positions in the sequence. If the model cannot determine that "bank" is at position 3 and modifies "river" at position 4, it cannot correctly determine that "bank" is a noun functioning as part of a compound rather than as a financial institution. Positional relationships among adjacent tokens are essential for sequence labeling.

Generation tasks are catastrophically affected. Language modeling requires predicting the next token given the previous ones, and causal masking means only earlier positions are visible. Without positional encoding, the model cannot determine which tokens came earlier and which came later, making causal prediction undefined. The model cannot enforce that the output at position $t$ conditions only on positions $1, \ldots, t-1$ without knowing what those position indices mean.

Question answering and reading comprehension sit somewhere in between. Finding the answer span in a passage requires understanding that a specific sequence of tokens at a specific location in the passage is the answer. This is heavily positional. But recognizing that the passage contains the relevant information is often more content-based.

The breadth of tasks affected makes positional encoding not an optional enhancement but a fundamental requirement for any transformer-based system doing real natural language processing.