Position Problem › Why Position Matters for Language
Language AI HandbookAll chapters Position Problem › Why Position Matters for LanguageOne section · Sections in this chapter3 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
This position blindness might seem like a minor technical detail, but it is catastrophic for language understanding. Human language is deeply structured around word order, and this structure operates at multiple levels simultaneously. Let's examine why position is essential across several dimensions of linguistic meaning.
Grammatical Roles and Argument Structure
In English and many other languages, word order is the primary mechanism for encoding grammatical roles. Who is the agent (the one doing) and who is the patient (the one being done to) depends critically on position relative to the verb.
Consider these sentences:
- "The dog chased the cat."
- "The cat chased the dog."
The same words appear in both sentences, but they play opposite grammatical roles. In the first sentence, "dog" is the subject (the chaser) and "cat" is the object (the chased). In the second, these roles reverse entirely. Pure self-attention, seeing only that "dog," "chased," and "cat" co-occur, cannot distinguish which is the agent and which is the patient. It knows these three words appear together. It does not know who is biting whom.
This matters enormously for any downstream task. Sentiment analysis, information extraction, question answering, and reading comprehension all require knowing who does what to whom. A model that cannot distinguish subject from object cannot extract the event structure of a sentence. And event structure, it turns out, is most of what sentences express.
The technical term for this positional encoding of grammatical roles is argument structure. Verbs take arguments, and those arguments have specific semantic roles called thematic roles or theta roles. These include agent and patient. Other examples include beneficiary, instrument, or location. In English, the mapping from syntactic position to thematic role is largely positional. The entity before the verb tends to be the agent; the entity after the verb tends to be the patient. Without positional information, the model has no way to infer these roles.
Negation Scope
Word order determines what negation applies to, which can completely reverse the meaning of a sentence:
- "I did not say he stole the money." (I said nothing; the accusation is false or came from someone else)
- "I did say he did not steal the money." (I spoke up; I claimed his innocence)
The positioning of "not" relative to "say" and "steal" completely changes the assertion. Moving a single word maps an accusation of silence into a defense of character. A bag-of-words model sees "not," "say," and "steal" together in both sentences and cannot determine which action is negated.
This extends to other scope-sensitive phenomena. Quantifiers and modal verbs interact with scope in ways that depend on linear order; adverbs and conditionals do too. "Every student read some book" has a different meaning from "Some book was read by every student," and this meaning difference is entirely carried by word order.
Modifier Attachment and Structural Ambiguity
Consider the well-known ambiguity in:
- "I saw the man with the telescope."
Does "with the telescope" modify "saw" (I used a telescope to see) or "man" (the man was carrying a telescope)? Both interpretations are syntactically and semantically plausible. The resolution depends on which positional interpretation the reader adopts. More precisely, it depends on which element in the sentence the prepositional phrase is attached to structurally, and that attachment is partially determined by proximity and order.
A second classic example:
- "The horse raced past the barn fell."
This garden-path sentence initially appears to have "raced past the barn" as the main clause, but the final word "fell" reveals a different structure: "The horse [that was] raced past the barn fell." Readers process this sentence positionally, building a parse tree as they read left to right. Self-attention without positional information cannot capture this left-to-right progressive parsing that human language processing depends on.
Temporal and Causal Ordering
Sequence often implies temporal or causal relationships:
- "She got her degree, then found a job."
- "She found a job, then got her degree."
The order of clauses suggests different life trajectories. Both sentences contain the same events, but their relative positions encode temporal precedence. In narrative text, the order in which events are mentioned generally corresponds to their order in time (the "narrative sequence" principle). A model that cannot track positional order cannot extract the timeline of a story.
Causality also frequently depends on sequential position. "The bridge collapsed because the engineers cut corners" has a different causal structure from "The engineers cut corners because the bridge collapsed," even though both sentences contain the same propositions. Without position, the model cannot determine which event caused which.
Semantic Composition and Compound Formation
Languages compose meaning hierarchically, and order often determines the composition structure:
- "Hot dog" (a type of food, very different from a warm canine)
- "Dog hot" (not standard English, but in languages with post-nominal adjectives, this would mean "a dog that is warm")
English compounds typically have the modifier before the head noun. Without positional information, "hot" and "dog" are just two tokens that appear together, with no way to determine which modifies which, or even whether this is a compound or a two-word phrase.
At the phrasal level, the same words in different orders can produce entirely different semantic compositions:
- "Only she spoke to him." (no one else spoke to him)
- "She only spoke to him." (she spoke but did nothing else)
- "She spoke only to him." (she excluded everyone else as a listener)
The adverb "only" is a classic focus particle that must be positioned immediately before the element it focuses on. Moving it changes the scope of exclusion and therefore the meaning of the sentence. Self-attention without positional information cannot recover which element "only" is focusing on.
# Simulate the attention pattern for "dog bites man" vs "man bites dog"
# We'll use random embeddings but give them distinct identities
np.random.seed(123)
# Create distinct embeddings for each word
word_embeddings = {
"dog": np.random.randn(embed_dim),
"bites": np.random.randn(embed_dim),
"man": np.random.randn(embed_dim),
}
# Construct the two sequences
sentence1 = np.stack(
[word_embeddings["dog"], word_embeddings["bites"], word_embeddings["man"]]
)
sentence2 = np.stack(
[word_embeddings["man"], word_embeddings["bites"], word_embeddings["dog"]]
)
# Compute attention weights for both
_, weights1 = self_attention(sentence1, W_Q, W_K, W_V)
_, weights2 = self_attention(sentence2, W_Q, W_K, W_V)
--- Key Observation --- Attention from 'dog' to 'bites': 0.3439 (sentence 1, dog at position 0) Attention from 'dog' to 'bites': 0.3439 (sentence 2, dog at position 2) These are identical because attention only depends on content, not position!
The heatmaps reveal the core problem vividly. Despite "dog bites man" and "man bites dog" having opposite meanings (opposite agent-patient roles), the attention weight between "dog" and "bites" is identical in both sentences. Self-attention cannot tell that in one sentence the dog is the biter, while in the other the dog is the bitten. The pairwise content relationships are the same; only the positions differ. The attention mechanism computes rich contextual representations, but all of that context is semantically based, with no positional signal anywhere in the computation.
A Worked Example: Positional Ambiguity at Scale
To make the problem even more concrete, consider a longer passage:
"The committee voted to ban the organization, and the organization appealed the decision."
A model with no positional encoding sees: {committee, voted, ban, organization, organization, appealed, decision}. It cannot tell that the first "organization" is the object of "ban" and the second is the subject of "appealed." In a bag-of-words view, these two occurrences of "organization" are identical tokens. Only their positions differentiate their grammatical roles and their referential relationship to the surrounding verbs.
Now consider coreference resolution, which is the task of identifying which noun phrases refer to the same entity. In this sentence, does "organization" in the second clause refer to the same organization as in the first? Almost certainly yes, but arriving at that conclusion requires tracking that "the organization" appears first as a patient (being banned) and then as an agent (doing the appealing). Positional information is essential for this tracking.
This example is not exotic. Most sentences of any complexity require positional reasoning. The longer and more syntactically complex a sentence becomes, the more critical it is for the model to track where each token appears relative to the governing verbs and the surrounding clause structure and connectives.