Bag of Words › The Loss of Word Order
Language AI HandbookAll chapters Bag of Words › The Loss of Word OrderOne section · Sections in this chapter8 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The most striking limitation of Bag of Words is its complete discard of word sequence information. "The cat chased the dog" and "The dog chased the cat" produce identical vectors, despite having opposite meanings about who is doing the chasing. This is not a minor technical footnote; it is a fundamental consequence of the bag metaphor. Once you pour the words into the bag and shake it, all positional information is irretrievably lost.
In practice, this means that BoW cannot represent a large class of linguistic phenomena that depend on word order. Negation is perhaps the most important: "I love this movie" and "I don't love this movie" produce nearly identical vectors, differing only by the addition of "don't". A BoW classifier trained to recognize positive sentiment might confidently classify the second sentence as positive, because all the positive words are still there. Similarly, comparative statements like "cats are better than dogs" and "dogs are better than cats" become indistinguishable, as do passive and active constructions, questions and statements, and most conditional relationships.
# Demonstrate word order loss
sentences = [
"The cat chased the dog",
"The dog chased the cat",
"Dog the cat the chased", # Nonsense with same words
]
vec = CountVectorizer()
vectors = vec.fit_transform(sentences)Word Order Demonstration: -------------------------------------------------- Vocabulary: ['cat', 'chased', 'dog', 'the'] 'The cat chased the dog' Vector: [1 1 1 2] 'The dog chased the cat' Vector: [1 1 1 2] 'Dog the cat the chased' Vector: [1 1 1 2] Sentence 1 == Sentence 2: True Sentence 1 == Sentence 3: True All three sentences produce IDENTICAL vectors!
This is the fundamental limitation of Bag of Words. It cannot distinguish between:
- Active and passive voice: "John hit Mary" vs "Mary was hit by John"
- Negation scope: "I love this movie" vs "I don't love this movie" have nearly identical vectors
- Questions and statements: "Is this good?" vs "This is good"
- Any semantic difference that depends on word order
