Bag of Words › Limitations and Impact

Bag of Words has fundamental limitations that motivated the development of more sophisticated representations. Understanding these limitations precisely is important: they reveal what BoW cannot do and why, and that "why" directly motivates every subsequent advancement in NLP.

The first limitation is the complete loss of word order, which we have already examined in detail. The second is the absence of semantic knowledge: in the BoW framework, "good" and "excellent" are treated as completely unrelated words, even though they are near-synonyms. The vocabulary is treated as a set of arbitrary symbols with no relationship to one another. This means that a model trained on documents containing "excellent" has no idea how to handle "outstanding" if that word wasn't in the training vocabulary. Semantic relationships between words, the fact that some words mean similar things, that some words are opposites, that some words appear in related contexts, are entirely invisible to BoW.

The third limitation is vocabulary explosion when you try to extend BoW to capture some phrase structure. Adding n-grams helps capture some sequential patterns, but the vocabulary grows rapidly. Moving from unigrams to bigrams can multiply vocabulary size by a factor of 10 to 100, and most of those bigrams will be extremely rare. Moving to trigrams is even worse. The curse of dimensionality sets in: your vectors grow enormous while your data stays the same size, making it increasingly difficult for models to learn reliable patterns.

The fourth limitation concerns sparsity in the context of neural networks. High-dimensional sparse vectors are inefficient for neural networks, which prefer dense, lower-dimensional inputs. A neural network that receives a sparse input vector of 50,000 dimensions, of which 49,700 are zero, is wasting most of its computation processing those zeros. The dense, lower-dimensional word embeddings that replaced BoW representations for deep learning were motivated directly by this inefficiency.

Finally, out-of-vocabulary words present a fundamental challenge. Words not seen during training have no representation. A model trained on formal news text may fail entirely on social media text full of slang or abbreviations, including unconventional spellings. The vocabulary is a closed system: everything outside it is invisible.

These limitations drove the development of better representations. TF-IDF improved on raw counts by weighting words by their rarity. Word embeddings (Word2Vec, GloVe) created dense, semantic representations that capture word relationships. Contextual embeddings (ELMo, BERT) gave words different representations depending on their context, finally solving the word order problem. Yet Bag of Words remains the conceptual starting point for all of these. Understanding document-term matrices, vocabulary construction, and sparse representations supports understanding why each subsequent technique was invented and what problem it solved.