Bag of Words › Summary

Bag of Words converts text into numerical vectors by counting word occurrences, ignoring grammar and word order entirely. Despite this simplification, it powers effective text classification, information retrieval, and topic modeling systems, and it established the conceptual vocabulary that all subsequent NLP methods build upon.

Key takeaways:

• Vocabulary construction extracts unique words from a corpus, mapping each to a vector dimension. The decisions made here, what to tokenize, how to normalize, and which words to keep, directly affect model quality.
• Document-term matrices represent documents as rows and vocabulary words as columns, with counts (or binary indicators) as values. The matrix entry $M_{ij}$ is the count of vocabulary word $j$ in document $i$.
• Cosine similarity measures document similarity by computing the normalized dot product between vectors, making it insensitive to differences in document length.
• Vocabulary pruning with min_df and max_df removes uninformative rare and common words. The steep drop in vocabulary size from min_df=1 to min_df=2 reflects the Zipfian nature of language, where a large fraction of words appear only once.
• Sparse matrices (CSR format) efficiently store the mostly-zero document-term matrices, reducing memory by 99%+ for realistic corpora by storing only non-zero values and their positions.
• scikit-learn's CountVectorizer handles tokenization, vocabulary building, and sparse matrix creation in one optimized package, with a clear fit/transform API that enforces vocabulary consistency between training and inference.
• Word order loss is the fundamental limitation: "The cat chased the dog" and "The dog chased the cat" produce identical vectors.

In the next chapters, we'll extend these ideas with n-grams to capture some word sequences, and with TF-IDF weighting to emphasize discriminative terms over common ones.