Bag of Words › Bag of Words

How do you teach a computer to understand text? Computers are fundamentally numeric machines. Their calculations operate on binary data and floating-point numbers. Text, on the other hand, organizes symbols into words that form sentences and larger passages. Before any learning can happen, we need to bridge this gap. We need a way to translate the fuzzy, contextual world of human language into the crisp, numerical world of mathematics.

The first step is deceptively simple: count words. The Bag of Words (BoW) model converts documents into numerical vectors by tallying how often each word appears. This representation ignores grammar, word order, and context entirely. It treats a document as nothing more than a collection of words tossed into a bag, hence the name. Think of it like taking a book, ripping out every page, cutting out every word, throwing all those scraps into a literal bag, and then counting how many times each unique word appears. You lose the story, but you preserve the vocabulary, and that turns out to be surprisingly useful.

Despite its simplicity, Bag of Words powered text classification, spam detection, and information retrieval for decades before more sophisticated methods emerged. It remains a surprisingly effective baseline for many NLP tasks, and in some domains it still outperforms more complex models when labeled data is scarce. Understanding BoW matters for historical reasons and because it introduces the core concepts that persist throughout all of modern NLP: vocabulary construction, document-term matrices, and sparse vector representations. Every technique covered in this book, from TF-IDF to word embeddings to transformer attention mechanisms, builds on ideas that first appear in their clearest form in the Bag of Words model.

This chapter walks you through building a Bag of Words representation from scratch. You'll learn how to construct vocabularies, create document-term matrices, handle the explosion of dimensionality with sparse matrices, and understand when this simple approach works and when it fails. Along the way, we'll examine the mathematical foundations of document similarity, the engineering practicalities of vocabulary pruning, and the philosophical limitations that motivated everything that came after BoW.