Bag of Words: Document-Term Matrices

How do you teach a computer to understand text? Computers are fundamentally numeric machines. Their calculations operate on binary data and floating-point numbers. Text, on the other hand, organizes symbols into words that form sentences and larger passages. Before any learning can happen, we need to bridge this gap. We need a way to translate the fuzzy, contextual world of human language into the crisp, numerical world of mathematics.

The first step is deceptively simple: count words. The Bag of Words (BoW) model converts documents into numerical vectors by tallying how often each word appears. This representation ignores grammar, word order, and context entirely. It treats a document as nothing more than a collection of words tossed into a bag, hence the name. Think of it like taking a book, ripping out every page, cutting out every word, throwing all those scraps into a literal bag, and then counting how many times each unique word appears. You lose the story, but you preserve the vocabulary, and that turns out to be surprisingly useful.

Despite its simplicity, Bag of Words powered text classification, spam detection, and information retrieval for decades before more sophisticated methods emerged. It remains a surprisingly effective baseline for many NLP tasks, and in some domains it still outperforms more complex models when labeled data is scarce. Understanding BoW matters for historical reasons and because it introduces the core concepts that persist throughout all of modern NLP: vocabulary construction, document-term matrices, and sparse vector representations. Every technique covered in this book, from TF-IDF to word embeddings to transformer attention mechanisms, builds on ideas that first appear in their clearest form in the Bag of Words model.

This chapter walks you through building a Bag of Words representation from scratch. You'll learn how to construct vocabularies, create document-term matrices, handle the explosion of dimensionality with sparse matrices, and understand when this simple approach works and when it fails. Along the way, we'll examine the mathematical foundations of document similarity, the engineering practicalities of vocabulary pruning, and the philosophical limitations that motivated everything that came after BoW.

The Bag of Words model rests on a single, powerful observation: documents that discuss the same topic tend to use the same words. A sports article will mention "goal", "player", and "score". A cooking article will mention "ingredient", "recipe", and "bake". Even if the sentences are completely different, the vocabulary overlaps strongly within a topic. This means that word counts alone carry a surprising amount of information about what a document is about.

The key insight is that we can represent any document as a point in a high-dimensional vector space, where each dimension corresponds to a word in the vocabulary, and the coordinate along that dimension tells us how many times that word appears. Two documents that are about the same topic will be nearby points in this space, because they use similar words with similar frequencies. Two documents about completely different topics will be far apart. This geometric intuition makes it possible to apply distance metrics, classification algorithms, and clustering techniques directly to text, using the same mathematical toolkit we use for any other numerical data.

Consider three short documents:

1. "The cat sat on the mat"
2. "The dog sat on the log"
3. "The cat and the dog"

To represent these numerically, we first build a vocabulary: a list of all unique words across all documents. Then we count how many times each vocabulary word appears in each document. Notice that Documents 1 and 2 share a similar structure ("The [animal] sat on the [object]"), while Document 3 has a different structure. We would expect Documents 1 and 2 to be more similar to each other than either is to Document 3, and that is exactly what the vector representation will capture.

Let's implement this step by step:

Look at the matrix structure. Document 1 has two occurrences of "the" (the cat... the mat), reflected in the count of 2. Document 3 shares vocabulary with both other documents, which we can see from the overlapping non-zero entries. The zeros tell us which words are absent. In practice, this means the matrix will be dominated by zeros, because any given document uses only a small fraction of the total vocabulary. We will return to this sparsity problem shortly.

Once documents become vectors, we can measure their similarity using cosine similarity. This metric computes the cosine of the angle between two vectors, effectively measuring how similar their directions are in high-dimensional space, regardless of their magnitudes. Think of it as measuring not how long the vectors are, but which direction they point. Two documents that use the same words in similar proportions will point in roughly the same direction, even if one is much longer than the other.

The motivation for cosine similarity over simpler measures like Euclidean distance is important to understand. Euclidean distance is sensitive to document length: a long document that repeats many of the same words as a short document would appear far away in Euclidean space, simply because its word counts are larger. Cosine similarity normalizes by vector magnitude, so a 500-word article and a 50-word summary on the same topic will score close to 1.0 similarity even though their raw count vectors look very different in scale.

Given two document vectors $\mathbf{a}$ and $\mathbf{b}$, cosine similarity computes the normalized dot product between them. The dot product captures how many words the two documents share (and in what quantities), while the normalization by vector magnitudes accounts for document length differences:

where:

• $\mathbf{a} \cdot \mathbf{b}$: the dot product of vectors $\mathbf{a}$ and $\mathbf{b}$, computed as $\sum_{i=1}^{n} a_i b_i$
• $\|\mathbf{a}\|$: the Euclidean norm (magnitude) of vector $\mathbf{a}$, computed as $\sqrt{\sum_{i=1}^{n} a_i^2}$
• $n$: the vocabulary size (number of dimensions in each vector)
• $a_i, b_i$: the word counts at position $i$ in vectors $\mathbf{a}$ and $\mathbf{b}$ respectively

The result ranges from 0 (completely different, no shared vocabulary) to 1 (identical word distributions). By normalizing by vector magnitudes, cosine similarity ensures that documents with similar word proportions are considered similar even if one document is much longer than the other.

Documents 1 and 2 are most similar because they share the phrase structure "The [animal] sat on the [object]". Document 3, with its different structure, shows lower similarity to both. This demonstrates how BoW captures topical similarity through shared vocabulary, even though it ignores word order. Notice that the similarity matrix is symmetric: the similarity from Document 1 to Document 2 equals the similarity from Document 2 to Document 1, which makes intuitive sense since cosine similarity is commutative.

Building a vocabulary seems straightforward at first glance, but real-world text introduces significant complications. Raw text varies in capitalization and punctuation and may contain numbers or contractions. These variations make vocabulary construction a surprisingly subtle problem. How do you handle punctuation? Should "cat" and "Cat" be the same word? What about "run", "running", and "ran"? What about rare words that appear only once across millions of documents? What about extremely common words like "the" that appear everywhere but say nothing distinctive about a document's content?

These are not academic questions. The answers dramatically affect the size of your vocabulary, the density of your vectors, and ultimately the quality of the models you build on top. A naive vocabulary that includes every unique character sequence as a separate word quickly grows to tens of millions of entries, most of which appear only once and contribute only noise. A carefully pruned vocabulary that retains only the most informative terms produces tighter, more discriminative representations that generalize better to new documents.

The vocabulary construction process involves several sequential decisions. First, you tokenize the text, splitting it into individual word units. Then you normalize those tokens, typically by lowercasing and stripping punctuation. Finally, you filter the resulting vocabulary using frequency thresholds to remove both extremely rare and extremely common terms. Each of these steps involves tradeoffs that depend on your specific task and domain.

A corpus is a collection of documents. The vocabulary is the set of unique terms extracted from this corpus. Let's work with a slightly more realistic example:

The vocabulary is significantly smaller than the total token count because many words repeat across documents. This compression, from raw tokens to unique vocabulary terms, is a key characteristic of text data. In practice, this ratio becomes even more pronounced as the corpus grows: adding more documents adds fewer and fewer new vocabulary words because the most common words appear in the very first documents.

Before finalizing the vocabulary, examining word frequencies helps identify potential issues. The distribution of word frequencies in natural language follows Zipf's Law, which states that the frequency of a word is roughly inversely proportional to its rank in the frequency table. The most common word appears about twice as often as the second most common word, three times as often as the third, and so on. This mathematical regularity directly affects vocabulary construction.

Zipf's Law means that any natural language corpus contains a small number of very high-frequency words (function words like "the", "is", "a") and a very large number of low-frequency words (specific nouns, technical terms, proper names). The high-frequency words consume disproportionate space in your vectors but carry little discriminative information, since they appear in nearly every document regardless of topic. The low-frequency words are often highly informative when they do appear, but appear so rarely that they may be noise rather than signal.

The word "learning" appears 5 times, "machine" appears 4 times, but many words appear only once. This pattern, a few high-frequency words and many rare words, follows Zipf's Law and is characteristic of natural language. In practice, this means that the decision of where to draw the frequency cutoff for your vocabulary has a large impact on both vocabulary size and model quality.

Raw vocabularies from large corpora can contain millions of unique words. Many of these are noise: typos, rare technical terms, proper nouns that appear in only one document, or words so common that they appear everywhere. Vocabulary pruning removes uninformative terms to reduce dimensionality and improve model performance.

The case for pruning is both computational and statistical. On the computational side, every word in your vocabulary adds a dimension to every document vector, and operations over high-dimensional sparse vectors are expensive. On the statistical side, rare words that appear in only one or two documents provide almost no information for a classifier: the model cannot learn reliable patterns from a single or double observation. And common words that appear in nearly every document also provide no discriminative power, because a word that is equally likely in all classes tells you nothing about which class a document belongs to.

In practice, this means that the most informative words for text classification tend to be those in the middle of the frequency distribution: not so rare that they appear in only a handful of documents, and not so common that they appear everywhere. Vocabulary pruning attempts to focus the representation on exactly this informative middle zone.

The document frequency of a word $w$, denoted $\text{df}(w)$, counts how many distinct documents in the corpus contain that word. Notice that document frequency is different from term frequency: if a word appears 10 times in a single document, its term frequency is 10 but its document frequency is still just 1. Document frequency tells us how widespread a word is across the corpus, which is a better measure of a word's general informativeness than raw frequency.

Formally, document frequency is the size of the set of documents containing the word:

where:

• $D$: the corpus (collection of all documents)
• $d$: an individual document in the corpus
• $w \in d$: indicates that word $w$ appears in document $d$
• $|\cdot|$: the count of elements in the set

Words that appear in very few documents provide little discriminative power and may represent noise. The min_df parameter sets a threshold: words must appear in at least this many documents (or this fraction of documents) to be included.

Words like "artificial", "industries", and "networks" appear in only one document. Removing them reduces our vocabulary while keeping words that appear across multiple documents. Notice that this pruning is conservative: we are only removing words that appear in a single document, which is the lowest possible threshold. In practice with large corpora, you might set min_df=5 or min_df=10 to be more aggressive.

At the other extreme, words that appear in almost every document provide no discriminative power. The word "the" might appear in 95% of documents, making it useless for distinguishing between them. If "the" appears in sports articles, cooking articles, and science articles in roughly equal proportions, a classifier that learns to look for "the" gains nothing. The max_df parameter sets an upper threshold to filter out these ubiquitous words.

This is closely related to the concept of stop words, a predefined list of common words (like "the", "is", "and", "of") that are routinely excluded from text representations. The max_df approach is more principled than fixed stop word lists because it adapts to the corpus. In a specialized medical corpus, a word like "patient" might appear in 80% of documents and should be filtered out, even though it would not appear on any general-purpose stop word list.

In our small corpus, "learning" appears in all 5 documents (100%), exceeding our 80% threshold. In real applications, you might filter out words appearing in more than 90% of documents to remove uninformative terms like "the", "is", and "a". The combined effect of min_df and max_df pruning tends to keep precisely the words that carry the most discriminative information about document content.

How aggressively should you prune? The answer depends on your corpus size and your task. For very large corpora with millions of documents, even a min_df of 5 or 10 removes a substantial fraction of the vocabulary while keeping all words with meaningful statistical support. For small corpora like our example, even min_df=2 has a significant effect.

Let's visualize how vocabulary size changes as we increase the min_df threshold:

The steep drop from min_df=1 to min_df=2 is typical. In real corpora, a large fraction of words appear only once (called hapax legomena, from the Greek for "said only once"). Removing these rare words often improves model performance by reducing noise without losing much signal. This is because hapax legomena are disproportionately likely to be misspellings, unusual proper names, or highly specialized jargon that appears in only one domain-specific document. Keeping them inflates the vocabulary and creates dimensions that are always zero except for that one document.

So far, we've counted word occurrences. But sometimes presence matters more than frequency. In a binary representation, each cell contains 1 if the word appears in the document and 0 otherwise, regardless of how many times it appears.

The choice between count and binary representation depends on what the frequency signal means for your task. Counts carry information about emphasis: a document that uses the word "excellent" ten times is making a stronger statement than one that uses it once. But counts can also mislead. A very long document will have higher counts for almost every word simply because it's longer, not because those words are more important. This length bias is one reason why TF-IDF weighting (covered in the next chapter) was developed as an improvement over raw counts.

Binary representations have a natural interpretation: they answer the question "is this topic present in the document?" rather than "how intensely is this topic discussed?" For some classification tasks, the distinction matters. Classifying whether a document is about sports or cooking depends mostly on presence of domain vocabulary, and binary features work just as well as counts. For other tasks, like distinguishing a rave review from a mild positive review, the frequency of strong positive words matters, and counts carry more signal.

Notice that "learning" appears twice in this document. The count representation records 2, while the binary representation records 1. Which is better? It depends on the task. For document classification, binary representations often work as well as counts. For tasks where word frequency carries meaning (like authorship attribution), counts are more informative. In practice, the best approach is to treat this as a hyperparameter and evaluate both on held-out data.

Real-world vocabularies contain tens of thousands of words, yet most documents use only a small fraction. A news article with 500 words might touch only 200 unique vocabulary terms out of 50,000. If you store this as a dense array with one float per vocabulary term, you need 50,000 floating-point numbers to represent each document, of which 49,800 are exactly zero. That's 49,800 zeros sitting in memory, contributing nothing to any computation.

This is the sparsity problem, and it gets dramatically worse as vocabulary size grows. The mathematics are unforgiving: if your vocabulary has $n$ words and each document uses an average of $k$ unique words, then the sparsity is approximately $1 - k/n$. For a vocabulary of 100,000 words and documents that use 300 unique words, that's 99.7% zeros. Storing and operating on dense matrices of this kind is not just inefficient, it's often completely impractical.

Sparsity measures the proportion of zero elements in a matrix. For a document-term matrix $\mathbf{M}$ with $m$ documents (rows) and $n$ vocabulary words (columns), the total number of elements in a dense representation is $m \times n$. The sparsity tells us what fraction of those elements are zero:

where:

• $m$: the number of documents (rows in the matrix)
• $n$: the vocabulary size (columns in the matrix)
• $\text{nnz}$: the number of non-zero elements in the matrix

A sparsity of 0.99 means 99% of the matrix elements are zeros. In NLP, high sparsity is the norm because each document uses only a tiny fraction of the total vocabulary.

Let's quantify the sparsity in a typical document-term matrix:

Even in this small example, most of the matrix consists of zeros. Each document only uses a fraction of the vocabulary. In real applications with vocabularies of 100,000+ words and millions of documents, sparsity typically exceeds 99%. Storing a dense matrix would require terabytes of memory for mostly zeros.

The Compressed Sparse Row (CSR) format stores only non-zero values along with their column indices and row boundaries. This is the standard format for document-term matrices because NLP operations typically process one document (row) at a time.

Think of CSR as storing three parallel arrays. The data array holds the actual non-zero values. The indices array holds the column position (vocabulary index) of each non-zero value. The indptr array holds boundary markers that tell you where each row's values begin and end in the other two arrays. To reconstruct any row of the original matrix, you look up the row's boundary in indptr, then read the corresponding slice of data and indices. This design makes row access very fast, which is exactly what you need when processing documents one at a time.

The indptr array is the key to CSR. To find the non-zero values in row $i$, look at positions indptr[i] through indptr[i+1] in both data and indices. This makes row slicing extremely efficient. In a dense matrix, accessing a row requires touching all $n$ columns even if 99% are zero. In CSR format, you touch only the non-zero elements, making operations on sparse matrices scale with the actual content density rather than the theoretical maximum.

The memory advantage of sparse matrices grows dramatically with vocabulary size:

While understanding the internals is valuable, in practice you'll use scikit-learn's CountVectorizer. It handles tokenization, vocabulary building, and sparse matrix creation in a single, optimized package. Under the hood, CountVectorizer implements all the same steps we've built from scratch: tokenizing text, building a vocabulary, counting occurrences, and storing the result as a CSR sparse matrix. The difference is that it does all of this efficiently, with built-in support for vocabulary pruning parameters and other preprocessing options.

One important practical detail is the fit/transform distinction. You fit the vectorizer on your training data to build the vocabulary. You then transform new documents using that same fixed vocabulary. This separation preserves the vector dimensions used during training: when you deploy a model in production, you cannot modify the vocabulary based on new documents you see at inference time. Any word in a new document that wasn't in the training vocabulary is simply ignored.

CountVectorizer offers extensive customization through its constructor parameters. Understanding what each parameter does allows you to tune the vocabulary construction process for your specific domain and dataset. The right parameter choices can dramatically improve downstream model performance by keeping informative words and discarding noise.

Each parameter setting produces different results. The binary setting keeps the same vocabulary but changes counts to presence indicators. Setting min_df=2 dramatically reduces vocabulary by removing rare words. The ngram_range=(1, 2) setting includes both unigrams and bigrams, capturing two-word phrases like "machine learning" and "deep learning". This dramatically increases vocabulary size but can capture meaningful phrases.

The key parameters for CountVectorizer are:

• lowercase (default: True): Convert all text to lowercase before tokenizing. Set to False if case carries meaning (for example, proper nouns or acronyms).
• min_df: Minimum document frequency threshold. If an integer, the word must appear in at least this many documents. If a float between 0.0 and 1.0, represents a proportion of documents. Use min_df=2 or higher to remove rare words and typos.
• max_df: Maximum document frequency threshold. Words appearing in more than this fraction of documents are excluded. Use max_df=0.9 to remove extremely common words that provide no discriminative power.
• binary (default: False): If True, all non-zero counts are set to 1. Use binary representation when word presence matters more than frequency.
• ngram_range (default: (1, 1)): Tuple specifying the range of n-gram sizes to include. (1, 2) includes unigrams and bigrams, capturing phrases like "machine learning". Higher values dramatically increase vocabulary size.
• stop_words: Either 'english' for built-in stop word list, or a custom list of words to exclude. Removes common words like "the", "is", "and" that typically add noise.
• max_features: Limit vocabulary to the top N most frequent terms. Useful for controlling dimensionality in very large corpora.

The most striking limitation of Bag of Words is its complete discard of word sequence information. "The cat chased the dog" and "The dog chased the cat" produce identical vectors, despite having opposite meanings about who is doing the chasing. This is not a minor technical footnote; it is a fundamental consequence of the bag metaphor. Once you pour the words into the bag and shake it, all positional information is irretrievably lost.

In practice, this means that BoW cannot represent a large class of linguistic phenomena that depend on word order. Negation is perhaps the most important: "I love this movie" and "I don't love this movie" produce nearly identical vectors, differing only by the addition of "don't". A BoW classifier trained to recognize positive sentiment might confidently classify the second sentence as positive, because all the positive words are still there. Similarly, comparative statements like "cats are better than dogs" and "dogs are better than cats" become indistinguishable, as do passive and active constructions, questions and statements, and most conditional relationships.

To make the BoW model concrete, let's trace through a complete spam detection pipeline from raw text to a classification decision. Email spam detection was one of the first real-world applications where BoW excelled, and the reason is instructive: spam emails use a distinctive vocabulary (words like "free", "winner", "urgent", "prize") that differs reliably from legitimate email. You don't need to understand the grammatical structure of "CONGRATULATIONS! You have won a FREE prize!" to know it's likely spam. The presence of those words is sufficient.

Imagine you are building a spam filter for a small email service. You have 100 labeled emails: 50 spam and 50 legitimate. The process unfolds in three stages. In the first stage, preprocessing, you tokenize each email, lowercase all words, and strip punctuation. You build a vocabulary from all the words that appear in the training set. In the second stage, vectorization, you transform each email into a count vector using that vocabulary. In the third stage, classification, you train a model (such as Naive Bayes) on the count vectors and their labels.

Even this trivial example shows BoW capturing sentiment through word presence. Words like "brilliant", "fantastic", and "terrible" carry strong sentiment signals regardless of context. Notice that the model correctly classifies both test reviews despite having very small training data. The key insight is that sentiment-bearing words cluster reliably: positive reviews use a different vocabulary than negative ones, and that vocabulary-level signal is sufficient for classification on simple examples.

In practice, this pattern extends to many classification tasks. Topic classification (sports vs. politics vs. technology) works well with BoW because different topics use distinctly different vocabularies. Language detection (is this English or French?) works extremely well because the vocabularies have almost no overlap. Even author identification has been done with BoW: different authors have characteristic word preferences that persist across their writings.

Despite its limitations, Bag of Words remains useful for many tasks. Understanding when it works helps you make the right architectural choice for your problem.

The tasks where BoW excels share a common property: the relevant signal is captured by vocabulary rather than syntax. When you need to classify a document by topic, the presence or absence of domain-specific words is highly predictive. A document containing "electrode", "voltage", and "circuit" is almost certainly about electrical engineering regardless of how those words are arranged. BoW can exploit that signal without any knowledge of grammar or syntax.

For document classification, categorizing news articles, spam detection, or sentiment analysis on long texts, word presence often matters more than order. A movie review containing "terrible", "boring", and "waste" is likely negative regardless of how those words are arranged. The signal becomes more stable as the document contains more words, which is why BoW tends to work better on longer documents than on short phrases or sentences.

Information retrieval is another natural home for BoW. Search engines match query terms against document terms. The classic TF-IDF weighting scheme (covered in the next chapter) builds directly on the Bag of Words foundation, using word counts as the raw material for computing how important each word is to each document. Topic modeling algorithms like Latent Dirichlet Allocation (LDA) assume documents are mixtures of topics, each characterized by word distributions. The bag-of-words assumption is baked directly into the generative model.

Perhaps most importantly, BoW provides an essential baseline for any text classification task. Before deploying complex neural networks, a BoW model provides a sanity check. If a simple model achieves 90% accuracy, you know the task is learnable from word frequencies alone, and you should ask whether the additional complexity of a neural model is justified. If BoW achieves only 60%, you know that word order or semantic relationships matter for this task, which guides you toward more sophisticated approaches.

Bag of Words has fundamental limitations that motivated the development of more sophisticated representations. Understanding these limitations precisely is important: they reveal what BoW cannot do and why, and that "why" directly motivates every subsequent advancement in NLP.

The first limitation is the complete loss of word order, which we have already examined in detail. The second is the absence of semantic knowledge: in the BoW framework, "good" and "excellent" are treated as completely unrelated words, even though they are near-synonyms. The vocabulary is treated as a set of arbitrary symbols with no relationship to one another. This means that a model trained on documents containing "excellent" has no idea how to handle "outstanding" if that word wasn't in the training vocabulary. Semantic relationships between words, the fact that some words mean similar things, that some words are opposites, that some words appear in related contexts, are entirely invisible to BoW.

The third limitation is vocabulary explosion when you try to extend BoW to capture some phrase structure. Adding n-grams helps capture some sequential patterns, but the vocabulary grows rapidly. Moving from unigrams to bigrams can multiply vocabulary size by a factor of 10 to 100, and most of those bigrams will be extremely rare. Moving to trigrams is even worse. The curse of dimensionality sets in: your vectors grow enormous while your data stays the same size, making it increasingly difficult for models to learn reliable patterns.

The fourth limitation concerns sparsity in the context of neural networks. High-dimensional sparse vectors are inefficient for neural networks, which prefer dense, lower-dimensional inputs. A neural network that receives a sparse input vector of 50,000 dimensions, of which 49,700 are zero, is wasting most of its computation processing those zeros. The dense, lower-dimensional word embeddings that replaced BoW representations for deep learning were motivated directly by this inefficiency.

Finally, out-of-vocabulary words present a fundamental challenge. Words not seen during training have no representation. A model trained on formal news text may fail entirely on social media text full of slang or abbreviations, including unconventional spellings. The vocabulary is a closed system: everything outside it is invisible.

These limitations drove the development of better representations. TF-IDF improved on raw counts by weighting words by their rarity. Word embeddings (Word2Vec, GloVe) created dense, semantic representations that capture word relationships. Contextual embeddings (ELMo, BERT) gave words different representations depending on their context, finally solving the word order problem. Yet Bag of Words remains the conceptual starting point for all of these. Understanding document-term matrices, vocabulary construction, and sparse representations supports understanding why each subsequent technique was invented and what problem it solved.

When working with Bag of Words representations, CountVectorizer from scikit-learn is the primary tool. Knowing its parameters and their effects allows you to adapt the vocabulary construction process to your specific domain and scale.

The most important consideration is the min_df and max_df balance. Setting min_df too low keeps noisy rare words. Setting min_df too high removes informative but moderately rare terms. In practice, start with min_df=2 on small corpora and min_df=5 on larger ones, then tune based on downstream performance. Similarly, max_df=0.95 is a reasonable default that removes very common terms while preserving domain-specific high-frequency words.

The ngram_range parameter deserves careful consideration. Bigrams can capture important phrases like "machine learning", "not good", and "highly recommend" that unigrams miss entirely. However, they also introduce a huge number of uninformative combinations. A good practice is to start with unigrams, measure performance, and then try ngram_range=(1, 2) to see if bigrams help. If vocabulary size becomes unmanageable, combine bigrams with a stricter min_df or a max_features cap.

Bag of Words converts text into numerical vectors by counting word occurrences, ignoring grammar and word order entirely. Despite this simplification, it powers effective text classification, information retrieval, and topic modeling systems, and it established the conceptual vocabulary that all subsequent NLP methods build upon.

Key takeaways:

• Vocabulary construction extracts unique words from a corpus, mapping each to a vector dimension. The decisions made here, what to tokenize, how to normalize, and which words to keep, directly affect model quality.
• Document-term matrices represent documents as rows and vocabulary words as columns, with counts (or binary indicators) as values. The matrix entry $M_{ij}$ is the count of vocabulary word $j$ in document $i$.
• Cosine similarity measures document similarity by computing the normalized dot product between vectors, making it insensitive to differences in document length.
• Vocabulary pruning with min_df and max_df removes uninformative rare and common words. The steep drop in vocabulary size from min_df=1 to min_df=2 reflects the Zipfian nature of language, where a large fraction of words appear only once.
• Sparse matrices (CSR format) efficiently store the mostly-zero document-term matrices, reducing memory by 99%+ for realistic corpora by storing only non-zero values and their positions.
• scikit-learn's CountVectorizer handles tokenization, vocabulary building, and sparse matrix creation in one optimized package, with a clear fit/transform API that enforces vocabulary consistency between training and inference.
• Word order loss is the fundamental limitation: "The cat chased the dog" and "The dog chased the cat" produce identical vectors.

In the next chapters, we'll extend these ideas with n-grams to capture some word sequences, and with TF-IDF weighting to emphasize discriminative terms over common ones.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about Bag of Words representations.