Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Bag of Words › The Core Idea

Michael BrenndoerferUpdated March 24, 20259 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 6Bag of Words › The Core Idea

Reading section 2 of 14 · The Core Idea · 9 minSection 2 of 14 · 9 min

Section 2 of 14 · 9 min read

Bag of Words › The Core IdeaOne section · Sections in this chapter2 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The Bag of Words model rests on a single, powerful observation: documents that discuss the same topic tend to use the same words. A sports article will mention "goal", "player", and "score". A cooking article will mention "ingredient", "recipe", and "bake". Even if the sentences are completely different, the vocabulary overlaps strongly within a topic. This means that word counts alone carry a surprising amount of information about what a document is about.

The key insight is that we can represent any document as a point in a high-dimensional vector space, where each dimension corresponds to a word in the vocabulary, and the coordinate along that dimension tells us how many times that word appears. Two documents that are about the same topic will be nearby points in this space, because they use similar words with similar frequencies. Two documents about completely different topics will be far apart. This geometric intuition makes it possible to apply distance metrics, classification algorithms, and clustering techniques directly to text, using the same mathematical toolkit we use for any other numerical data.

Consider three short documents:

"The cat sat on the mat"
"The dog sat on the log"
"The cat and the dog"

To represent these numerically, we first build a vocabulary: a list of all unique words across all documents. Then we count how many times each vocabulary word appears in each document. Notice that Documents 1 and 2 share a similar structure ("The [animal] sat on the [object]"), while Document 3 has a different structure. We would expect Documents 1 and 2 to be more similar to each other than either is to Document 3, and that is exactly what the vector representation will capture.

Bag of Words

The Bag of Words model represents text as an unordered collection of words, disregarding grammar and word order but keeping track of word frequency. Each document becomes a vector where each dimension corresponds to a vocabulary word, and the value indicates how often that word appears.

Let's implement this step by step:

In[3]:

Code

# Our corpus of documents
documents = [
    "The cat sat on the mat",
    "The dog sat on the log",
    "The cat and the dog",
]

# Step 1: Tokenize and lowercase
tokenized_docs = [doc.lower().split() for doc in documents]

# Step 2: Build vocabulary from all documents
all_words = []
for doc in tokenized_docs:
    all_words.extend(doc)

vocabulary = sorted(set(all_words))
word_to_idx = {word: idx for idx, word in enumerate(vocabulary)}

Out[4]:

Console

Tokenized documents:
  Doc 1: ['the', 'cat', 'sat', 'on', 'the', 'mat']
  Doc 2: ['the', 'dog', 'sat', 'on', 'the', 'log']
  Doc 3: ['the', 'cat', 'and', 'the', 'dog']

Vocabulary (8 words):
  ['and', 'cat', 'dog', 'log', 'mat', 'on', 'sat', 'the']

Word to index mapping:
  'and' → 0
  'cat' → 1
  'dog' → 2
  'log' → 3
  'mat' → 4
  'on' → 5
  'sat' → 6
  'the' → 7

The vocabulary contains all unique words found across the documents. Each word maps to a unique integer index that will become a dimension in our vector representation. Notice that "the" appears multiple times in our documents but only once in the vocabulary, because the vocabulary stores unique words only. This compression, from raw tokens to unique vocabulary terms, is one of the defining features of the BoW representation.

In[5]:

Code

import numpy as np


# Step 3: Create document vectors by counting word occurrences
def document_to_vector(doc_tokens, word_to_idx):
    """Convert a tokenized document to a count vector."""
    vector = np.zeros(len(word_to_idx))
    for token in doc_tokens:
        if token in word_to_idx:
            vector[word_to_idx[token]] += 1
    return vector


# Create the document-term matrix
doc_term_matrix = np.array(
    [document_to_vector(doc, word_to_idx) for doc in tokenized_docs]
)

Out[6]:

Console

Document-Term Matrix:
------------------------------------------------------------
Doc    and    cat    dog    log    mat     on    sat    the
------------------------------------------------------------
  1      0      1      0      0      1      1      1      2
  2      0      0      1      1      0      1      1      2
  3      1      1      1      0      0      0      0      2

Each row represents a document, and each column represents a word from our vocabulary. Formally, we can express the document-term matrix as $\mathbf{M} \in \mathbb{R}^{m \times n}$ where element $M_{ij}$ represents the count of vocabulary word $j$ in document $i$ . Here, $m$ is the number of documents and $n$ is the vocabulary size.

Out[7]:

Visualization

Heatmap showing document-term matrix with documents as rows and vocabulary words as columns, cell colors indicating word counts. — Document-term matrix visualization for three sample documents. Each row represents a document, each column a vocabulary word. Cell intensity indicates word frequency, with darker cells showing higher counts. Notice how 'the' appears twice in Documents 1 and 2 (it occurs twice in each), while most words appear only once or not at all.

Look at the matrix structure. Document 1 has two occurrences of "the" (the cat... the mat), reflected in the count of 2. Document 3 shares vocabulary with both other documents, which we can see from the overlapping non-zero entries. The zeros tell us which words are absent. In practice, this means the matrix will be dominated by zeros, because any given document uses only a small fraction of the total vocabulary. We will return to this sparsity problem shortly.

Document Similarity from Word CountsLink Copied

Once documents become vectors, we can measure their similarity using cosine similarity. This metric computes the cosine of the angle between two vectors, effectively measuring how similar their directions are in high-dimensional space, regardless of their magnitudes. Think of it as measuring not how long the vectors are, but which direction they point. Two documents that use the same words in similar proportions will point in roughly the same direction, even if one is much longer than the other.

The motivation for cosine similarity over simpler measures like Euclidean distance is important to understand. Euclidean distance is sensitive to document length: a long document that repeats many of the same words as a short document would appear far away in Euclidean space, simply because its word counts are larger. Cosine similarity normalizes by vector magnitude, so a 500-word article and a 50-word summary on the same topic will score close to 1.0 similarity even though their raw count vectors look very different in scale.

Given two document vectors $\mathbf{a}$ and $\mathbf{b}$ , cosine similarity computes the normalized dot product between them. The dot product captures how many words the two documents share (and in what quantities), while the normalization by vector magnitudes accounts for document length differences:

\text{cosine\_similarity}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \|\mathbf{b}\|} = \frac{\sum_{i=1}^{n} a_i b_i}{\sqrt{\sum_{i=1}^{n} a_i^2} \cdot \sqrt{\sum_{i=1}^{n} b_i^2}}

where:

$\mathbf{a} \cdot \mathbf{b}$ : the dot product of vectors $\mathbf{a}$ and $\mathbf{b}$ , computed as $\sum_{i=1}^{n} a_i b_i$
$\|\mathbf{a}\|$ : the Euclidean norm (magnitude) of vector $\mathbf{a}$ , computed as $\sqrt{\sum_{i=1}^{n} a_i^2}$
$n$ : the vocabulary size (number of dimensions in each vector)
$a_i, b_i$ : the word counts at position $i$ in vectors $\mathbf{a}$ and $\mathbf{b}$ respectively

The result ranges from 0 (completely different, no shared vocabulary) to 1 (identical word distributions). By normalizing by vector magnitudes, cosine similarity ensures that documents with similar word proportions are considered similar even if one document is much longer than the other.

In[8]:

Code

from sklearn.metrics.pairwise import cosine_similarity

# Compute pairwise cosine similarity
similarity_matrix = cosine_similarity(doc_term_matrix)

Out[9]:

Visualization

Heatmap showing pairwise cosine similarity between three documents, with values ranging from 0.67 to 1.00. — Cosine similarity matrix between our three sample documents. Documents 1 and 2 share 'the', 'sat', and 'on', giving them moderate similarity. Document 3 shares 'the', 'cat', and 'dog' with the others but lacks the verb phrase, resulting in lower similarity scores. The diagonal shows perfect self-similarity (1.0), confirming that each document is identical to itself.

Documents 1 and 2 are most similar because they share the phrase structure "The [animal] sat on the [object]". Document 3, with its different structure, shows lower similarity to both. This demonstrates how BoW captures topical similarity through shared vocabulary, even though it ignores word order. Notice that the similarity matrix is symmetric: the similarity from Document 1 to Document 2 equals the similarity from Document 2 to Document 1, which makes intuitive sense since cosine similarity is commutative.