TF-IDF and Bag of Words: Text Representation & Search › Worked Example

Let's work through a concrete example with three short documents:

• Document 1: "the cat sat on the mat"
• Document 2: "the dog sat on the log"
• Document 3: "the cat and dog sat"

First, we build our vocabulary by collecting all unique words: $\{ \text{the}, \text{cat}, \text{sat}, \text{on}, \text{mat}, \text{dog}, \text{log}, \text{and} \}$. That's 8 words, so our vectors will have 8 dimensions.

Bag of Words vectors:

• Document 1: $[2, 1, 1, 1, 1, 0, 0, 0]$ (2×"the", 1×"cat", 1×"sat", 1×"on", 1×"mat")
• Document 2: $[2, 0, 1, 1, 0, 1, 1, 0]$ (2×"the", 1×"sat", 1×"on", 1×"dog", 1×"log")
• Document 3: $[1, 1, 1, 0, 0, 1, 0, 1]$ (1×"the", 1×"cat", 1×"sat", 1×"dog", 1×"and")

Notice that "the" appears in all three documents, "sat" appears in all three, but "mat", "log", and "and" appear in only one document each.

Calculating IDF:

• "the": appears in 3/3 documents → $\text{IDF} = \log(3/3) = 0$
• "sat": appears in 3/3 documents → $\text{IDF} = \log(3/3) = 0$
• "cat": appears in 2/3 documents → $\text{IDF} = \log(3/2) \approx 0.405$
• "dog": appears in 2/3 documents → $\text{IDF} = \log(3/2) \approx 0.405$
• "mat": appears in 1/3 documents → $\text{IDF} = \log(3/1) \approx 1.099$
• "log": appears in 1/3 documents → $\text{IDF} = \log(3/1) \approx 1.099$
• "and": appears in 1/3 documents → $\text{IDF} = \log(3/1) \approx 1.099$

Calculating TF-IDF for Document 1:

Using raw counts for TF:

• "the": $\text{TF} = 2$, $\text{IDF} = 0$ → $\text{TF-IDF} = 2 \times 0 = 0$
• "cat": $\text{TF} = 1$, $\text{IDF} = 0.405$ → $\text{TF-IDF} = 1 \times 0.405 = 0.405$
• "mat": $\text{TF} = 1$, $\text{IDF} = 1.099$ → $\text{TF-IDF} = 1 \times 1.099 = 1.099$

The word "mat" gets the highest TF-IDF score in Document 1 because it's unique to that document. "The" gets zero weight because it appears everywhere. This is exactly the behavior we want: distinctive words are emphasized, common words are suppressed.