Inverse Document Frequency › Visualizing IDF in Action

Michael BrenndoerferMarch 28, 20256 min read
Language AI HandbookAll chapters
Part 2 of 67Classical Text Representations
Chapter 12Inverse Document Frequency › Visualizing IDF in Action
Reading section 8 of 14 · Visualizing IDF in Action · 6 minSection 8 of 14 · 6 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 12 of 513

Section 8 of 14 · 6 min read

Inverse Document Frequency › Visualizing IDF in ActionOne section · Sections in this chapter8 / 14 ↓
Sections in this chapter14
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's see how IDF transforms our understanding of word importance. We'll build several visualizations that show the inverse relationship between document frequency and IDF weights, and how IDF identifies the most discriminative terms in a corpus.

Out[26]:
Visualization
Scatter plot with terms colored by IDF value showing inverse relationship between document frequency and information weight.
Scatter plot showing the inverse relationship between document frequency and IDF weight for all terms in the corpus. Rare terms at DF=1 achieve the maximum IDF of 1.61, while the most common terms in this corpus reach DF=4 and IDF=0.22. Each point represents a unique vocabulary term, colored by IDF value from red (low) to green (high); the neutral panel preserves the names of all 34 terms at DF=1 without stacking them on one point.

Terms Ranked by Informativeness

The table below shows corpus terms ranked by IDF value, revealing which words are most discriminative for identifying individual documents within this machine learning paper corpus:

Terms ranked by IDF value. Topic-specific words like "reinforcement" and "convolutional" achieve maximum IDF (1.61), while ubiquitous words like "learning" receive zero weight. The DF column shows how many of the 5 documents contain each term.
RankTermDFIDFInterpretation
1reinforcement11.61Unique to Doc 4 (RL topic)
2convolutional11.61Unique to Doc 5 (CV topic)
3vision11.61Unique to Doc 5 (CV topic)
4reward11.61Unique to Doc 4 (RL topic)
5image11.61Unique to Doc 5 (CV topic)
6agents11.61Unique to Doc 4 (RL topic)
7neural20.92Specific to deep learning docs
8deep20.92Specific to deep learning docs
9networks20.92Specific to deep learning docs
10data20.92Appears in 2 docs
...............
.learning40.22Appears in four documents
.from30.51Appears in three documents

Notice the groupings: words tied to specific sub-fields of machine learning (reinforcement learning, computer vision) get maximum IDF and can perfectly identify their source documents. Words shared between a few sub-fields (neural, deep, networks) get intermediate IDF and help narrow down document subsets. No term appears in all five documents here; if one did, it would receive zero IDF and contribute nothing to identification.

This is exactly the discrimination hierarchy we want. A search for "reinforcement learning" should privilege documents that use words unique to that field. A search for "neural networks" should surface documents that discuss deep learning broadly. And a search for "learning" alone should find all documents equally, since they all discuss some form of learning.

IDF Heatmap Across Documents

Let's visualize how IDF weights apply across our document collection:

Out[27]:
Visualization
Heatmap showing IDF-weighted term presence with documents as rows and terms as columns, colored green for high-IDF and red for low-IDF.
IDF-weighted term presence across the five-document corpus for the ten highest-IDF and ten lowest-IDF terms. Each colored cell shows the IDF value for a present term, with green indicating high-IDF rare terms and red indicating lower-IDF common terms; paper-colored cells indicate absence. The left half cleanly separates documents through rare terms, while the right half shows terms shared across multiple documents.

The heatmap reveals document structure with striking clarity. Document 4 (the reinforcement learning abstract) is characterized exclusively by "reinforcement", "reward", and "agents", all of which have the maximum IDF of 1.61. Document 5 (the computer vision abstract) is uniquely identified by "vision", "convolutional", and "image". Each high-IDF term on the left side of the heatmap appears in exactly one document, making it a perfect discriminator. The lower-IDF words on the right side of the heatmap spread across multiple documents.

When you build a TF-IDF vector for document retrieval, you're essentially asking: which cells in this heatmap have the largest values for this document? Those are the words that define the document's identity within the corpus. A query that uses those words will match this document; a query that uses only common words will match everything equally.