Inverse Document Frequency › The IDF Formula

We've established that document frequency reveals which words are common versus rare across a corpus. But document frequency measures commonality, while the task requires a measure of informativeness. The more documents a word appears in, the less useful it is for distinguishing between documents. We need to flip the relationship.

Think about what makes a word useful for identifying a document's topic. If someone mentions "convolutional" in a conversation about machine learning papers, you immediately know they're discussing computer vision or deep learning architectures. That single word narrows down the possibilities dramatically. But if they mention "learning", you've learned almost nothing, because every paper in the corpus discusses learning in some form.

This intuition suggests a simple principle: the fewer documents a word appears in, the more informative it is. A word appearing in 1 out of 100 documents carries far more signal than a word appearing in 99 out of 100. We want to assign weights that reflect this inverse relationship.

The most direct approach would be to use the inverse of the document frequency fraction. If a word appears in $\text{df}(t)$ documents out of $N$ total, its "rarity" could be measured as:

$$\frac{N}{\text{df}(t)}$$

where:

• $N$: the total number of documents in the corpus
• $\text{df}(t)$: the document frequency of term $t$ (how many documents contain it)

This ratio captures the essence of what we want. For a word appearing in just 1 document out of 100, the ratio is $100/1 = 100$. For a word appearing in all 100 documents, the ratio is $100/100 = 1$. Rare words get high values; common words get low values.

But there's a problem with using this ratio directly.

Consider a corpus of 1 million documents. A word appearing once would get weight 1,000,000. A word appearing in half the documents would get weight 2. This 500,000-fold difference is extreme. In practice, it would mean that a single rare word would completely dominate any calculation, drowning out the contribution of all other words.

What we need is a function that preserves the ordering (rare words still get higher weights than common words) but compresses the range of values. The logarithm is the natural choice for this transformation. It converts multiplicative differences into additive ones, turning that 500,000x gap into something more manageable.

With the logarithm applied to our 1-million-document example, the IDF values become:

• Word appearing once: $\text{idf} = \log(1{,}000{,}000 / 1) = \log(1{,}000{,}000) \approx 13.8$
• Word appearing in 500,000 documents: $\text{idf} = \log(1{,}000{,}000 / 500{,}000) = \log(2) \approx 0.69$

The ratio is now about 20:1 instead of 500,000:1. Rare words still matter more, but they don't completely overwhelm everything else.

There is also a deeper reason why logarithm fits here. When you combine IDF with term frequency to produce TF-IDF, both components should contribute to the final weight. If IDF ranged from 1 to 1,000,000, a word's corpus-wide rarity would completely swamp its within-document frequency. The logarithm brings both factors into a compatible scale, allowing each to shape the final weight proportionally.

This brings us to the complete IDF formula:

Let's trace through what happens at the extremes to build intuition:

When a word appears in every document ($\text{df}(t) = N$):

Substituting $\text{df}(t) = N$ into the IDF formula gives:

$$\text{idf}(t) = \log\left(\frac{N}{N}\right) = \log(1) = 0$$

A word appearing everywhere provides zero discriminative information. This makes sense: if every document contains "the", knowing a document contains "the" tells you nothing about which document it is.

When a word appears in exactly one document ($\text{df}(t) = 1$):

Substituting $\text{df}(t) = 1$ into the IDF formula gives:

$$\text{idf}(t) = \log\left(\frac{N}{1}\right) = \log(N)$$

This is the maximum possible IDF value for a given corpus size. Such words are maximally informative because they uniquely identify specific documents.

When a word appears in half the documents ($\text{df}(t) = N/2$):

Substituting $\text{df}(t) = N/2$ into the IDF formula gives:

$$\text{idf}(t) = \log\left(\frac{N}{N/2}\right) = \log(2) \approx 0.69$$

This value is independent of corpus size. A word appearing in half the documents always has the same IDF, whether the corpus has 10 documents or 10 million. This corpus-size invariance for fractional document frequencies is a useful property: IDF weights are comparable across corpora of different sizes, as long as you express document frequency as a fraction rather than an absolute count.

The choice of logarithm base affects the scale of IDF values but not their relative ordering. Natural log (ln), log base 2, and log base 10 are all common choices. scikit-learn uses natural log by default, which gives IDF values in the range $[0, \ln(N)]$.

Let's translate the formula into code. The implementation is straightforward: for each term, we compute the log of the ratio of total documents to document frequency.

The output confirms what we predicted. Words appearing in all 5 documents (like "learning" and "from") have IDF = 0, since $\log(5/5) = \log(1) = 0$. Words appearing in only 1 document (like "reinforcement" and "convolutional") achieve the maximum IDF of $\log(5) \approx 1.61$. This range, from 0 to $\log(N)$, is characteristic of the standard IDF formula.

The relationship between document frequency and IDF follows a logarithmic curve. As document frequency increases, IDF decreases, but the rate of decrease slows down. The table below shows the exact IDF values for each possible document frequency in our 5-document corpus:

Notice how the IDF decrease is steeper between DF=1 and DF=2 (dropping by 0.69) than between DF=4 and DF=5 (dropping by only 0.22). This non-linear relationship means that the distinction between "unique" and "appears twice" matters more than the distinction between "very common" and "ubiquitous." At the top end of the rarity scale, small differences in document frequency translate to large differences in IDF weight. This is by design: a term appearing in only 2 documents has one-fifth the document frequency of one appearing in 10 documents, and IDF captures this with appropriately scaled weights.

The logarithm's importance becomes apparent at larger corpus scales. Without it, rare words receive weights that completely dominate calculations.