Inverse Document Frequency › Summary

Inverse Document Frequency measures how informative a term is across a corpus by computing the logarithm of the inverse document frequency ratio:

where $N$ is the total number of documents in the corpus and $\text{df}(t)$ is the number of documents containing term $t$.

Key insights from this chapter:

• Document frequency counts how many documents contain each term, revealing corpus-wide patterns that term frequency alone cannot capture. It is the global counterpart to TF's local perspective.
• IDF gives higher weights to rare words that appear in few documents. These words are more informative for distinguishing between documents because their presence is more surprising.
• The logarithm compresses the range of weights, preventing rare words from completely dominating common words. Without the log, a corpus of one million documents would give the rarest words weights one million times larger than universal words.
• IDF equals self-information from information theory. The formula derives directly from the concept of surprisal: $I(t) = \log(1/p(t)) = \log(N/\text{df}(t))$. This grounding is not coincidental; it's a mathematical identity that explains why IDF works so well.
• Smoothed variants handle edge cases like terms appearing in all documents (standard IDF gives zero weight) or out-of-vocabulary terms (standard IDF divides by zero). scikit-learn's default adds 1 inside and outside the logarithm. This ensures all terms receive a positive weight.
• Train/test splits require computing IDF only on training data to avoid information leakage. Applying training IDF to test documents is the correct procedure; recomputing IDF on combined data inflates test performance estimates.
• Efficient implementation uses a single pass through the corpus rather than iterating over each term separately, reducing complexity from $O(V \times D)$ to $O(\text{total tokens})$.
• IDF is corpus-specific. Words rare in one domain may be common in another. This domain sensitivity is one of IDF's core limitations and motivates modern neural representations that can transfer across domains.

IDF addresses the key limitation of term frequency: TF treats all words equally regardless of their corpus-wide distribution. By combining TF with IDF, we get TF-IDF, a representation that captures both within-document importance and cross-document discriminative power. The next chapter brings these two components together into the complete TF-IDF scheme.