Inverse Document Frequency › Inverse Document Frequency

Term frequency tells you how important a word is within a document. But it says nothing about how important that word is across your entire corpus. The word "learning" appearing 5 times in a document about machine learning is meaningful. The word "the" appearing 5 times is not. Both have the same term frequency, yet one carries far more information.

Inverse Document Frequency (IDF) addresses this gap. It measures how rare or common a word is across all documents, giving higher weights to words that appear in fewer documents. When combined with term frequency, IDF creates TF-IDF, one of the most successful text representations in information retrieval history. Its elegance lies in being both intuitively obvious (rare words are informative) and theoretically grounded (it derives directly from information theory's concept of self-information).

This chapter develops IDF from first principles. You'll learn why rare words matter more, derive the IDF formula step-by-step with careful attention to the design choices, explore smoothing variants that prevent mathematical edge cases, and implement efficient IDF computation. Along the way, we'll trace the formula's connection to information theory, examine how different smoothing choices behave across corpus sizes, and see how the ideas behind IDF show up in modern retrieval systems. By the end, you'll understand what IDF computes and why every design decision was made, leaving you ready to combine it with TF in the next chapter.