Vector Space Model & TF-IDF (1968) › TF-IDF: Weighing What Matters

The simplest approach to term weighting would be to just count occurrences: if the word "neural" appears 10 times in a document, give it a weight of 10 in that dimension. This is called term frequency (TF), and it captures the intuition that words that appear many times in a document are probably important to its content. If a research paper mentions "transformer" fifty times, it's likely about transformers.

Raw term frequency gives high weight to common words that appear in almost every document. In a computer-science collection, terms such as "the," "is," "algorithm," and "result" may occur throughout the corpus even though they do little to distinguish one document. IDF reduces the contribution of such collection-wide terms.

This insight led to the inverse document frequency (IDF) component. The IDF of a term is higher when the term appears in fewer documents. If a term appears in every document in the collection, its IDF is very low (approaching zero). If a term appears in only a handful of documents, its IDF is high. Mathematically, IDF is typically defined as:

where $N$ is the total number of documents in the collection and $df(t)$ is the number of documents containing term $t$. The logarithm smooths the scale and ensures that the values remain computationally manageable.

Combining these two components gives us TF-IDF weighting:

A term receives a high TF-IDF weight when it occurs frequently in one document but rarely across the collection. The word "the" may have high TF and very low IDF, producing a low combined weight. In a collection of AI papers, "backpropagation" may have moderate TF but high IDF and therefore distinguish a smaller group of documents.

With TF-IDF weighting, each coordinate estimates how strongly a term distinguishes a document within the collection. Documents with similar weighted term distributions point in similar directions and receive higher cosine-similarity scores.