TF-IDF › Summary

TF-IDF combines term frequency (local prominence) and inverse document frequency (global rarity) through multiplication, creating document vectors where high scores identify terms that are both frequent in a document and rare across the corpus. The elegance of the formula lies in its selectivity: the multiplication gate ensures that only terms satisfying both conditions simultaneously receive high scores.

Key takeaways:

• The formula: $\text{tfidf}(t, d, D) = \text{tf}(t, d) \times \text{idf}(t, D)$, where multiplication rewards both local prominence and global rarity simultaneously.
• Why the logarithm: The log in IDF compresses the ratio $N / \text{df}(t)$ so that very rare terms don't receive outsized weights, and scales gracefully across corpora of different sizes.
• Variants matter: Log-scaled TF compresses the effect of high counts; smoothed IDF prevents zero weights for universal terms. Scikit-learn's defaults (log TF off, smooth IDF on, L2 norm) are a sensible starting point, but enabling sublinear_tf=True is recommended for most classification tasks.
• Normalization: L2 normalization is standard, enabling cosine similarity as a simple dot product between normalized vectors and removing document-length bias.
• Document similarity: TF-IDF vectors combined with cosine similarity form the foundation of classical vector space model information retrieval. With L2-normalized vectors, the full similarity matrix is computed as $\hat{M} \hat{M}^T$.
• Feature extraction: TF-IDF + logistic regression is a strong baseline for text classification, often competitive with far more complex approaches when categories have distinctive vocabulary. The combination is fast and interpretable, with coefficients that can be audited.
• sklearn TfidfVectorizer: The production-ready implementation with parameters to control TF scaling (sublinear_tf), IDF smoothing (smooth_idf), normalization (norm), vocabulary pruning (min_df, max_df, max_features), and n-gram range (ngram_range).
• BM25 extends TF-IDF: By adding TF saturation via a rational function and explicit document-length normalization, BM25 addresses TF-IDF's main retrieval weakness. It is the standard algorithm in modern search systems.
• Semantic blindness is the core limitation: TF-IDF cannot handle synonyms, polysemy, word order, or context. It matches exact terms. Dense embeddings from neural models address this limitation by encoding semantics directly into vector geometry, at the cost of interpretability and computational efficiency.