Inverse Document Frequency › Implementing IDF Efficiently
Language AI HandbookAll chapters Inverse Document Frequency › Implementing IDF EfficientlyOne section · Sections in this chapter7 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
For large corpora, efficient IDF computation matters. Let's compare a naive implementation with an optimized approach:
import time
from collections import defaultdict
# Naive implementation: iterate through all documents for each term
def compute_idf_naive(tokenized_corpus):
"""Naive O(V * D) implementation."""
all_terms = set()
for doc in tokenized_corpus:
all_terms.update(doc)
num_docs = len(tokenized_corpus)
idf = {}
for term in all_terms:
df = sum(1 for doc in tokenized_corpus if term in set(doc))
idf[term] = np.log(num_docs / df)
return idf
# Optimized implementation: single pass through corpus
def compute_idf_optimized(tokenized_corpus):
"""Optimized O(total_tokens) implementation."""
doc_freq = defaultdict(int)
for doc in tokenized_corpus:
for term in sorted(set(doc)): # Stable order across theme renders
doc_freq[term] += 1
num_docs = len(tokenized_corpus)
idf = {term: np.log(num_docs / df) for term, df in doc_freq.items()}
return idf
# Benchmark on repeated corpus
benchmark_corpus = tokenized_corpus * 100 # 500 documents
start = time.time()
for _ in range(10):
_ = compute_idf_naive(benchmark_corpus)
naive_time = time.time() - start
start = time.time()
for _ in range(10):
_ = compute_idf_optimized(benchmark_corpus)
optimized_time = time.time() - startIDF Computation Benchmark: ================================================== Corpus size: 500 documents Iterations: 10 -------------------------------------------------- Naive implementation: 0.031s Optimized implementation: 0.005s Speedup: 6.5x
The speedup demonstrates why algorithm choice matters. The optimized version makes a single pass through the corpus, using a set to count each term once per document. The naive version iterates through all documents for each vocabulary term, making it instead of . For production systems with millions of documents and large vocabularies, this difference can mean hours versus seconds of computation time.
The key insight in the optimized version is to invert the loop structure. Instead of asking "for each term, how many documents contain it?", ask "for each document, which terms does it contain?" The latter question requires only a single pass through the data, building up document frequency counts incrementally.
The second plot reveals an important property: IDF values for terms at fixed fractions of the corpus are completely corpus-size independent. A term appearing in 10% of documents always has IDF , whether you have 100 documents or 100 million. This scale-invariance is one of IDF's practical strengths: you can compare IDF weights across corpora as long as you think in terms of fractions rather than absolute document counts.
Using scikit-learn for Production
For production systems, use scikit-learn's TfidfVectorizer, which computes IDF efficiently and handles all edge cases:
from sklearn.feature_extraction.text import TfidfVectorizer
# Create vectorizer (use_idf=True is default)
vectorizer = TfidfVectorizer(use_idf=True, smooth_idf=True, norm=None)
# Fit on corpus to learn vocabulary and IDF weights
vectorizer.fit(corpus)
# Access IDF values
sklearn_idf = dict(zip(vectorizer.get_feature_names_out(), vectorizer.idf_))scikit-learn IDF Values (smooth_idf=True): ======================================================= Term sklearn IDF Our sklearn ------------------------------------------------------- agent 2.0986 2.0986 agents 2.0986 2.0986 algorithms 2.0986 2.0986 and 2.0986 2.0986 computer 2.0986 2.0986 convolutional 2.0986 2.0986 core 2.0986 2.0986 cumulative 2.0986 2.0986 error 2.0986 2.0986 for 2.0986 2.0986 hierarchical 2.0986 2.0986 image 2.0986 2.0986
The values match exactly between scikit-learn's implementation and our manual calculation of the sklearn variant formula. This confirms that TfidfVectorizer with smooth_idf=True uses . For production applications, always use scikit-learn rather than implementing IDF manually, as it handles edge cases, optimizes memory usage, and integrates directly with machine learning pipelines.