Inverse Document Frequency › The Problem with Term Frequency Alone
Language AI HandbookAll chapters Inverse Document Frequency › The Problem with Term Frequency AloneOne section · Sections in this chapter2 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
In the previous chapter, we computed term frequency to measure word importance within documents. But TF has a blind spot: it treats all words equally regardless of their corpus-wide distribution.
Consider a corpus of research papers about machine learning:
from collections import Counter
# Sample corpus of ML research abstracts
corpus = [
"Neural networks learn hierarchical representations from data. Deep learning uses neural networks.",
"Machine learning algorithms learn patterns from training data. Supervised learning requires labeled data.",
"Natural language processing uses deep learning for text classification. Language models learn from text.",
"Reinforcement learning agents learn through trial and error. The agent maximizes cumulative reward.",
"Computer vision uses convolutional neural networks. Image classification is a core vision task.",
]
def tokenize(text):
"""Simple tokenization: lowercase and extract words."""
import re
return re.findall(r"\b[a-z]+\b", text.lower())
# Tokenize corpus
tokenized_corpus = [tokenize(doc) for doc in corpus]
# Compute term frequency for Document 1
tf_doc1 = Counter(tokenized_corpus[0])Term Frequencies in Document 1: --------------------------------------------- neural 2 ██ networks 2 ██ learn 1 █ hierarchical 1 █ representations 1 █ from 1 █ data 1 █ deep 1 █ learning 1 █ uses 1 █
Both "neural" and "from" appear twice in Document 1, giving them equal term frequency. But "neural" is specific to this document's topic, while "from" appears in almost every English text. Term frequency alone cannot distinguish between these cases.
This failure has real consequences. Imagine building a search engine that represents every document purely by term frequency and then ranks documents by how many times query terms appear. A query for "neural network architectures" would rank documents containing "the", "of", and "in" almost as high as those containing "neural", because stopwords inflate frequencies. Even removing explicit stopwords doesn't fully solve the problem, since domain-specific filler words ("study", "analysis", "result") are common within a field but vary in how discriminative they are.
The root problem is that TF is a local statistic. It sees only the document in front of it. It has no way to ask, "Compared to everything in my corpus, how unusual is it to see this word here?" That global perspective is exactly what IDF provides.
Document Frequency Reveals Corpus-Wide Patterns
To understand which words are informative, we need to look beyond individual documents. Document frequency (DF) counts how many documents contain each word:
Document frequency measures how many documents in the corpus contain a given term. For term in a corpus , the document frequency is:
where:
- : the term (word) we're measuring
- : the collection of all documents in the corpus
- : an individual document in the corpus
- : the set of documents that contain term
- : the cardinality (size) of the set, i.e., the count of documents
A high DF indicates a common word appearing across many documents. A low DF indicates a rare word appearing in few documents.
def compute_document_frequency(tokenized_corpus):
"""Count how many documents contain each term."""
df = Counter()
for doc in tokenized_corpus:
# Sort the per-document set so repeated theme renders are identical.
unique_terms = sorted(set(doc))
df.update(unique_terms)
return df
# Compute document frequency across corpus
doc_freq = compute_document_frequency(tokenized_corpus)
num_docs = len(tokenized_corpus)Document Frequency Analysis (5 documents): ======================================================= Term DF Fraction Appears in ------------------------------------------------------- learn 4 80% ████░ learning 4 80% ████░ from 3 60% ███░░ uses 3 60% ███░░ classification 2 40% ██░░░ data 2 40% ██░░░ deep 2 40% ██░░░ networks 2 40% ██░░░ neural 2 40% ██░░░ a 1 20% █░░░░ agent 1 20% █░░░░ agents 1 20% █░░░░ algorithms 1 20% █░░░░ and 1 20% █░░░░ computer 1 20% █░░░░
The pattern emerges clearly. Words like "learning" and "from" appear in most documents. This provides little discriminative power. Words like "reinforcement", "vision", and "convolutional" appear in only one document. This makes them highly specific to particular topics.
The Discriminative Power Gap
The histograms make the core problem visual. A massive number of terms sit at DF=1, where they are uniquely tied to a single document. In this corpus, the most widely shared terms reach DF=4; a hypothetical DF=5 term would appear everywhere. The terms in the middle ground, appearing in two or three documents, occupy an interesting position: they distinguish subsets of documents from one another.
A good weighting scheme should assign weights that reflect this hierarchy. Maximum weight for DF=1 terms (they identify individual documents uniquely). Zero or minimal weight for DF=N terms (they tell you nothing about which document you're looking at). Intermediate weights for everything in between.
That's exactly what IDF does.