Inverse Document Frequency › IDF Across Corpus Splits
Language AI HandbookAll chapters Inverse Document Frequency › IDF Across Corpus SplitsOne section · Sections in this chapter6 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
In machine learning, we often split data into training and test sets. How should we handle IDF in this scenario? This is where many practitioners make a subtle but important mistake.
The key principle is: compute IDF only on training data, then apply it to test data.
If we compute IDF on the full dataset (including test data), we're leaking information from the test set into our features. When the model sees the IDF weights, it implicitly knows something about the test documents it was never supposed to see. This can lead to overly optimistic performance estimates that don't generalize to truly new documents.
from sklearn.model_selection import train_test_split
# Simulate a larger corpus for meaningful split
larger_corpus = corpus * 4 # 20 documents
labels = [0, 1, 1, 0, 1] * 4 # Dummy labels
# Split into train/test
train_docs, test_docs, train_labels, test_labels = train_test_split(
larger_corpus, labels, test_size=0.3, random_state=42
)
# Compute IDF on training set only
train_tokenized = [tokenize(doc) for doc in train_docs]
train_df = compute_document_frequency(train_tokenized)
train_idf = compute_idf(train_df, len(train_docs))Train/Test Split IDF Handling: ================================================== Training documents: 14 Test documents: 6 IDF computed on training set only: -------------------------------------------------- hierarchical IDF = 2.6391 representations IDF = 2.6391 algorithms IDF = 1.5404 labeled IDF = 1.5404 machine IDF = 1.5404 patterns IDF = 1.5404 requires IDF = 1.5404 supervised IDF = 1.5404
When applying IDF to test documents, terms not seen in training get a default IDF value. There are three approaches:
- Ignore OOV terms: Simply skip words not in the training vocabulary. This is scikit-learn's default.
- Assign maximum IDF: Treat unknown words as maximally rare. This gives OOV terms the most influence, which can be either good (novel technical terms deserve attention) or bad (typos become important).
- Assign zero IDF: Treat unknown words as uninformative. This effectively removes OOV terms, which is conservative but can miss important new terms.
The Vocabulary Mismatch Problem
Test documents may contain words not in the training vocabulary. This out-of-vocabulary (OOV) problem requires a careful decision that depends on your application.
# Demonstrate OOV handling
test_tokenized = [tokenize(doc) for doc in test_docs]
# Find OOV terms
train_vocab = set(train_df.keys())
oov_terms = set()
for doc in test_tokenized:
for term in doc:
if term not in train_vocab:
oov_terms.add(term)Out-of-Vocabulary Analysis: -------------------------------------------------- Training vocabulary size: 43 OOV terms in test set: 0 No OOV terms (test vocabulary is a subset of training vocabulary)
In a real-world scenario with a training corpus of 10,000 documents and a test corpus drawn from a different time period or slightly different domain, you'd typically see 5-15% of test vocabulary falling outside the training vocabulary. Every new product name, every acronym, every domain-specific term that wasn't in training data will be OOV.
The scikit-learn approach of ignoring OOV terms is pragmatic. When you call vectorizer.transform(test_docs), any word not in vectorizer.vocabulary_ is simply treated as absent. The resulting vectors have zeros in those positions. This maintains consistency of the feature space between training and test, which is required for machine learning models to work correctly.
IDF Stability Across Train/Test Splits
A related concern is whether IDF values computed on a training sample are stable estimates of the true corpus-wide IDF. With small training sets, a word that appears in 10% of all documents might, by chance, appear in only 5% of the training sample, inflating its IDF weight. With large training sets, this variance decreases and IDF values become reliable estimates.
As a practical rule, IDF values computed on training sets of 10,000 or more documents are usually stable enough for production use. With smaller training sets, you might consider using smoothing more aggressively, or using domain knowledge to set IDF weights for known common words directly.