TF-IDF › sklearn TfidfVectorizer Deep Dive
Language AI HandbookAll chapters TF-IDF › sklearn TfidfVectorizer Deep DiveOne section · Sections in this chapter7 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Implementing TF-IDF from scratch teaches the mechanics, but scikit-learn's TfidfVectorizer is the standard tool for production use. It handles tokenization and IDF computation, then normalizes the output and stores it as a sparse matrix through a single, well-tested API. Understanding its parameters helps you get the most out of it.
Basic Usage
from sklearn.feature_extraction.text import TfidfVectorizer
# Default settings: uses smoothed IDF and L2 normalization
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)Matrix type: <class 'scipy.sparse._csr.csr_matrix'>
Matrix shape: (5, 38)
Non-zero entries: 48
Sparsity: 74.74%
Vocabulary size: 38
Sample vocabulary: [('machine', 18), ('learning', 17), ('algorithms', 1), ('learn', 16), ('patterns', 25)]TfidfVectorizer.fit_transform() returns a sparse CSR matrix, not a dense NumPy array. This is critical for performance: a real corpus with tens of thousands of documents and a vocabulary of hundreds of thousands of terms would consume many gigabytes as a dense matrix, while the same data stored as a sparse CSR matrix might fit comfortably in a few hundred megabytes.
The fit_transform() call does two things at once: it learns the vocabulary and IDF values from the training data (fit), and it converts the training data into TF-IDF vectors (transform). When you encounter new documents later, you call transform() alone to apply the already-learned IDF values to new text. This separation is important: you don't recompute IDF from each incoming document; you use the IDF values estimated from your training corpus.
Key Parameters
The TfidfVectorizer has many parameters. The most important ones to understand are:
use_idf(defaultTrue): WhenTrue, multiplies TF by IDF. Set toFalseto get pure TF weighting, effectively reverting to a simple count-based representation.smooth_idf(defaultTrue): WhenTrue, uses the smoothed IDF formula to prevent zero weights for terms appearing in all documents.sublinear_tf(defaultFalse): WhenTrue, applies log-scaled TF. Recommended for longer documents where raw counts can grow very large.norm(default'l2'): Normalization applied to output vectors. Options are'l2','l1', orNone.max_df(default1.0): Ignores terms with document frequency above this threshold. Setmax_df=0.9to exclude terms appearing in more than 90% of documents, effectively implementing a corpus-specific stopword list.min_df(default1): Ignores terms with document frequency below this threshold. Setmin_df=2to exclude hapax legomena (terms appearing in exactly one document), which are often noise in large corpora.max_features(defaultNone): Limits vocabulary to the top terms by term frequency across the corpus. This is the fastest way to control vocabulary size on large corpora.ngram_range(default(1,1)): Controls which n-grams are included. Settingngram_range=(1,2)includes both unigrams and bigrams, which can capture multi-word expressions like "machine learning" or "deep neural."analyzer(default'word'): Whether to analyze at word or character level. Character n-grams (analyzer='char'with a range like(2,4)) can be surprisingly effective and more tolerant of misspellings.tokenizer: Custom tokenization function. You can replace the default whitespace/punctuation tokenizer with anything, including language-specific tokenizers for languages without word boundaries.
# Tuned vectorizer with log TF and vocabulary controls
tuned_vectorizer = TfidfVectorizer(
sublinear_tf=True, # Log-scaled TF
smooth_idf=True, # Smoothed IDF
norm="l2", # L2 normalization
min_df=1, # Include all terms (small corpus)
max_df=1.0, # No upper cutoff
ngram_range=(1, 2), # Include bigrams
)
X_tuned = tuned_vectorizer.fit_transform(corpus)Vocabulary size with bigrams: 86 Sample bigrams in vocabulary: 'agents learn' 'algorithms learn' 'analyzes images' 'computer vision' 'data is'
Adding bigrams to the vocabulary captures multi-word phrases that are more specific than their individual component words. The phrase "deep learning" has a very different meaning from "deep" and "learning" separately, and treating it as a single token gives TF-IDF a way to capture that specificity.
Inspecting IDF Values
After fitting, you can inspect IDF values to understand how the vectorizer weights different terms. This is a useful interpretability property of TF-IDF: you can directly audit what the system considers common versus rare.
# Inspect IDF values from the fitted vectorizer
feature_names = vectorizer.get_feature_names_out()
idf_values = vectorizer.idf_
idf_df = sorted(zip(feature_names, idf_values), key=lambda x: x[1])Lowest IDF (most common terms): learning IDF = 1.1823 is IDF = 1.4055 learn IDF = 1.4055 deep IDF = 1.6931 from IDF = 1.6931 Highest IDF (rarest terms): rewards IDF = 2.0986 techniques IDF = 2.0986 text IDF = 2.0986 through IDF = 2.0986 vision IDF = 2.0986
Terms like "learning" that appear in most documents receive low IDF scores. Terms like "backpropagation" or "hierarchical" that appear in very few documents receive high IDF scores. This transparency is valuable in practice: you can quickly check whether your IDF values reflect domain intuition, and if they don't (perhaps because your corpus has unusual statistical properties), you can diagnose the issue.
TF-IDF Scores Across Variants
How much do the variant choices matter? Let's compare raw versus log TF combined with raw versus smoothed IDF on a similarity task.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
# Four variant combinations
configs = {
"Raw TF + Raw IDF": TfidfVectorizer(
sublinear_tf=False, smooth_idf=False, norm="l2"
),
"Log TF + Raw IDF": TfidfVectorizer(
sublinear_tf=True, smooth_idf=False, norm="l2"
),
"Raw TF + Smooth IDF": TfidfVectorizer(
sublinear_tf=False, smooth_idf=True, norm="l2"
),
"Log TF + Smooth IDF": TfidfVectorizer(
sublinear_tf=True, smooth_idf=True, norm="l2"
),
}
# Compute similarity between Doc 1 and Doc 5 under each variant
variant_sims = {}
for name, vec in configs.items():
try:
X_v = vec.fit_transform(corpus)
sim = cosine_similarity(X_v[0], X_v[4])[0, 0]
variant_sims[name] = sim
except Exception:
variant_sims[name] = float("nan")
The differences between variants can be significant in practice, especially on large, heterogeneous corpora. Log TF + Smooth IDF is the recommended default for most text classification and retrieval tasks. The rationale is straightforward: log scaling prevents any single high-count term from dominating, and smoothed IDF prevents the pathological case where universal terms get zeroed out entirely.