TF-IDF › TF-IDF for Feature Extraction
Language AI HandbookAll chapters TF-IDF › TF-IDF for Feature ExtractionOne section · Sections in this chapter8 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Beyond information retrieval, TF-IDF is widely used to generate features for machine learning classifiers. The TF-IDF matrix provides a numerical representation of text that can feed directly into algorithms like logistic regression, support vector machines, or naive Bayes. The key advantage is that TF-IDF automatically prioritizes discriminative features (rare terms that appear prominently in some documents) and suppresses uninformative ones (common terms), doing implicitly what a feature engineer would need to do explicitly.
Text Classification with TF-IDF Features
Let's see how TF-IDF features perform on a real classification task using the 20 Newsgroups dataset, a standard benchmark for text classification.
from sklearn.datasets import fetch_20newsgroups
# Use 4 newsgroup categories (subset for speed)
categories = ["sci.med", "sci.space", "rec.sport.hockey", "comp.graphics"]
train_data = fetch_20newsgroups(
subset="train",
categories=categories,
remove=("headers", "footers", "quotes"),
random_state=42,
)
test_data = fetch_20newsgroups(
subset="test",
categories=categories,
remove=("headers", "footers", "quotes"),
random_state=42,
)The remove=("headers", "footers", "quotes") argument strips out metadata that would make the classification trivially easy. Without this, the classifier could learn to identify newsgroups from sender addresses and thread structure rather than from the actual content of the posts, which would tell you nothing about TF-IDF's ability to represent meaning.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report
from sklearn.pipeline import Pipeline
# Build TF-IDF + Logistic Regression pipeline
pipeline = Pipeline(
[
(
"tfidf",
TfidfVectorizer(
sublinear_tf=True,
smooth_idf=True,
norm="l2",
min_df=2,
max_df=0.95,
ngram_range=(1, 2),
max_features=50000,
),
),
("clf", LogisticRegression(max_iter=1000, C=5.0, random_state=42)),
]
)
pipeline.fit(train_data.data, train_data.target)
y_pred = pipeline.predict(test_data.data)
accuracy = accuracy_score(test_data.target, y_pred)
report = classification_report(
test_data.target, y_pred, target_names=categories, output_dict=True
)Test accuracy: 0.8669 Per-class F1 scores: sci.med F1=0.870 Precision=0.873 Recall=0.866 sci.space F1=0.900 Precision=0.867 Recall=0.935 rec.sport.hockey F1=0.859 Precision=0.879 Recall=0.841 comp.graphics F1=0.837 Precision=0.849 Recall=0.825
TF-IDF features plus logistic regression form a strong baseline for text classification. Categories with highly distinctive vocabulary (like "rec.sport.hockey," which features terms like "puck," "goalie," "penalty," and "overtime") achieve very high F1 scores because TF-IDF naturally amplifies these discriminative signals. The classifier doesn't need to learn much beyond "if this document has high TF-IDF for hockey terms, it's probably a hockey post." This transparency is both the method's strength and its limitation: it works extremely well when categories have distinctive vocabulary, but it can struggle with categories that express similar ideas in different words.
The pipeline abstraction in scikit-learn prevents data leakage. By wrapping TfidfVectorizer and LogisticRegression in a Pipeline, you ensure that IDF values are learned only from training data and applied consistently to test data. If you fit the vectorizer on all data before splitting, you'd be encoding information about the test set into the IDF values, which would artificially inflate performance estimates.
Most Informative Features
Inspecting which TF-IDF features the classifier weights most strongly is one of the great advantages of this approach. With neural methods, you often can't easily ask "what exactly caused this classification?" With logistic regression on TF-IDF features, you can directly inspect the model's coefficients.
# Extract most informative features per class
tfidf_step = pipeline.named_steps["tfidf"]
clf_step = pipeline.named_steps["clf"]
feature_names = np.array(tfidf_step.get_feature_names_out())
top_n = 8
informative = {}
for i, cat in enumerate(categories):
coefs = clf_step.coef_[i]
top_pos = np.argsort(coefs)[-top_n:][::-1]
informative[cat] = [(feature_names[j], coefs[j]) for j in top_pos]sci.med: graphics coef = 5.4764 file coef = 3.3411 image coef = 3.2663 3d coef = 2.9488 computer coef = 2.9073 sci.space: hockey coef = 5.2792 game coef = 4.7090 team coef = 4.5531 games coef = 3.4404 he coef = 3.3974 rec.sport.hockey: my coef = 3.1425 msg coef = 2.8746 disease coef = 2.7053 doctor coef = 2.5882 medical coef = 2.5193 comp.graphics: space coef = 7.4500 orbit coef = 3.1984 nasa coef = 3.0392 launch coef = 2.7842 moon coef = 2.5760
The most informative features are exactly what domain knowledge would predict: topic-specific terms with high TF-IDF weights that the logistic regression has learned to associate with each category. This interpretability is one of TF-IDF's key advantages over dense neural embeddings, where the analogous "most important features" are usually not human-readable dimensions.
Vocabulary Tuning Strategies
The min_df, max_df, and max_features parameters are tuning levers that trade vocabulary size against coverage and noise. The right settings depend on your corpus and task.
Setting min_df=2 removes hapax legomena (terms appearing in only one document). These terms can't be generalized from during training and often represent noise (misspellings, OCR errors, rare proper nouns specific to one training document). Removing them shrinks the vocabulary, speeds up computation, and often improves generalization.
Setting max_df=0.95 removes terms appearing in more than 95% of documents. These are corpus-specific stopwords: words so common in your domain that they provide no discriminative signal. In a medical corpus, terms like "patient" or "clinical" might appear in essentially every document; filtering them out reduces noise without losing much information.
Setting max_features=50000 or similar caps vocabulary size when your corpus is large enough that the full vocabulary would be unwieldy. Scikit-learn selects features by overall term frequency across the corpus, so this keeps the most common (and presumably most informative) terms.
These three parameters interact: min_df and max_df are relative frequency cutoffs, while max_features is an absolute vocabulary cap. In practice, setting min_df=2 and then max_features=50000 is a reasonable default for most English text classification tasks.