Inverse Document Frequency › Key Functions and Parameters

When working with IDF in scikit-learn, the TfidfVectorizer class handles both TF and IDF computation:

TfidfVectorizer(use_idf, smooth_idf, sublinear_tf, norm)

The key parameters are:

• use_idf (default: True): Whether to apply IDF weighting. Set to False to compute only term frequency without IDF. Useful when you want to isolate the effect of IDF in an experiment.

• smooth_idf (default: True): Whether to add 1 to document frequencies to prevent division by zero and ensure all terms get positive IDF values. Uses the formula $\log\left(\frac{1 + N}{1 + \text{df}(t)}\right) + 1$, where $N$ is the total number of documents and $\text{df}(t)$ is the document frequency of term $t$. Set to False to use the standard formula $\log\left(\frac{N}{\text{df}(t)}\right)$.

• sublinear_tf (default: False): Whether to apply log-scaling to term frequency. When True, uses $1 + \log(\text{tf})$ instead of raw counts, where $\text{tf}$ is the raw term frequency (count of the term in the document). Sublinear TF prevents very high-frequency terms in a single document from dominating the TF-IDF score.

• norm (default: 'l2'): Normalization applied to output vectors. Use 'l2' for cosine similarity, 'l1' for Manhattan distance, or None for raw TF-IDF values. L2 normalization ensures that document length doesn't affect similarity scores, making short and long documents comparable.

The idf_ attribute contains the learned IDF weights after fitting, accessible via vectorizer.idf_. The vocabulary_ attribute maps terms to their column indices in the output matrix. Both attributes are set during fit() and remain fixed when you call transform(). This ensures consistent feature spaces between training and test data.