Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

TF-IDF › TF-IDF Variants

Michael BrenndoerferUpdated March 21, 20267 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 13TF-IDF › TF-IDF Variants

Reading section 3 of 13 · TF-IDF Variants · 7 minSection 3 of 13 · 7 min

Section 3 of 13 · 7 min read

TF-IDF › TF-IDF VariantsOne section · Sections in this chapter3 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The formula $\text{tfidf} = \text{tf} \times \text{idf}$ is a template, not a fixed prescription. Both the TF and IDF components have well-known variants that address specific problems in the basic formula. Understanding these variants helps you choose the right formulation for your application, and helps you understand why scikit-learn's defaults are what they are.

Term Frequency VariantsLink Copied

Raw term frequency has a flaw: a document where "neural" appears 10 times doesn't necessarily discuss neural networks twice as deeply as a document where it appears 5 times. Long documents naturally have higher raw counts for all terms, which means a long document discussing many topics might outscore a focused short document on the same single topic. Several variants address this.

Raw count is $\text{tf}(t, d) = f_{t,d}$ , the simplest option, directly counting occurrences. It's appropriate when document lengths are similar across the corpus, but it can bias retrieval toward longer documents.

Log-scaled TF compresses the raw count:

\text{tf}(t, d) = \begin{cases} 1 + \log(f_{t,d}) & \text{if } f_{t,d} > 0 \\ 0 & \text{otherwise} \end{cases}

The logarithm ensures that a term appearing 100 times doesn't count 100 times more than a term appearing once. Under log-scaled TF, going from 1 to 10 occurrences adds $\log(10) \approx 2.3$ to the score, while going from 10 to 100 occurrences adds the same amount again. This reflects diminishing returns: after a term appears several times, each additional occurrence contributes less new information.

Boolean TF simplifies radically: $\text{tf}(t, d) = 1$ if the term appears at all, and 0 otherwise. This discards all frequency information, treating presence and absence as the only distinction. Boolean TF can work well when the relevant signal is simply whether a document mentions a topic, not how often.

Augmented TF normalizes by the maximum count in the document:

\text{tf}(t, d) = 0.5 + 0.5 \cdot \frac{f_{t,d}}{\max_{t'} f_{t',d}}

The constant 0.5 prevents zero TF for terms appearing less than the maximum. The normalization by the most frequent term in the document ensures that TF is always bounded between 0.5 and 1.0, completely decoupling the score from document length. This variant is particularly useful for comparing documents of very different lengths.

Among these, log-scaled TF (sublinear TF) is the most commonly recommended variant in modern systems. It preserves relative ordering while preventing high counts from dominating, and it's available in scikit-learn via sublinear_tf=True.

IDF VariantsLink Copied

The classic IDF formula $\log(N / \text{df}(t))$ has two edge cases that warrant attention. First, if a term appears in every document, $\text{df}(t) = N$ and IDF becomes 0, completely zeroing out its TF-IDF score regardless of how often the term appears in any individual document. Whether this is desirable depends on the application: sometimes you want to suppress universal terms, and sometimes a universal term still provides useful signal when it appears with different frequencies in different documents. Second, if a query contains a term not present in the corpus, $\text{df}(t) = 0$ causes a division-by-zero error.

Smoothed IDF addresses both issues:

\text{idf}(t, D) = \log\!\left(\frac{N + 1}{\text{df}(t) + 1}\right) + 1

where:

$N$ : total number of documents
$\text{df}(t)$ : number of documents containing term $t$
Adding 1 to numerator and denominator prevents division-by-zero and ensures IDF remains positive for all terms
The final $+1$ outside the logarithm ensures that even terms appearing in all documents receive a small positive weight

The $+1$ outside the logarithm is subtle but important. Without it, a term present in all $N$ documents would get $\log\!\left(\frac{N+1}{N+1}\right) = \log(1) = 0$ , the same as the raw formula. With it, even universal terms receive a weight of exactly 1. This ensures their TF contribution is preserved rather than erased. This is the default behavior in scikit-learn's TfidfVectorizer.

Max IDF caps the IDF at a maximum value, preventing very rare terms (possibly misspellings or OCR errors appearing in exactly one document) from receiving outsized weights.

Probabilistic IDF, used in BM25, takes a different form: $\log\!\left(\frac{N - \text{df}(t) + 0.5}{\text{df}(t) + 0.5}\right)$ . This variant derives from the Robertson-Spärck Jones probabilistic relevance model and can produce negative values for very common terms, which are then typically clipped to 0.

In[6]:

Code

def log_tf(f):
    """Log-scaled term frequency."""
    return 1 + np.log(f) if f > 0 else 0


def augmented_tf(f, max_f):
    """Augmented term frequency normalized by max count in document."""
    return 0.5 + 0.5 * (f / max_f)


def smoothed_idf(df_t, N):
    """Smoothed IDF: prevents zero IDF for universal terms."""
    return np.log((N + 1) / (df_t + 1)) + 1


# Compare raw vs smoothed IDF for terms with different document frequencies
sample_terms = {
    "backpropagation": 1,  # Appears in 1 of 5 docs
    "neural": 2,  # Appears in 2 of 5 docs
    "learning": 5,  # Appears in all 5 docs
}

results = {}
for term, df_t in sample_terms.items():
    raw = np.log(N / df_t)
    smoothed = smoothed_idf(df_t, N)
    results[term] = {"df": df_t, "idf_raw": raw, "idf_smoothed": smoothed}

Out[7]:

Console

Term                   df    IDF (raw)   IDF (smoothed)
--------------------------------------------------------
backpropagation         1        1.6094          2.0986
neural                  2        0.9163          1.6931
learning                5        0.0000          1.0000

Notice how "learning" (present in all 5 documents) gets IDF = 0 under the raw formula, completely zeroing out its contribution to any TF-IDF score. Under smoothed IDF, it gets a small positive value of 1.0 (the floor added by the constant $+1$ ). This prevents complete suppression of universally-present terms, which matters when those terms appear more frequently in some documents than others. The word "learning" appearing 20 times in a document about learning theory versus once in a document about computer graphics still encodes useful signal; smoothed IDF preserves it.

Visualizing IDF DecayLink Copied

It's worth visualizing how IDF scores change as a term becomes more common across the corpus.

Out[8]:

Visualization

Line plot showing IDF score versus document frequency for raw and smoothed IDF in a 1000-document corpus. — IDF score as a function of document frequency for a corpus of 1,000 documents, comparing raw IDF and smoothed IDF. Raw IDF drops to zero when a term appears in all documents, while smoothed IDF maintains a floor of 1.0. Both variants show rapid decay as document frequency increases from 1 to 50, confirming that even moderate commonality substantially reduces a term's discriminative value.

Both IDF variants show rapid decay as document frequency grows. A term appearing in 10 out of 1,000 documents receives an IDF of about 4.6 under raw IDF, while a term appearing in 100 documents receives about 2.3, half as much. The decay is steep at low document frequencies and flattens out toward the high end. This shape is a direct consequence of the logarithm: a halving of document frequency always adds a fixed $\log(2) \approx 0.69$ to the IDF score, regardless of the starting value.