Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Term Frequency › Log-Scaled Term Frequency

Michael BrenndoerferMarch 30, 20253 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 11Term Frequency › Log-Scaled Term Frequency

Reading section 4 of 17 · Log-Scaled Term Frequency · 3 minSection 4 of 17 · 3 min

Section 4 of 17 · 3 min read

Term Frequency › Log-Scaled Term FrequencyOne section · Sections in this chapter4 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To address diminishing returns, logarithmic scaling compresses the count scale. The key insight is that each successive doubling of count should add a fixed increment of importance, not double it.

\text{tf}_{\text{log}}(t, d) = \begin{cases} 1 + \log(\text{count}(t, d)) & \text{if } \text{count}(t, d) > 0 \\ 0 & \text{otherwise} \end{cases}

where:

$\text{tf}_{\text{log}}(t, d)$ : the log-scaled term frequency weight
$\log(\cdot)$ : the natural logarithm (base $e$ ), though other bases work equally well since log base conversion is just a constant scale factor
The $+1$ offset ensures that a term appearing exactly once receives weight 1, making the zero-versus-nonzero distinction crisp. Without the $+1$ , a term appearing once would receive weight $\log(1) = 0$ , which would be indistinguishable from an absent term.

Some formulations instead use $\log(1 + \text{count}(t, d))$ , which maps count 0 to 0 continuously without needing a piecewise definition. The two forms give very similar results in practice, but the piecewise version is more common in information retrieval literature because it preserves the conceptually clear distinction between "absent" (weight exactly 0) and "present once" (weight exactly 1).

The intuition behind log scaling. The log function captures diminishing returns mathematically. Moving from 0 to 1 occurrence is the most important jump: from absent to present. Moving from 1 to 10 occurrences is meaningful but less dramatic. Moving from 100 to 1000 occurrences adds barely any signal. The log function maps these ratios to equal intervals:

\begin{aligned} \log(1) &= 0 \\ \log(10) &\approx 2.3 \\ \log(100) &\approx 4.6 \\ \log(1000) &\approx 6.9 \end{aligned}

Each tenfold increase in count adds roughly 2.3 to the weight, rather than multiplying it by 10. This reflects how human language behaves: an author who mentions "photosynthesis" 20 times is not necessarily ten times more focused on the topic than one who mentions it twice. Both documents are clearly about photosynthesis; the high count in the first document likely reflects a longer article or a different writing style rather than ten times more relevance.

The $+1$ in the log formula is not arbitrary. Without it, a term appearing exactly once has log-weight of 0, which is the same as a term that doesn't appear at all. That would mean a document mentioning "quantum" once is treated identically to a document that never mentions quantum at all, which is clearly wrong. The $+1$ shift lifts all present terms above zero while still compressing the high-count range.

Why Natural Log vs Log Base 2 vs Log Base 10?

The choice of logarithm base does not change the relative ordering of term weights. It only rescales all weights by a constant factor (since $\log_b(x) = \log(x) / \log(b)$ ). In practice, natural log ( $\ln$ ) is most common in implementations. Log base 2 is theoretically meaningful in information-theoretic contexts where it measures bits of information. Log base 10 gives weights that are easy to reason about in decimal terms. scikit-learn uses $\log(\text{count}) + 1$ with natural log in its TF-IDF implementation.

When to use log TF: Log scaling is the most commonly used variant in information retrieval and is the default in most TF-IDF implementations. It's a good default choice whenever you want to weight by frequency but reduce the influence of very high counts. For most classification and retrieval tasks, log TF is a better starting point than raw TF.