Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Term Frequency › Choosing a TF Variant

Michael BrenndoerferMarch 30, 20252 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 11Term Frequency › Choosing a TF Variant

Reading section 14 of 17 · Choosing a TF Variant · 2 minSection 14 of 17 · 2 min

Section 14 of 17 · 2 min read

Term Frequency › Choosing a TF VariantOne section · Sections in this chapter14 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Rather than prescribing a single "best" variant, it helps to think about what assumption each variant makes about your data and task.

The table below summarizes the key properties:

TF weighting variants, their formulas, key assumptions, and recommended use cases.
Variant	Formula	Key Assumption	Best For
Raw	$\text{count}(t,d)$	More occurrences = proportionally more important	Fixed-length documents; absolute frequency matters
Log	$1 + \log(\text{count})$	Diminishing returns to repetition	General-purpose IR; variable length documents
Boolean	$\mathbb{1}[\text{count} > 0]$	Presence/absence matters; not quantity	Short texts; topic detection; binary classifiers
Augmented	$K + (1-K) \cdot \frac{\text{count}}{\max_{t'} \text{count}}$	Relative frequency within document matters	Authorship attribution; cross-document comparison
L2 Norm	$\hat{\mathbf{v}} = \mathbf{v}/\\|\mathbf{v}\\|_2$	Cosine similarity is the right distance metric	Cosine-based retrieval; neural network inputs

These variants are often combined: log TF followed by L2 normalization (the scikit-learn default for TfidfVectorizer(use_idf=False)) is the most common choice in practice. The log step addresses diminishing returns and the L2 step addresses length bias, so you get both benefits at once.

A few practical guidelines:

For most classification tasks: Start with log TF + L2 normalization. It's the most principled default and what scikit-learn gives you out of the box.
For short documents (tweets, titles, product names): Boolean TF often works just as well as log TF and is simpler to reason about.
For authorship attribution or style analysis: Augmented TF is worth trying because it captures within-document relative frequency patterns.
For absolute frequency tasks (keyword counting, anomaly detection in logs): Raw TF is appropriate and adds no distortion.
Before feeding into a neural network: Always normalize. Dense embedding models expect unit-variance or unit-norm inputs. L2 normalization is a safe default.