Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

TF-IDF › TF-IDF Normalization

Michael BrenndoerferUpdated March 21, 20264 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 13TF-IDF › TF-IDF Normalization

Reading section 5 of 13 · TF-IDF Normalization · 4 minSection 5 of 13 · 4 min

Section 5 of 13 · 4 min read

TF-IDF › TF-IDF NormalizationOne section · Sections in this chapter5 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

After computing TF-IDF scores, the resulting document vectors have different magnitudes. A long document will naturally accumulate more non-zero TF-IDF scores than a short one. If you compute the dot product between a long document vector and a short document vector, the longer document will tend to win simply by virtue of having more non-zero entries, even if both documents discuss the same topics. Before computing similarities or using vectors as classifier features, normalization removes this length bias.

L2 NormalizationLink Copied

The standard approach is L2 (Euclidean) normalization, which rescales each document vector to unit length:

\hat{\mathbf{v}}_d = \frac{\mathbf{v}_d}{\|\mathbf{v}_d\|_2}

where:

$\mathbf{v}_d$ : the TF-IDF vector for document $d$ (length $V$ , one entry per vocabulary term)
$\|\mathbf{v}_d\|_2 = \sqrt{\sum_{t=1}^{V} \text{tfidf}(t, d)^2}$ : the L2 norm (Euclidean length) of the vector
$\hat{\mathbf{v}}_d$ : the normalized vector, lying on the unit hypersphere

After L2 normalization, all document vectors have length 1. They differ only in direction, not in magnitude. The cosine similarity between two documents becomes simply the dot product of their normalized vectors:

\text{cosine\_sim}(\hat{\mathbf{u}}, \hat{\mathbf{v}}) = \hat{\mathbf{u}} \cdot \hat{\mathbf{v}} = \sum_{t=1}^{V} \hat{u}_t \cdot \hat{v}_t

This simplification is computationally significant. Cosine similarity without normalization requires computing each vector's magnitude separately (two square-root operations per pair of documents). With pre-normalized vectors, computing similarity between all pairs in a corpus of $N$ documents reduces to a single matrix multiplication: $\hat{M} \hat{M}^T$ , where $\hat{M}$ is the matrix of normalized row vectors. This is highly optimized in NumPy and sparse matrix libraries.

L1 NormalizationLink Copied

L1 normalization divides by the sum of absolute values:

\hat{\mathbf{v}}_d = \frac{\mathbf{v}_d}{\|\mathbf{v}_d\|_1} = \frac{\mathbf{v}_d}{\sum_{t=1}^{V} |\text{tfidf}(t, d)|}

L1-normalized vectors interpret each entry as a fraction of the total TF-IDF weight assigned to the document. They have a probabilistic flavor: each entry represents the proportion of discriminative weight that flows through a given term. They're less common than L2 normalization in practice, but useful when you want to reason about the relative contribution of different terms to a document's representation.

In[12]:

Code

def l2_normalize(matrix):
    """L2-normalize each row (document vector) to unit length."""
    norms = np.linalg.norm(matrix, axis=1, keepdims=True)
    norms[norms == 0] = 1  # Avoid division by zero for zero vectors
    return matrix / norms


def l1_normalize(matrix):
    """L1-normalize each row to sum to 1."""
    norms = np.abs(matrix).sum(axis=1, keepdims=True)
    norms[norms == 0] = 1
    return matrix / norms


tfidf_l2 = l2_normalize(tfidf_matrix)
tfidf_l1 = l1_normalize(tfidf_matrix)

Out[13]:

Console

Before normalization:
  Doc 1 vector norm (L2): 4.9801
  Doc 5 vector norm (L2): 4.3420

After L2 normalization:
  Doc 1 vector norm (L2): 1.0000
  Doc 5 vector norm (L2): 1.0000

After L1 normalization:
  Doc 1 vector L1 sum: 1.0000
  Doc 5 vector L1 sum: 1.0000

After L2 normalization, all document vectors have unit length (norm = 1). The vectors for documents of very different lengths are now comparable: they point in directions determined by their content, not by their size. Documents discussing the same topic but at different lengths will still score high similarity because their vectors point in the same direction, even if their original magnitudes differed by an order of magnitude.

When Normalization MattersLink Copied

Normalization matters most when comparing documents of substantially different lengths. In a corpus of news articles where pieces range from 200 to 2,000 words, skipping normalization would cause long articles to systematically outscore shorter ones in similarity comparisons. This is the key reason L2 normalization is on by default in scikit-learn.

There are edge cases where you might want to skip normalization. If you're using TF-IDF features as input to a machine learning model that includes document length as a separate feature, normalizing TF-IDF would destroy the length signal you're trying to preserve. Similarly, if you're ranking documents for a specific query and longer, more detailed documents are preferred over shorter ones, raw (unnormalized) TF-IDF scores might better reflect that preference. These are exceptions to the rule, but understanding why normalization is done helps you decide when to deviate from the default.