Term Frequency › From Counts to Weights

As we saw in the Bag of Words chapter, a corpus of $N$ documents over a vocabulary $V$ can be represented as a matrix $\mathbf{M} \in \mathbb{R}^{N \times |V|}$ where each entry $M_{d,t}$ holds the count of term $t$ in document $d$. This is called the term-document matrix. Raw counts are a natural starting point, but they come with two fundamental problems that make them unsuitable for most downstream tasks.

These problems are not obscure edge cases. They arise whenever documents in your corpus vary in length (which is almost always), and whenever some terms appear many times in a document (which is common). Before diving into the specific weighting formulas, it's worth understanding both problems clearly, because each TF variant is essentially a different strategy for solving one or both of them.

Suppose you have two documents:

• Document A (100 words): the word "climate" appears 5 times.
• Document B (1000 words): the word "climate" appears 5 times.

Both have the same raw count, but Document A is clearly much more focused on climate: 5% of its words are "climate" versus 0.5% in Document B. If you use raw counts directly in a retrieval or classification system, longer documents receive an artificial advantage simply because they have more opportunities to accumulate counts for any given term. A 10,000-word encyclopedia article will score higher than a focused 500-word summary on nearly every topic, even if the summary is more relevant to the query.

This is the length bias problem, and it motivates every normalization strategy in this chapter. The goal is to represent "how much does this document talk about topic X" in a way that is independent of how many words the document contains overall. That requires some form of normalization, whether dividing by total document length, by the maximum count in the document, or by the Euclidean norm of the count vector.

Length bias is asymmetric: it penalizes short documents and rewards long ones regardless of their relevance. In a retrieval system, this means a verbose document that tangentially mentions your query keyword many times will systematically outscore a short, highly focused document. For classification, length bias causes models to pick up on document length as a spurious feature instead of learning topical content signals. Normalizing term frequency is the primary tool for removing this bias.

Now suppose you're comparing two documents where one mentions "neural" 2 times and another mentions "neural" 20 times. Does the second document talk about neural networks ten times more than the first? Probably not. Human writing doesn't scale linearly: an author who really cares about a topic will mention it frequently, but mentioning it 20 times is not ten times more meaningful than mentioning it twice. There are diminishing returns to repetition.

Think about how you write. When you write a paper about transformers, you'll mention "transformer" in the introduction, in each section header, when introducing a key idea, and in the conclusion. The count accumulates, but the informational content of each additional mention decreases. The first mention establishes the topic; subsequent mentions reinforce it. The 50th mention adds negligible new information beyond what the 10th mention conveyed.

This is the diminishing returns problem, and it motivates logarithmic and augmented frequency variants. The key insight is that what matters is whether a term is mentioned "a little" versus "a lot," not the precise count. A logarithmic function captures exactly this: it maps large multiplicative differences in count to small additive differences in weight.

Throughout this chapter we use the following notation consistently:

• $t$: a term (word or token)
• $d$: a document
• $\text{count}(t, d)$: the raw count of term $t$ in document $d$
• $\text{tf}(t, d)$: the term frequency weight (specific formula depends on the variant)
• $|d|$: the total number of tokens in document $d$
• $V$: the vocabulary (set of all unique terms in the corpus)
• $\max_{t'} \text{count}(t', d)$: the count of the most frequent term in document $d$