Term Frequency › Summary

Term frequency weighting converts raw word counts into document representations that are more analytically useful. The key variants, each encoding different assumptions about what frequency means:

• Raw TF is the simplest form, directly using word counts. It suffers from length bias and gives disproportionate weight to high-frequency terms.
• Log TF applies logarithmic compression to model diminishing returns: going from 0 to 1 occurrence matters most; going from 100 to 101 barely matters at all.
• Boolean TF discards all count information and records only presence or absence, which is surprisingly effective when counts are mostly 1 or when topic presence matters more than emphasis.
• Augmented TF normalizes each term's count by the maximum count in the document, making the representation length-invariant and capturing relative emphasis within a document.
• L2-normalized TF places document vectors on the unit sphere, enabling cosine similarity to be computed as a simple dot product and making the representation independent of document length.

In practice, log TF + L2 normalization is the most common choice and is the default behavior of scikit-learn's TfidfVectorizer before IDF is applied.

TF representations are inherently sparse: most documents use only a small fraction of the total vocabulary, and term-document matrices have 95-99%+ sparsity in practice. This sparsity follows directly from Zipf's law: the highly skewed distribution of word frequencies in natural language means that most vocabulary items appear rarely and thus produce mostly-zero columns in any practical corpus. Storing these matrices in sparse formats (like CSR) provides dramatic memory savings and computational speedups compared to dense storage.

The geometry of TF-weighted document spaces matters for downstream tasks. Raw TF places documents at varying distances from the origin based on their length; L2 normalization projects everything onto the unit sphere so that cosine similarity equals the dot product. Log TF reshapes the distribution within that sphere, pulling content words toward more equal influence with stopwords and making topical signals more visible.

Finally, TF weighting alone is insufficient for discriminative retrieval or classification because it doesn't account for a term's importance across the corpus. A word that appears frequently in one document tells you about that document's focus, but if that same word appears frequently in every document, it's not a useful discriminator. The next chapter introduces Inverse Document Frequency, which provides the missing global signal, and together they form the TF-IDF representations that remain foundational to text analysis and information retrieval.