Term Frequency: Weighting Schemes for Text Analysis

You've seen how the Bag of Words model counts word occurrences and the n-gram model captures local word sequences. But not all word counts are created equal. A word that appears 100 times in a document tells a very different story depending on whether the document is 200 words long or 20,000 words long. And a word appearing twice differs from one appearing once, but is it twice as important? These questions sit at the heart of term frequency weighting.

Term frequency is the simplest and most intuitive way to transform raw word counts into something more analytically useful. The idea is straightforward: words that appear more often in a document are more representative of that document's content. But "more often" hides a surprising amount of nuance. Raw counts, logarithmic scaling, boolean presence, augmented normalization, and L2 normalization all encode different assumptions about what "frequency" means and how it should influence downstream comparisons between documents. Understanding each variant, when to use it, and why it was designed the way it was will equip you to make principled decisions about text representation.

This chapter covers every major variant of term frequency weighting, the mathematical intuitions behind each, and how sparsity shapes the practical efficiency of these representations. It also grounds the discussion in real-world data, so you leave with an understanding of what TF distributions look like in practice and how different weighting choices affect the geometry of the resulting document space.

As we saw in the Bag of Words chapter, a corpus of $N$ documents over a vocabulary $V$ can be represented as a matrix $\mathbf{M} \in \mathbb{R}^{N \times |V|}$ where each entry $M_{d,t}$ holds the count of term $t$ in document $d$. This is called the term-document matrix. Raw counts are a natural starting point, but they come with two fundamental problems that make them unsuitable for most downstream tasks.

These problems are not obscure edge cases. They arise whenever documents in your corpus vary in length (which is almost always), and whenever some terms appear many times in a document (which is common). Before diving into the specific weighting formulas, it's worth understanding both problems clearly, because each TF variant is essentially a different strategy for solving one or both of them.

Suppose you have two documents:

• Document A (100 words): the word "climate" appears 5 times.
• Document B (1000 words): the word "climate" appears 5 times.

Both have the same raw count, but Document A is clearly much more focused on climate: 5% of its words are "climate" versus 0.5% in Document B. If you use raw counts directly in a retrieval or classification system, longer documents receive an artificial advantage simply because they have more opportunities to accumulate counts for any given term. A 10,000-word encyclopedia article will score higher than a focused 500-word summary on nearly every topic, even if the summary is more relevant to the query.

This is the length bias problem, and it motivates every normalization strategy in this chapter. The goal is to represent "how much does this document talk about topic X" in a way that is independent of how many words the document contains overall. That requires some form of normalization, whether dividing by total document length, by the maximum count in the document, or by the Euclidean norm of the count vector.

Length bias is asymmetric: it penalizes short documents and rewards long ones regardless of their relevance. In a retrieval system, this means a verbose document that tangentially mentions your query keyword many times will systematically outscore a short, highly focused document. For classification, length bias causes models to pick up on document length as a spurious feature instead of learning topical content signals. Normalizing term frequency is the primary tool for removing this bias.

Now suppose you're comparing two documents where one mentions "neural" 2 times and another mentions "neural" 20 times. Does the second document talk about neural networks ten times more than the first? Probably not. Human writing doesn't scale linearly: an author who really cares about a topic will mention it frequently, but mentioning it 20 times is not ten times more meaningful than mentioning it twice. There are diminishing returns to repetition.

Think about how you write. When you write a paper about transformers, you'll mention "transformer" in the introduction, in each section header, when introducing a key idea, and in the conclusion. The count accumulates, but the informational content of each additional mention decreases. The first mention establishes the topic; subsequent mentions reinforce it. The 50th mention adds negligible new information beyond what the 10th mention conveyed.

This is the diminishing returns problem, and it motivates logarithmic and augmented frequency variants. The key insight is that what matters is whether a term is mentioned "a little" versus "a lot," not the precise count. A logarithmic function captures exactly this: it maps large multiplicative differences in count to small additive differences in weight.

Throughout this chapter we use the following notation consistently:

• $t$: a term (word or token)
• $d$: a document
• $\text{count}(t, d)$: the raw count of term $t$ in document $d$
• $\text{tf}(t, d)$: the term frequency weight (specific formula depends on the variant)
• $|d|$: the total number of tokens in document $d$
• $V$: the vocabulary (set of all unique terms in the corpus)
• $\max_{t'} \text{count}(t', d)$: the count of the most frequent term in document $d$

The simplest possible definition of term frequency uses the raw count directly. Raw TF answers the question: "How many times does this term appear in this document?"

where:

• $\text{tf}_{\text{raw}}(t, d)$: the raw term frequency weight for term $t$ in document $d$
• $\text{count}(t, d)$: the number of times term $t$ appears in document $d$

This is exactly what you get from the Bag of Words vectorizer: each entry in the document vector is the number of times the corresponding word appears in the document.

Raw counts are intuitive and fast to compute. They preserve the full information in the term occurrences, which means they can distinguish between a document that mentions "algorithm" twice and one that mentions it twenty times. For some applications, this direct proportionality is exactly what you want. Imagine counting how many times a legal document mentions a specific clause or how many times a customer review mentions "refund." In those cases, absolute counts are the signal.

When to use it: Raw counts are appropriate when your downstream task is insensitive to document length, or when all documents in your corpus are roughly the same length. They're also a natural fit when you want to weight by absolute frequency, such as counting how many times a specific keyword appears in a legal contract to flag potential issues, or when building features for a system where document length is itself a signal you want to preserve.

Limitation: As described above, raw counts suffer from length bias. They also allow very high-frequency terms to dominate the representation. The word "the" might appear 50 times in an essay, drowning out more meaningful terms that appear only a handful of times. Without normalization, your document vectors will be heavily influenced by stopwords and will conflate short focused documents with long discursive ones.

To address diminishing returns, logarithmic scaling compresses the count scale. The key insight is that each successive doubling of count should add a fixed increment of importance, not double it.

where:

• $\text{tf}_{\text{log}}(t, d)$: the log-scaled term frequency weight
• $\log(\cdot)$: the natural logarithm (base $e$), though other bases work equally well since log base conversion is just a constant scale factor
• The $+1$ offset ensures that a term appearing exactly once receives weight 1, making the zero-versus-nonzero distinction crisp. Without the $+1$, a term appearing once would receive weight $\log(1) = 0$, which would be indistinguishable from an absent term.

Some formulations instead use $\log(1 + \text{count}(t, d))$, which maps count 0 to 0 continuously without needing a piecewise definition. The two forms give very similar results in practice, but the piecewise version is more common in information retrieval literature because it preserves the conceptually clear distinction between "absent" (weight exactly 0) and "present once" (weight exactly 1).

The intuition behind log scaling. The log function captures diminishing returns mathematically. Moving from 0 to 1 occurrence is the most important jump: from absent to present. Moving from 1 to 10 occurrences is meaningful but less dramatic. Moving from 100 to 1000 occurrences adds barely any signal. The log function maps these ratios to equal intervals:

Each tenfold increase in count adds roughly 2.3 to the weight, rather than multiplying it by 10. This reflects how human language behaves: an author who mentions "photosynthesis" 20 times is not necessarily ten times more focused on the topic than one who mentions it twice. Both documents are clearly about photosynthesis; the high count in the first document likely reflects a longer article or a different writing style rather than ten times more relevance.

The $+1$ in the log formula is not arbitrary. Without it, a term appearing exactly once has log-weight of 0, which is the same as a term that doesn't appear at all. That would mean a document mentioning "quantum" once is treated identically to a document that never mentions quantum at all, which is clearly wrong. The $+1$ shift lifts all present terms above zero while still compressing the high-count range.

When to use log TF: Log scaling is the most commonly used variant in information retrieval and is the default in most TF-IDF implementations. It's a good default choice whenever you want to weight by frequency but reduce the influence of very high counts. For most classification and retrieval tasks, log TF is a better starting point than raw TF.

At the extreme end of diminishing returns, you can ignore count entirely and just record whether a term is present or absent:

Boolean weighting discards all frequency information. Two documents where "economy" appears once and 50 times respectively receive identical representations for that term. This might seem like throwing away useful information, but in many practical situations it's exactly the right choice.

Consider a dataset of tweets. A tweet is typically 20 to 30 words long. In such a short document, most meaningful words appear exactly once. The difference between a count of 1 and a count of 2 for "election" in a 25-word tweet is dominated by noise: the author may have rephrased a sentence or mentioned the word in passing. Boolean representation treats these as identical, which is appropriate. In contrast, for a 1000-word news article, the difference between mentioning "election" twice and ten times is informative and worth preserving.

Boolean weighting also reduces the influence of verbose writing styles. Some authors repeat key terms frequently as a rhetorical device, while others state a concept once and move on. Boolean representation treats these authors as equally focused on a topic, which is often the right behavior for topic classification.

When to use boolean TF: Boolean weighting is surprisingly effective for tasks where the presence or absence of a concept matters more than how much it's discussed. Short documents like tweets or product titles often have counts of 1 or 2 for any non-stopword, so count differences are uninformative. Boolean features are also useful when working with binary classifiers or when you want to reduce the effect of verbose documents that repeat key terms excessively. In binary text classification benchmarks, boolean features sometimes outperform raw counts because models trained on boolean features are less susceptible to the length bias problem.

The Bag of Words chapter introduced binary vs count representations. Boolean TF is the same as binary BoW, with the only distinction being a matter of framing: BoW as a representation versus TF as a weighting scheme applied to counts.

A clever variant designed specifically to address length bias without discarding frequency information is augmented term frequency. Instead of normalizing by total document length, it normalizes each term's count by the count of the most frequent term in the same document:

where:

• $K \in [0, 1]$: a floor parameter, typically set to 0.5, that controls the minimum weight given to any present term
• $\text{count}(t, d)$: the raw count of term $t$ in document $d$
• $\max_{t'} \text{count}(t', d)$: the count of the most frequent term in document $d$, used as a normalizing constant

The result always lies in $[K, 1]$:

• The most frequent term in any document always receives weight exactly 1.
• A term appearing half as often as the most frequent term receives weight $K + (1-K) \cdot 0.5 = 0.75$ when $K=0.5$.
• Terms with very low counts receive weight close to $K$.
• Absent terms (count = 0) are excluded from the representation and their entries remain 0.

Why the floor $K$? Without the floor, a term appearing once while the maximum count is 1000 would receive near-zero weight ($\approx 0.001$), effectively being treated as absent. The floor $K = 0.5$ ensures that even rare terms receive at least half the weight of the most frequent term, preventing the representation from being dominated entirely by the top few terms. The choice $K = 0.5$ is conventional but not universal; smaller $K$ values give more relative weight to the most frequent term, while larger $K$ values compress all present terms closer together.

Why max normalization instead of total-length normalization? Normalizing by the maximum count within the document makes the representation document-length invariant in an intuitive way: what matters is not how many times you said "climate" in absolute terms, but how often you said "climate" relative to whatever you talked about most. This is particularly useful in authorship attribution and stylometric analysis, where the relative usage patterns of words reveal more about an author's style than absolute counts. Two authors writing on the same topic for the same publication might produce documents of similar length, but their stylistic choices about how often to repeat key terms differ, and augmented TF captures those differences cleanly.

The max normalization also has a nice property for corpus-wide comparisons. Consider documents in different genres: a news article and an academic paper may cover the same topic but with very different writing densities. Normalizing by max count within each document makes the frequency signals comparable across genres, because the normalization is always relative to the author's own most-used term.

When to use augmented TF: It's widely used in information retrieval applications where document length varies significantly, and it was historically popular in the SMART notation system, developed at Cornell University in the 1960s and 70s. SMART used letter codes to describe IR weighting schemes: 'a' for augmented, 'n' for natural (raw), 'l' for logarithmic, and so on. SMART's systematic taxonomy of weighting options directly inspired modern TF-IDF implementations, and understanding it helps explain why the field settled on the specific formulas it did. In practice, L2 normalization has largely supplanted augmented TF for machine learning applications because of its better theoretical properties, but augmented TF remains relevant for understanding the design space and for applications where per-document relative emphasis is the primary signal of interest.

The most mathematically principled normalization strategy places every document vector on the unit sphere, making length-invariant comparison a simple dot product.

To normalize a vector to unit length, divide it by its Euclidean norm:

where:

• $\mathbf{v}_d \in \mathbb{R}^{|V|}$: the raw (or log-scaled) term frequency vector for document $d$, with one entry per vocabulary term
• $\|\mathbf{v}_d\|_2$: the Euclidean (L2) norm of the vector, computed as

• $\hat{\mathbf{v}}_d$: the L2-normalized document vector, which satisfies $\|\hat{\mathbf{v}}_d\|_2 = 1$

After L2 normalization, every document vector lies on the unit hypersphere in $\mathbb{R}^{|V|}$.

Why L2 normalization? Two key properties make it highly useful.

First, length invariance: if you double all counts in a document (as would happen if you copy-pasted it twice), the unnormalized vector doubles, but the normalized vector remains unchanged. The representation captures relative word proportions, not absolute volumes. This property is essential for fair comparison across documents of different lengths.

Second, cosine similarity simplification: the cosine similarity between two documents measures the angle between their vectors:

When both vectors are already L2-normalized (so $\|\hat{\mathbf{v}}_d\|_2 = \|\hat{\mathbf{v}}_{d'}\|_2 = 1$), this simplifies to:

The division by norms disappears entirely. This is extremely convenient: once vectors are normalized, you can compute document similarity with a single dot product, which is highly optimized in numerical libraries. For a retrieval system comparing a query against millions of documents, this simplification translates directly into speed.

L2 normalization vs. L1 normalization. An alternative is L1 normalization, which divides by the sum of absolute values rather than the Euclidean norm:

L1 normalization produces a probability distribution over terms: each entry $\hat{v}_{d,t} = \text{count}(t, d) / |d|$ is the fraction of document $d$'s words that are term $t$. This is equivalent to dividing by document length and gives the relative frequency interpretation. L1 normalization is more interpretable (each weight is literally a proportion), while L2 normalization gives better geometric properties for similarity computation. Both are valid; the choice depends on whether you prioritize interpretability or computational convenience.

Let's work through all five variants on a small example to build concrete intuition. Consider two documents:

• Doc A: "the cat sat on the mat the cat" (8 tokens)
• Doc B: "the dog chased the cat across the yard the dog" (10 tokens)

The vocabulary is: {across, cat, chased, dog, mat, on, sat, the, yard}

Raw counts:

Applying each weighting scheme to the term "the":

• Raw: Doc A = 3, Doc B = 4
• Log: Doc A = $1 + \ln(3) \approx 2.10$, Doc B = $1 + \ln(4) \approx 2.39$
• Boolean: Doc A = 1, Doc B = 1 (both contain "the")
• Augmented ($K=0.5$): In Doc A, the maximum count is 3 (for "the"), so $\text{tf}_{\text{aug}} = 0.5 + 0.5 \cdot (3/3) = 1.0$. In Doc B, the maximum count is 4 (also "the"), so $\text{tf}_{\text{aug}} = 0.5 + 0.5 \cdot (4/4) = 1.0$. Both get weight 1.0 since "the" is the dominant term in both.
• L2 normalization: Divide each raw count vector by its Euclidean norm before comparison.

Notice how augmented TF reveals that "the" dominates both documents equally (weight 1.0 in both), while "cat" gets weight $0.5 + 0.5 \cdot (2/3) \approx 0.833$ in Doc A and $0.5 + 0.5 \cdot (1/4) = 0.625$ in Doc B, correctly showing that "cat" is proportionally more important in Doc A.

Now let's compute the L2 norms for these two documents:

After normalization, the "cat" entry in Doc A becomes $2/4 = 0.5$, and in Doc B it becomes $1/4.90 \approx 0.204$. So the L2 representation also correctly reflects that "cat" plays a larger role in Doc A than in Doc B, and does so without requiring any free parameter like $K$.

This worked example illustrates a key insight: all five variants agree qualitatively on the relative importance of terms within each document, but they differ in how much weight they assign to high-count terms like "the" versus low-count terms like "sat." The choice of variant affects the geometry of the document space, which in turn affects the behavior of any downstream classifier or retrieval system.

Let's implement all five variants from scratch, then compare them to scikit-learn's output to verify our understanding.

First, set up our sample corpus and compute raw counts.

The implementations agree, which confirms that scikit-learn's TfidfVectorizer(use_idf=False, sublinear_tf=True, norm='l2') computes exactly what we've described as log TF followed by L2 normalization.

One of the most important practical characteristics of term frequency matrices is their sparsity: most entries are zero. For a vocabulary of 50,000 terms and documents of average length 200 words, a typical document uses at most a few hundred distinct terms, so at least 49,800 out of 50,000 entries are zero. That's 99.6% sparsity.

This sparsity affects storage and computation as well as modeling. It is a fundamental property of language. Human vocabulary is enormous, but any individual document engages with only a tiny slice of it. A sports article uses sports vocabulary; a medical paper uses medical terminology. The overlap between their vocabularies is small, which means their term-document vectors are nearly orthogonal and most entries in each vector are zero.

Consider the storage implications concretely. A corpus of 100,000 documents with vocabulary size 100,000 would require $100{,}000 \times 100{,}000 \times 8$ bytes = 80 GB as a dense float64 matrix. That's untenable for most systems. The same data stored in Compressed Sparse Row (CSR) format might occupy only a few hundred megabytes, since only non-zero entries and their locations need to be stored. At 99.5% sparsity, the sparse representation is 200 times smaller.

The computational benefits are equally significant:

• Storage: Dense matrices waste enormous memory on zeros. Sparse matrix formats store only non-zero values and their positions, giving orders-of-magnitude memory savings.
• Computation: Sparse matrix operations skip zero-valued entries entirely. Computing cosine similarities between a query and 1 million documents is tractable with sparse vectors but would be prohibitive with dense ones.
• Modeling: Many later models (linear SVMs, logistic regression, naive Bayes) work directly on sparse representations. Neural networks require dense inputs, which is why text is typically projected to dense embeddings in later stages.

Let's examine how sparsity behaves across different vocabulary sizes.

Notice that increasing min_df dramatically reduces vocabulary size (filtering out words that appear in fewer than 20 documents cuts vocabulary by more than 80%), but sparsity decreases only modestly. This is because rare words, while numerous, contribute very few non-zero entries to the matrix. The sparsity is driven by the structure of language itself: any given document uses only a tiny fraction of the vocabulary, regardless of how large that vocabulary is.

The extreme sparsity of term frequency matrices is not coincidental. It follows directly from Zipf's law, which describes a universal pattern in natural language: the frequency of any word is roughly inversely proportional to its rank in the frequency table. The most common word appears about twice as often as the second most common word, three times as often as the third most common, and so on.

Zipf's law means that a small number of words (mostly function words like "the," "a," "of," "and") account for a large fraction of all word occurrences, while the vast majority of vocabulary items are rare. In a typical English corpus, the top 100 most frequent words account for roughly 50% of all word tokens, while the bottom 50% of vocabulary items appear only once or twice. Words that appear only once in the entire corpus are called hapax legomena (Greek for "said only once"), and they typically constitute 40-60% of the vocabulary in large corpora.

This distribution has direct implications for term frequency matrices. Because most words are rare, most vocabulary entries will be zero in any given document. The sparsity you observed above (98%+) is not an artifact of the particular corpus or vocabulary size, it's a mathematical consequence of Zipf's law applied to any large natural language corpus.

The extreme skewness of the raw TF distribution (most non-zero entries are 1 or 2) explains why log TF and boolean TF often perform similarly in practice. When the count is 1 for the majority of terms in a document, the distinction between $\text{tf}_{\text{log}} = 1 + \log(1) = 1$ and $\text{tf}_{\text{bool}} = 1$ vanishes. Log TF only provides a different result from boolean TF for terms that appear multiple times, which is the minority of term-document pairs.

This insight has a practical implication: if you're working with a corpus where most documents are short (tweets, product titles, abstracts), boolean TF may be sufficient. The extra complexity of log TF only pays off when your documents are long enough for terms to accumulate meaningful multi-occurrence counts. For news articles, academic papers, or books, log TF is clearly preferable. For social media data, the simpler boolean representation often works just as well.

When working with large corpora, computing and storing term frequency matrices efficiently requires more than just calling .fit_transform() and hoping for the best.

scikit-learn returns term frequency matrices as Compressed Sparse Row (CSR) matrices, which store only non-zero values and their positions. A CSR matrix consists of three arrays:

• .data: the non-zero values
• .indices: the column index of each non-zero value
• .indptr: pointers indicating where each row's non-zeros start in .data

Understanding the CSR format helps you reason about when sparse operations are fast and when they may degrade to dense performance. Operations that iterate over all entries of a row are efficient in CSR format. Operations that require column-wise access (like accessing all documents that contain a specific term) are slower because the column indices are not contiguous in memory, which is why CSC (Compressed Sparse Column) format is sometimes preferred for column-oriented access patterns.

For large matrices, computing L2 norms row-by-row in Python is slow. scikit-learn provides sklearn.preprocessing.normalize which calls optimized C routines and operates directly on sparse matrices without expanding them to dense form:

One important detail: when you apply log scaling by modifying X_log.data directly, you're operating only on non-zero values, because .data contains only the non-zero entries. This is both efficient and correct: the zeros in the matrix represent absent terms, and they should remain zero after log scaling. If you naively applied 1 + np.log(X_full_matrix) to a dense matrix, you'd compute $1 + \log(0) = -\infty$ for all the absent terms, which is wrong. Working with the sparse .data array avoids this pitfall automatically.

Let's time the full pipeline for the main TF variants at scale.

The raw count construction step (tokenizing text and counting terms before building the sparse matrix) dominates the total runtime. All subsequent transformations (log scaling, boolean conversion, L2 normalization) are fast because they operate only on the non-zero entries stored in .data, which is a small fraction of the full matrix. This reinforces the key practical insight: the bottleneck in a TF pipeline is almost always the initial vectorization, not the weighting scheme.

Understanding what term frequency distributions look like in practice is as important as understanding the weighting formulas. Let's visualize the distribution of raw term frequencies in a real corpus.

The side-by-side heatmaps make the effect of normalization concrete. In the raw count matrix, "the" dominates both Doc A and Doc B with values of 3 and 4, while content words like "cat," "dog," and "patterns" are barely visible. After log scaling and L2 normalization, the weights spread more evenly across terms: "the" no longer overwhelms everything else, and the distinctive content of each document becomes more apparent in the representation. Doc A becomes characterized by roughly equal weights for "cat," "sat," and "the"; Doc C shows "patterns" as a distinctively high-weight term relative to the other terms.

It's worth pausing to think about what these weighting schemes are doing to the geometry of the document space, because the geometric picture connects TF weighting to the behavior of downstream classifiers and retrieval systems.

Without any normalization, documents are represented as points in $\mathbb{R}^{|V|}$ where the distance from the origin reflects document length. Two documents about the same topic but of different lengths will be far apart in the raw TF space simply because one vector has larger magnitude. A nearest-neighbor classifier working in this space would be strongly influenced by document length, which is rarely the right inductive bias.

After L2 normalization, all documents are projected onto the unit hypersphere. The Euclidean distance between two unit vectors is $\|\hat{\mathbf{v}}_d - \hat{\mathbf{v}}_{d'}\|_2 = \sqrt{2 - 2\cos\theta}$, where $\theta$ is the angle between them. This means Euclidean distance on the unit sphere is a monotone function of cosine similarity: minimizing Euclidean distance is equivalent to maximizing cosine similarity. You get the same nearest neighbors whether you use Euclidean distance or cosine similarity, as long as all vectors are L2-normalized.

This geometric perspective also clarifies why cosine similarity is the natural similarity metric for text. Euclidean distance in the raw count space is sensitive to document length, while cosine similarity is not. By normalizing to the unit sphere, you bring all documents to the same "radius" and make Euclidean distance in the normalized space equivalent to the angle-based similarity that ignores magnitude differences.

The log transformation changes the shape of the unit sphere's "population." Without log scaling, long documents with high counts cluster near the "stopword directions" in the high-dimensional space, because high-frequency function words dominate the magnitude. With log scaling, the high-count advantage of stopwords is compressed, and content words with lower but meaningful counts contribute more to the direction of the vector. Log normalization effectively rotates documents in the high-dimensional space to align more with their topical content.

Rather than prescribing a single "best" variant, it helps to think about what assumption each variant makes about your data and task.

The table below summarizes the key properties:

These variants are often combined: log TF followed by L2 normalization (the scikit-learn default for TfidfVectorizer(use_idf=False)) is the most common choice in practice. The log step addresses diminishing returns and the L2 step addresses length bias, so you get both benefits at once.

A few practical guidelines:

• For most classification tasks: Start with log TF + L2 normalization. It's the most principled default and what scikit-learn gives you out of the box.
• For short documents (tweets, titles, product names): Boolean TF often works just as well as log TF and is simpler to reason about.
• For authorship attribution or style analysis: Augmented TF is worth trying because it captures within-document relative frequency patterns.
• For absolute frequency tasks (keyword counting, anomaly detection in logs): Raw TF is appropriate and adds no distortion.
• Before feeding into a neural network: Always normalize. Dense embedding models expect unit-variance or unit-norm inputs. L2 normalization is a safe default.

Term frequency weighting is powerful but incomplete. The central limitation is that TF alone treats all words as equally discriminative. A word like "the" might appear 50 times in a document, but this tells you nothing about what the document is about because "the" appears frequently in every document. Meanwhile, a technical term like "photosynthesis" appearing even twice is a strong signal that the document concerns biology. TF alone cannot distinguish between these cases, because it has no mechanism for comparing a term's frequency in a single document against its frequency across the corpus.

This gap is precisely why TF is almost never used in isolation for information retrieval or classification tasks. The next chapter introduces Inverse Document Frequency (IDF), which weights terms by their rarity across the corpus. The combination, TF-IDF, multiplies the term-specific frequency signal by a corpus-level rarity signal to produce discriminative document representations. Understanding TF variants deeply is essential background for understanding why the specific formulations chosen for TF-IDF (typically log TF with smooth IDF) make sense. Every TF variant in this chapter has a corresponding role in the TF-IDF formula, and the choices made in the TF part directly interact with the IDF weighting.

Another fundamental limitation is that term frequency, like the Bag of Words model it builds on, discards all word order information. The documents "the cat ate the mouse" and "the mouse ate the cat" have identical term frequency vectors. For tasks where semantic meaning depends on word order (which includes most tasks), this is a significant weakness. A TF-based document classifier cannot distinguish "the defendant is not guilty" from "the defendant is guilty" if both contain the same vocabulary. For coarse topic classification this usually doesn't matter much, but for sentiment analysis, question answering, or fine-grained semantic similarity, this limitation is severe.

The loss of order information is inherent to the bag-of-words approach and cannot be fixed by changing the TF weighting scheme alone. It requires either adding n-gram features (as covered in the N-grams chapter) or moving to sequence-aware representations like recurrent models or transformers. TF weighting remains competitive for many classification and retrieval tasks where topic modeling matters more than fine-grained semantics, and it offers interpretability advantages that dense embeddings sacrifice. You can always explain why a document received a high score: because it contained high-weight terms. With dense embeddings, this interpretability is largely lost.

The sparsity of TF vectors is both a limitation and a feature. For large corpora with diverse vocabularies, TF vectors become extremely high-dimensional and sparse, which is memory-intensive and can cause issues with distance metrics in high dimensions. The so-called "curse of dimensionality" describes how distances become less meaningful in high-dimensional spaces: as dimensionality grows, the ratio of the maximum to minimum distances between points shrinks, making all documents seem roughly equidistant. TF vectors live in spaces with tens of thousands of dimensions, and while sparsity partially mitigates the curse (because documents only use a small subspace), it remains a real challenge for methods that rely on geometric distances.

At the same time, the sparse structure enables efficient computation and makes individual features (terms) interpretable in a way that dense neural embeddings are not. Sparse retrieval systems built on TF-IDF can scale to billions of documents with modest computational resources, a property that dense retrieval systems have only recently matched through approximate nearest neighbor algorithms and hardware acceleration.

Despite these limitations, term frequency weighting had enormous practical impact. Through the 1990s and 2000s, TF-IDF (built directly on the TF variants in this chapter) powered the majority of web search engines and document retrieval systems. Early versions of web search engines like AltaVista and early Google prototypes used TF-IDF as their primary relevance scoring mechanism before learning-to-rank methods became dominant. Even today, TF-IDF baselines remain surprisingly competitive, and understanding TF is foundational to understanding BM25 (covered later in this part), which remains the dominant sparse retrieval method in production search systems. BM25 can be understood as an augmented version of TF-IDF with improved length normalization and a saturation parameter that acts like a more flexible version of log scaling.

Term frequency weighting converts raw word counts into document representations that are more analytically useful. The key variants, each encoding different assumptions about what frequency means:

• Raw TF is the simplest form, directly using word counts. It suffers from length bias and gives disproportionate weight to high-frequency terms.
• Log TF applies logarithmic compression to model diminishing returns: going from 0 to 1 occurrence matters most; going from 100 to 101 barely matters at all.
• Boolean TF discards all count information and records only presence or absence, which is surprisingly effective when counts are mostly 1 or when topic presence matters more than emphasis.
• Augmented TF normalizes each term's count by the maximum count in the document, making the representation length-invariant and capturing relative emphasis within a document.
• L2-normalized TF places document vectors on the unit sphere, enabling cosine similarity to be computed as a simple dot product and making the representation independent of document length.

In practice, log TF + L2 normalization is the most common choice and is the default behavior of scikit-learn's TfidfVectorizer before IDF is applied.

TF representations are inherently sparse: most documents use only a small fraction of the total vocabulary, and term-document matrices have 95-99%+ sparsity in practice. This sparsity follows directly from Zipf's law: the highly skewed distribution of word frequencies in natural language means that most vocabulary items appear rarely and thus produce mostly-zero columns in any practical corpus. Storing these matrices in sparse formats (like CSR) provides dramatic memory savings and computational speedups compared to dense storage.

The geometry of TF-weighted document spaces matters for downstream tasks. Raw TF places documents at varying distances from the origin based on their length; L2 normalization projects everything onto the unit sphere so that cosine similarity equals the dot product. Log TF reshapes the distribution within that sphere, pulling content words toward more equal influence with stopwords and making topical signals more visible.

Finally, TF weighting alone is insufficient for discriminative retrieval or classification because it doesn't account for a term's importance across the corpus. A word that appears frequently in one document tells you about that document's focus, but if that same word appears frequently in every document, it's not a useful discriminator. The next chapter introduces Inverse Document Frequency, which provides the missing global signal, and together they form the TF-IDF representations that remain foundational to text analysis and information retrieval.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about term frequency weighting.