Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Term Frequency › Visualizing Term Frequency Distributions

Michael BrenndoerferMarch 30, 20256 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 11Term Frequency › Visualizing Term Frequency Distributions

Reading section 12 of 17 · Visualizing Term Frequency Distributions · 6 minSection 12 of 17 · 6 min

Section 12 of 17 · 6 min read

Term Frequency › Visualizing Term Frequency DistributionsOne section · Sections in this chapter12 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding what term frequency distributions look like in practice is as important as understanding the weighting formulas. Let's visualize the distribution of raw term frequencies in a real corpus.

Out[20]:

Visualization

Log-scale histogram of raw term frequency values showing a highly right-skewed distribution with most values at 1. — Distribution of raw term frequency values across all non-zero entries in the 20 Newsgroups corpus. The distribution is heavily right-skewed: the majority of term occurrences are single appearances, while a small number of terms (mostly function words) accumulate large counts, confirming the Zipfian nature of natural language.

Line plot comparing raw, log, boolean, and augmented TF weights for counts 1 to 20. — How four TF weighting schemes map raw counts to weights, for counts 1 through 20. Raw TF grows linearly while log TF curves to reflect diminishing returns. Augmented TF is bounded between 0.5 and 1.0 and boolean TF remains constant at 1 regardless of count, illustrating how each variant encodes a different assumption about the value of repetition.

The raw TF distribution is extremely right-skewed: roughly half of all non-zero entries are 1 (a word appears exactly once in a document), and counts above 10 are relatively rare outside stopwords. This means log scaling does most of its work on the minority of high-count entries, while boolean and raw TF give similar results for the majority of term-document pairs where the count is 1.

We can also visualize how the different weighting schemes change the appearance of the full document-term matrix for our small corpus. A heatmap makes the differences between schemes immediately visible.

Out[21]:

Visualization

Heatmap of raw term frequency matrix with documents as rows and vocabulary terms as columns. — Raw term frequency matrix for the three-document corpus. The stopword 'the' dominates with counts of 3-4, visually overwhelming lower-frequency but more informative terms like 'cat', 'patterns', and 'data', which demonstrates why raw counts alone can mislead downstream classifiers.

Heatmap of log-L2 normalized term frequency matrix showing more evenly distributed term weights. — Log-scaled and L2-normalized term frequency matrix for the same corpus. After normalization, the dominance of 'the' is substantially reduced and content words receive more balanced weights, making the documents easier to compare on the basis of meaning rather than length.

The side-by-side heatmaps make the effect of normalization concrete. In the raw count matrix, "the" dominates both Doc A and Doc B with values of 3 and 4, while content words like "cat," "dog," and "patterns" are barely visible. After log scaling and L2 normalization, the weights spread more evenly across terms: "the" no longer overwhelms everything else, and the distinctive content of each document becomes more apparent in the representation. Doc A becomes characterized by roughly equal weights for "cat," "sat," and "the"; Doc C shows "patterns" as a distinctively high-weight term relative to the other terms.