Term Frequency › The Geometry of TF Weighting

It's worth pausing to think about what these weighting schemes are doing to the geometry of the document space, because the geometric picture connects TF weighting to the behavior of downstream classifiers and retrieval systems.

Without any normalization, documents are represented as points in $\mathbb{R}^{|V|}$ where the distance from the origin reflects document length. Two documents about the same topic but of different lengths will be far apart in the raw TF space simply because one vector has larger magnitude. A nearest-neighbor classifier working in this space would be strongly influenced by document length, which is rarely the right inductive bias.

After L2 normalization, all documents are projected onto the unit hypersphere. The Euclidean distance between two unit vectors is $\|\hat{\mathbf{v}}_d - \hat{\mathbf{v}}_{d'}\|_2 = \sqrt{2 - 2\cos\theta}$, where $\theta$ is the angle between them. This means Euclidean distance on the unit sphere is a monotone function of cosine similarity: minimizing Euclidean distance is equivalent to maximizing cosine similarity. You get the same nearest neighbors whether you use Euclidean distance or cosine similarity, as long as all vectors are L2-normalized.

This geometric perspective also clarifies why cosine similarity is the natural similarity metric for text. Euclidean distance in the raw count space is sensitive to document length, while cosine similarity is not. By normalizing to the unit sphere, you bring all documents to the same "radius" and make Euclidean distance in the normalized space equivalent to the angle-based similarity that ignores magnitude differences.

The log transformation changes the shape of the unit sphere's "population." Without log scaling, long documents with high counts cluster near the "stopword directions" in the high-dimensional space, because high-frequency function words dominate the magnitude. With log scaling, the high-count advantage of stopwords is compressed, and content words with lower but meaningful counts contribute more to the direction of the vector. Log normalization effectively rotates documents in the high-dimensional space to align more with their topical content.