Vector Space Model & TF-IDF (1968) › Representing Documents Geometrically

Michael BrenndoerferApril 4, 20252 min read
History of Language AIAll chapters
Part 1 of 9Signals & Symbols
Chapter 10Vector Space Model & TF-IDF (1968) › Representing Documents Geometrically
Reading section 3 of 12 · Representing Documents Geometrically · 2 minSection 3 of 12 · 2 min

Book contents

History of Language AI9 parts · 110 chaptersView the whole book →
Active chapter 10 of 110

Section 3 of 12 · 2 min read

Vector Space Model & TF-IDF (1968) › Representing Documents GeometricallyOne section · Sections in this chapter3 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

In this model, each document is a point in a high-dimensional vector space. Every dimension represents a term in the collection vocabulary, and a document's coordinates depend on the terms it contains and their assigned weights.

More precisely, a document vector is an ordered list of term weights. A vocabulary of 10,000 terms produces 10,000 dimensions. Most coordinates are zero because a document uses only a fraction of the vocabulary, so the representation is sparse.

Documents with similar weighted vocabularies point in similar directions. Cosine similarity measures the cosine of the angle between two vectors: a smaller angle gives a higher score, while vectors with little overlap approach orthogonality.

Cosine Similarity: Measuring Semantic Proximity

The cosine similarity between two vectors d1⃗\vec{d_1} and d2⃗\vec{d_2} is computed as:

similarity(d1⃗,d2⃗)=cos⁡(θ)=d1⃗⋅d2⃗∣d1⃗∣⋅∣d2⃗∣\text{similarity}(\vec{d_1}, \vec{d_2}) = \cos(\theta) = \frac{\vec{d_1} \cdot \vec{d_2}}{|\vec{d_1}| \cdot |\vec{d_2}|}

The numerator is the dot product of the vectors, and the denominator normalizes by their magnitudes. With nonnegative TF-IDF weights, the result ranges from 0 to 1. A value of 1 indicates the same direction, while 0 indicates orthogonal vectors with no weighted-term overlap.

Because cosine similarity compares direction rather than magnitude, a long document and a short one can receive a high score when their term-weight distributions are similar.

Queries use the same vector representation as documents. Retrieval computes the similarity between a query vector and each candidate document vector, then ranks the documents by score.

The remaining question was how to assign each term weight.