TF-IDF › Document Similarity with TF-IDF
Language AI HandbookAll chapters TF-IDF › Document Similarity with TF-IDFOne section · Sections in this chapter6 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
One of TF-IDF's most powerful applications is computing document similarity. Two documents are similar if they share the same distinctive terms with similar TF-IDF weights. The natural way to measure this is cosine similarity: the cosine of the angle between their TF-IDF vectors. Cosine similarity measures how aligned two vectors are in their direction, independent of their magnitude.
Understanding Cosine Similarity
The geometric intuition is useful here. Think of each document as a point in a high-dimensional space, where each dimension corresponds to one vocabulary term. The TF-IDF score is the coordinate along that dimension. Two documents discussing the same topic will have high coordinates along the same dimensions (the topic-specific terms) and low coordinates elsewhere. Their vectors point in similar directions, so the angle between them is small and the cosine is close to 1. Two documents on completely different topics have high coordinates along completely different dimensions, so their vectors are nearly perpendicular and the cosine is close to 0.
For two L2-normalized TF-IDF vectors and , cosine similarity reduces to the dot product:
where:
- : the L2-normalized TF-IDF score of term in document
- : the L2-normalized TF-IDF score of term in document
- The sum runs over all terms in the vocabulary
The result ranges from 0 (no shared distinctive terms, orthogonal vectors) to 1 (identical TF-IDF profiles). The formula naturally rewards shared high-scoring terms: if term has and , its contribution to the dot product is , much larger than a shared term with and contributing . High-TF-IDF shared terms dominate the similarity score, which is exactly what you want.
def cosine_similarity_matrix(tfidf_normalized):
"""
Compute pairwise cosine similarity for all documents.
With L2-normalized vectors, this is just the dot product matrix.
"""
return tfidf_normalized @ tfidf_normalized.T
sim_matrix = cosine_similarity_matrix(tfidf_l2)
The off-diagonal cells reveal document relationships that match intuition. Documents 2 and 4 ("deep learning" and "computer vision") score highest similarity because they share the distinctive term "deep learning." Documents on completely different topics score near 0, indicating little shared distinctive vocabulary. The similarity matrix is symmetric, and the diagonal is always 1.0. These are mathematical guarantees of the formulation, not coincidences.
Using Similarity for Search
Document similarity makes TF-IDF directly useful for search. Given a query string, you compute its TF-IDF vector using the corpus vocabulary and IDF values computed during training, then rank all documents by their cosine similarity to the query vector.
The design decision here is that the query uses corpus IDF values, not its own IDF values. You can't compute IDF from a single query string (it has only one "document"), so you reuse the IDF values estimated from the training corpus. This makes the IDF values a property of the corpus, not of individual queries, which is the right way to think about it: IDF captures something about which terms are informative in this domain, and the corpus is your best estimate of domain statistics.
def vectorize_query(query, vocab, idf, use_log_tf=False):
"""Convert a query string into a TF-IDF vector using corpus vocabulary."""
tokens = tokenize(query)
tf = compute_tf(tokens)
vector = np.zeros(len(vocab))
for term, count in tf.items():
if term in vocab:
tf_val = log_tf(count) if use_log_tf else count
idf_val = idf.get(term, 0) # Unknown terms get 0
vector[vocab[term]] = tf_val * idf_val
# L2 normalize
norm = np.linalg.norm(vector)
if norm > 0:
vector /= norm
return vector
query = "deep neural networks for image tasks"
query_vec = vectorize_query(query, vocab, idf)
scores = tfidf_l2 @ query_vec
ranked = sorted(enumerate(scores), key=lambda x: x[1], reverse=True)Query: 'deep neural networks for image tasks' Ranked documents: 1. Doc 2 (score=0.6337): Deep learning uses neural networks. Neural networks learn hi... 2. Doc 4 (score=0.2300): Computer vision analyzes images. Image recognition uses deep... 3. Doc 3 (score=0.1224): Natural language processing extracts meaning from text. Text... 4. Doc 1 (score=0.0000): Machine learning algorithms learn patterns from data. Learni... 5. Doc 5 (score=0.0000): Reinforcement learning agents learn through rewards. Learnin...
The most relevant document ("Computer vision analyzes images...") ranks first because it shares the distinctive terms "deep" and "images" with the query. This exact pattern, computing a query vector with corpus IDF values and ranking by dot product, is the core of classical vector space model information retrieval. It was the dominant approach in search for decades before neural methods.
Out-of-vocabulary terms in the query (words not seen in the training corpus) receive weight 0, as shown in the idf.get(term, 0) call. This is an important limitation: if a user searches for a neologism or a highly specialized term that doesn't appear in your corpus, the system has no way to represent it. Smoothed IDF doesn't help here (it only prevents zero-IDF for terms present in all documents, not absent terms). Handling vocabulary mismatch is one of the motivations for subword-based representations, which we cover in the tokenization chapters.