Dense Retrieval: Semantic Search & Bi-Encoder Implementation › Embedding Similarity Metrics

Once we have query and document embeddings, we need a similarity function to rank documents. The three most common metrics are dot product, cosine similarity, and Euclidean distance. Each of these metrics captures a different notion of what it means for two vectors to be "similar," and understanding their geometric interpretations helps us choose the right metric for a given application.

Think of the embedding space as a high-dimensional room where every piece of text has a specific location. Similarity metrics are different ways to measure how "close" two locations are. The dot product measures closeness while also caring about how far each vector is from the origin. Cosine similarity measures only the angle between vectors, ignoring how far they are from the origin. Euclidean distance measures the straight-line distance between the two locations in the room. Each notion of closeness has its uses, and the right choice depends on how the embeddings were trained and what properties you want retrieval to have.

The dot product quantifies the similarity between two vectors by aggregating their aligned components. To understand this intuitively, imagine two vectors as arrows pointing in some direction in high-dimensional space. The dot product measures how much one vector "projects" onto another, combining information about both their directions and their lengths. For vectors $\mathbf{q}$ and $\mathbf{d}$, the dot product is defined as:

$$\text{sim}(\mathbf{q}, \mathbf{d}) = \mathbf{q} \cdot \mathbf{d} = \sum_{i=1}^{D} q_i d_i$$

where:

• $\text{sim}(\mathbf{q}, \mathbf{d})$ is the similarity score
• $\mathbf{q}, \mathbf{d}$ are the dense vectors for the query and document
• $D$ is the dimensionality of the embedding space (e.g., 768)
• $q_i, d_i$ are the $i$-th scalar components of each vector
• $\sum$ aggregates alignment across all $D$ dimensions

The geometric interpretation of this formula is instructive. Each dimension of the embedding space captures some aspect of meaning. When both $q_i$ and $d_i$ are large and positive, their product contributes positively to the similarity, showing that both the query and document exhibit that particular semantic feature strongly. Conversely, when one component is positive and the other is negative, their product is negative, reducing the overall similarity score. The dot product is therefore highest when the two vectors point in the same direction and have large magnitudes.

The dot product is computationally efficient and captures both the alignment of vectors (their angular similarity) and their magnitudes. Larger embeddings produce larger dot products, which can be useful when magnitude carries semantic meaning. For example, longer, more detailed documents might have larger embedding norms, and this property could be desirable if we want to favor complete, information-rich documents over brief ones.

Many dense retrieval models, including the original DPR (Dense Passage Retrieval), use dot product similarity because it can be computed extremely efficiently using matrix multiplication. Given a query vector and a matrix of document vectors, all similarity scores can be computed in a single matrix-vector multiplication using highly optimized BLAS libraries that run at near-peak hardware throughput.

While the dot product captures both direction and magnitude, there are situations where we want to focus purely on semantic direction, ignoring how "long" the vectors are. Cosine similarity addresses this need by normalizing the dot product by the magnitudes of both vectors, measuring only the angular alignment:

$$\begin{aligned} \cos(\mathbf{q}, \mathbf{d}) &= \frac{\mathbf{q} \cdot \mathbf{d}}{\|\mathbf{q}\| \|\mathbf{d}\|} \\ &= \frac{\sum_{i=1}^{D} q_i d_i}{\sqrt{\sum_{i=1}^{D} q_i^2} \sqrt{\sum_{i=1}^{D} d_i^2}} \end{aligned}$$

where:

• $\cos(\mathbf{q}, \mathbf{d})$ is the cosine similarity score, ranging from $-1$ to $+1$
• $\mathbf{q} \cdot \mathbf{d}$ is the unnormalized dot product (raw alignment)
• $\|\mathbf{q}\|, \|\mathbf{d}\|$ are the Euclidean norms (lengths) of the vectors
• $\sqrt{\sum q_i^2}$ computes the vector's magnitude as the square root of the sum of squared components

The cosine similarity ranges from $-1$ (vectors pointing in opposite directions) to $+1$ (vectors pointing in the same direction), with $0$ showing orthogonality. By removing magnitude effects, cosine similarity focuses purely on semantic direction. This is particularly useful when we want to compare texts of different lengths on an equal footing, since a longer document might naturally produce a larger-magnitude embedding without being more relevant to any particular query.

The key insight is that by normalizing, we make the similarity score a function only of the angle between the vectors. Two documents that say exactly the same thing but one is much longer (perhaps with more examples and explanations) should have similar cosine similarity to a query, even if the longer document's raw embedding vector is larger in magnitude. This length-invariant behavior is often what we want in retrieval.

A useful equivalence emerges when embeddings are L2-normalized, meaning each vector is divided by its magnitude so that $\|\mathbf{q}\| = \|\mathbf{d}\| = 1$. In this case, cosine similarity equals the dot product:

$$\cos(\hat{\mathbf{q}}, \hat{\mathbf{d}}) = \hat{\mathbf{q}} \cdot \hat{\mathbf{d}}$$

Here, $\hat{\mathbf{q}}$ and $\hat{\mathbf{d}}$ are the unit-length vectors derived by dividing $\mathbf{q}$ and $\mathbf{d}$ by their respective norms. The dot product operation, applied to these normalized vectors, directly yields the cosine of the angle between the original vectors.

This equivalence is practically very useful: we can pre-normalize all document embeddings once during indexing, then use the faster dot product operation at query time while still computing cosine similarity. This is exactly what many production systems do, including most models in the Sentence Transformers library. Pre-normalizing once means we never need to compute norms at query time, keeping retrieval latency minimal.

The third common metric takes a different perspective entirely. Rather than measuring how vectors align, Euclidean (L2) distance measures the straight-line distance between vectors in the embedding space:

$$\text{dist}(\mathbf{q}, \mathbf{d}) = \|\mathbf{q} - \mathbf{d}\| = \sqrt{\sum_{i=1}^{D} (q_i - d_i)^2}$$

where:

• $\text{dist}(\mathbf{q}, \mathbf{d})$ is the Euclidean distance (lower means more similar)
• $\mathbf{q} - \mathbf{d}$ is the difference vector, computed element-wise
• $(q_i - d_i)^2$ is the squared difference in dimension $i$, which is always non-negative
• $\sqrt{\cdot}$ converts the sum of squared differences to linear distance units

Unlike the similarity metrics above, smaller Euclidean distances indicate greater similarity. This is an important distinction to keep in mind when implementing retrieval systems: with Euclidean distance, we search for the nearest neighbors rather than the highest-scoring documents.

Euclidean distance is related to dot product for normalized vectors. When vectors have unit length, minimizing Euclidean distance is mathematically equivalent to maximizing the dot product. To see why, expand the distance formula:

$$\begin{aligned} \|\mathbf{q} - \mathbf{d}\|^2 &= \|\mathbf{q}\|^2 - 2\mathbf{q} \cdot \mathbf{d} + \|\mathbf{d}\|^2 \\ &= 1 - 2\mathbf{q} \cdot \mathbf{d} + 1 \quad \text{(for unit vectors)} \\ &= 2(1 - \mathbf{q} \cdot \mathbf{d}) \end{aligned}$$

Since $2$ and $1$ are constants, minimizing $\|\mathbf{q} - \mathbf{d}\|^2$ is equivalent to maximizing $\mathbf{q} \cdot \mathbf{d}$, which for normalized vectors equals cosine similarity. This relationship allows certain approximate nearest neighbor indexing algorithms designed for Euclidean distance to be adapted for dot product or cosine similarity retrieval, which is important for efficient large-scale search.

The choice of similarity metric should match the training objective used when the embedding model was created. This is not a trivial constraint: using the wrong metric can substantially degrade retrieval quality even for an otherwise excellent model.

• Models trained with dot product loss perform best with dot product similarity at inference time
• Models trained with cosine loss perform best with cosine similarity
• Many embedding models normalize their outputs, making the choice less necessary in practice

In practice, cosine similarity (or equivalently, dot product with normalized embeddings) is the most common choice because it is reliable to variations in embedding magnitude and provides easily interpretable scores. When similarity ranges from $-1$ to $+1$, it becomes straightforward to set thresholds and compare scores across different queries and different corpora. A score of 0.8 means something consistent regardless of whether the query is long or short, whether the document is a sentence or a paragraph.

One subtle consideration is how the model was trained with respect to embedding norms. Some contrastive learning setups allow the model to encode confidence or importance in the embedding norm, making dot product the preferred metric. Others explicitly normalize embeddings during training, making cosine similarity the correct choice. Always check the documentation for any pre-trained embedding model to understand what metric it expects.