Embedding Models: Architecture, Pooling & Selection › Embedding Dimensions

The dimensionality of the output embedding, the length of the vector returned by the model, is a important parameter that affects retrieval quality, storage costs, and computation speed. Every embedding model produces vectors of a fixed size, and this size determines how much information the model can pack into each representation. To understand why dimensionality matters, think of each dimension as an independent axis along which the model can distinguish between texts. More dimensions mean more axes of distinction, giving the model a richer vocabulary of features to describe semantic content. But more dimensions also mean larger vectors to store, more computation to compare them, and more memory to load them into RAM.

The challenge, then, is to choose a dimensionality that is large enough to capture the semantic distinctions your application requires, without incurring unnecessary computational and storage costs. There is a well-established empirical relationship between dimensionality and quality: below a certain threshold, quality degrades sharply because the model lacks the representational capacity to encode fine-grained distinctions. Above a certain threshold, quality plateaus and the marginal benefit of additional dimensions is small. The location of this threshold depends on the complexity of your domain, the diversity of your corpus, and the model's architecture and training.

Embedding dimensions across popular models span a wide range, which reflects the diversity of model architectures and intended use cases:

The dimension is typically determined by the hidden size of the underlying transformer. BERT-base has a hidden size of 768, so BERT-based embedding models naturally produce 768-dimensional vectors. Smaller models like MiniLM distill into a 384-dimensional space, while larger models based on LLMs produce vectors with thousands of dimensions. Notice the general trend: as models grow in parameter count, their embedding dimensionality tends to increase. This correlation is not coincidental. Larger models have wider hidden layers, which means more dimensions are available at the output, and training objectives can exploit this additional capacity to encode finer-grained distinctions.

Higher dimensions offer more representational capacity. Each dimension is a feature axis that the model can use to encode some aspect of meaning. With more dimensions, the model can represent finer-grained distinctions between texts that differ in subtle ways, such as two scientific passages about related but distinct phenomena. The extra dimensions give the model room to encode features that smaller spaces cannot accommodate without interference between distinct features.

But higher dimensions come with real costs. Storage costs scale linearly: each vector requires $d \times 4$ bytes in float32 (or $d \times 2$ bytes in float16). For 10 million documents at 1024 dimensions in float32, that's roughly 40 GB of vector storage, before accounting for any index overhead. At 4096 dimensions (E5-Mistral), that grows to 160 GB. Computation costs also scale linearly: cosine similarity between two vectors requires $O(d)$ operations, so doubling the dimension doubles the similarity computation time. For brute-force search over $n$ vectors, the total cost is $O(nd)$, so both $n$ and $d$ directly multiply the search time. Approximate nearest neighbor indices (which the upcoming chapters on HNSW and IVF will cover) also scale with dimensionality, requiring more memory and longer index-build times for higher-dimensional vectors.

The practical sweet spot for most applications is 384 to 1024 dimensions. Below 256, quality degrades noticeably for complex semantic tasks because the model simply does not have enough dimensions to express the nuances that differentiate closely related but distinct meanings. Above 1024, the gains are marginal for most general-purpose retrieval scenarios, though specific domains or multilingual settings may benefit from higher dimensions. Multilingual models must simultaneously represent the semantic spaces of many different languages in the same vector space, and this additional complexity can benefit from the extra capacity that higher dimensionality provides.

A clever technique called Matryoshka Representation Learning (MRL), introduced by Kusupati et al. (2022), trains embeddings so that any prefix of the vector is a valid, useful embedding. The name comes from Russian nesting dolls (matryoshka): just as each doll contains a smaller doll inside, a 1024-dimensional Matryoshka embedding contains a useful 512-dimensional embedding in its first 512 components, a useful 256-dimensional embedding in its first 256 components, a useful 128-dimensional embedding in its first 128 components, and so on. This is a remarkable property because it decouples the choice of dimensionality from the choice of model. Without Matryoshka training, if you want embeddings of a different size, you need a different model entirely. With Matryoshka training, a single model can serve multiple dimensionality requirements simply by truncating its output vectors at different lengths.

The training procedure is elegant. During training, the loss is computed on the full-dimensional embedding and on truncated versions at multiple target sizes. The model is simultaneously asked to produce good embeddings at every target dimensionality, which means the learning signal flows back through the network from multiple truncation points at once:

$$\mathcal{L}_{\text{MRL}} = \sum_{d \in \mathcal{D}} \mathcal{L}_{\text{contrastive}}(\mathbf{e}_{[:d]})$$

where:

• $\mathcal{L}_{\text{MRL}}$: the total Matryoshka Representation Learning loss
• $\mathcal{D}$: a set of target output dimensions (e.g., $\{64, 128, 256, 512, 1024\}$)
• $d$: a specific dimension size from the set $\mathcal{D}$
• $\mathbf{e}_{[:d]}$: the embedding vector truncated to keep only the first $d$ dimensions
• $\mathcal{L}_{\text{contrastive}}$: the contrastive loss function applied to the truncated vectors

To understand what this loss function achieves, consider what happens at each truncation point. For a given target dimension $d$, the model takes only the first $d$ components of its output vector and computes the contrastive loss using those components alone. This loss penalizes the model if the truncated vectors do not preserve the correct ranking of similar and dissimilar pairs. By summing these penalties across all target dimensions in $\mathcal{D}$, the total loss forces the model to ensure that every prefix, from the shortest to the longest, produces embeddings that respect semantic relationships.

This loss has a fascinating effect on what the model learns: it forces the model to pack the most important information into the earlier dimensions. The first 64 dimensions must capture the broadest semantic content, the coarsest features that distinguish major topics and categories. The next 64 dimensions refine this picture, adding details that separate more closely related texts. Each subsequent block of dimensions adds progressively finer-grained distinctions, so that the full-dimensional vector captures the most fine-grained semantic information the model can represent. At inference time, you can truncate embeddings to any prefix length, trading some retrieval quality for storage and computation savings without retraining.

Think of Matryoshka embeddings as a compressed image in a hierarchical format. The first few dimensions give you a coarse, low-resolution picture of the document's meaning. Each additional block of dimensions adds detail, sharpening the picture. You can stop decoding at any resolution level depending on how much precision you need for your current task.

Many modern embedding models support Matryoshka embeddings, including nomic-embed-text-v1.5, models in the gte family, and OpenAI's text-embedding-3 series. Let's see this in action by encoding a query and three documents at multiple dimensionalities and observing how retrieval quality changes as we increase the number of dimensions.

The results demonstrate the Matryoshka property nicely. Even at just 64 dimensions, the model correctly identifies the aurora/magnetic field document as most relevant. As we increase dimensions, the separation between relevant and irrelevant documents improves, and the absolute similarity scores become more calibrated. For a production system, this means you could use 128 or 256 dimensions for a fast initial retrieval pass over a large corpus, then use the full 768 dimensions for a more precise re-scoring of the top candidates, all from the same embedding model and without any retraining.

The key parameters used in this implementation are:

• trust_remote_code: Required for models with custom architectures (like Nomic's) that use code not yet merged into the standard Transformers library. Always review the model's repository before letting this option.
• convert_to_tensor: Instructs the model to return PyTorch tensors instead of NumPy arrays, letting efficient GPU-accelerated operations.