Embedding Models: Architecture, Pooling & Selection › Encoding Long Documents

A practical challenge for embedding models is handling text that exceeds the model's context window. Transformers have a fixed maximum sequence length, determined by the positional encoding scheme used during training and the memory allocated to the attention matrices. Any input that exceeds this limit must be handled explicitly.

As we discussed in the document chunking chapter, the standard approach is to split documents into chunks that fit within the model's limit. Each chunk is encoded independently, creating one embedding per chunk. At retrieval time, the query is compared against all chunk embeddings, and the most relevant chunks are retrieved. This approach has several advantages: it is simple, it is compatible with any embedding model regardless of context length, and it produces fine-grained retrieval (the specific relevant section is retrieved, not the entire document). The disadvantage is that information spanning chunk boundaries is split across two embeddings, and inter-chunk context is lost.

There are alternative strategies worth understanding, each with distinct trade-offs. Chunked mean pooling encodes each chunk separately and then averages the resulting embeddings into a single document embedding. This is simple and captures some sense of the document's overall content, but it loses the fine-grained resolution needed for question-specific retrieval. If you're looking for the answer to a specific question, averaging embeddings from all sections of a 50-page document will bury the relevant section's signal in the noise from all the irrelevant sections.

Hierarchical encoding produces chunk-level embeddings, then uses a separate aggregation model to combine them into a document embedding. This can potentially capture inter-chunk relationships if the aggregation model has sufficient expressive power, but it adds a second model to the pipeline, increases complexity, and the aggregation model itself requires careful design and training.

Long-context embedding models offer a third path. Newer models like jina-embeddings-v2-base-en (8,192 tokens) and nomic-embed-text-v1.5 (8,192 tokens) natively handle sequences many times longer than the original BERT limit of 512 tokens. These models use modified positional encoding schemes, such as ALiBi or extended RoPE contexts, that generalize to longer sequences than those seen during training.

For RAG applications, per-chunk embeddings are almost always preferable to document-level embeddings, regardless of whether the embedding model supports long contexts. When you ask a specific question, you want to retrieve the specific chunk that answers it, not an entire document that might bury the relevant passage among thousands of irrelevant tokens. The granularity of retrieval directly determines the quality of the context you provide to the LLM: a 500-token chunk containing exactly the relevant information is far more useful than a 10,000-token document where that information appears on page 8.