Retrieval-Augmented Training › Retrieval-Augmented Training

Language models face a fundamental tension: the knowledge they contain is frozen at training time. No matter how large the model or how extensive its pretraining corpus, it cannot know about events that happened after the data cutoff, cannot access information that was never included in its training set, and must compress everything it has learned into billions of static parameters. This creates both a knowledge staleness problem and a storage efficiency problem. Every fact the model needs to recall must somehow be encoded in its weights, competing for capacity with the syntactic patterns, reasoning strategies, and linguistic regularities it also needs to internalize.

Retrieval-augmented generation (RAG), which you saw in earlier chapters on inference-time techniques, addresses this by supplying retrieved context to the model at inference time. The model reads relevant passages alongside the query and generates its response by conditioning on both. This works remarkably well, but it treats retrieval as an afterthought: the model was trained without any exposure to the retrieval interface, so it must cope with injected context through in-context learning rather than through internalized patterns for processing retrieved material.

Retrieval-augmented training takes a different approach. Instead of bolting retrieval onto a frozen model at inference time, the model is trained from the ground up with retrieval as an integral part of the forward pass. The model learns to read and use retrieved neighbors as part of its core computation, not as an incidental prompt prefix. The result is a model architecture and training regime where the boundary between "memory in weights" and "memory in retrieval database" is deliberately and efficiently managed.

The most influential architecture in this space is RETRO (Retrieval-Enhanced Transformer), introduced by Borgeaud et al. at DeepMind in 2021. RETRO showed that models with roughly 7 billion parameters could match the perplexity of GPT-3-scale models (175 billion parameters) when given access to a 2 trillion token retrieval database. The implication was provocative: a 25-fold reduction in parameter count with no quality loss, achieved by offloading factual storage to an external database rather than into model weights. This chapter unpacks how RETRO achieves this, why retrieval during training matters, and what the architecture teaches us about the separation of compute and memory in language modeling.