Retrieval-Augmented Training › The Case for Retrieval During Training

To understand why training with retrieval is qualitatively different from inference-time RAG, consider what a model must learn in each setting.

In standard pretraining, the model is exposed to text tokens and must predict each next token. Over trillions of tokens, it learns to encode factual associations in its weights: that Paris is the capital of France, that Einstein developed the theory of relativity, that the boiling point of water at sea level is 100 degrees Celsius. These facts become distributed representations entangled with all the other knowledge in the network. Recalling a specific fact requires routing computation through the right layers and attention heads, with no guarantee that rarely-seen facts are reliably stored or that common facts have not been overwritten by more frequent patterns.

In inference-time RAG, a frozen model receives retrieved text prepended to its context. The model must figure out how to use this external context purely through in-context learning. It was never explicitly trained to extract and integrate retrieved information, so the integration quality depends on how well the retrieved text resembles the kind of context the model saw during pretraining. If the model has never been shown a pattern like "Here is a retrieved passage: [passage]. Now answer: [question]," it must generalize from loosely related training examples.

Training with retrieval changes this entirely. The model sees retrieved neighbors during every training step and learns to compute loss conditioned on both the local context and the retrieved material. It develops specialized architectural mechanisms for reading retrieved text and integrating it into its representations. The training objective directly rewards the model for using retrieved context well. By the time the model is deployed, it has internalized a general strategy for using retrieval rather than trying to adapt a strategy developed for a different setting.

There is also a compression argument. When a model must store facts in its weights, those parameters must simultaneously encode the facts and participate in general language understanding and reasoning. These roles may not be perfectly compatible. A parameter that stores "the capital of France" cannot simultaneously be optimal for representing syntactic agreement patterns. Retrieval-augmented training lets the model specialize its parameters more heavily toward language understanding and reasoning, while delegating factual storage to the retrieval database where it can be stored verbatim and accessed precisely.