Retrieval-Augmented Training › Key Parameters
Michael BrenndoerferMarch 11, 20261 min read
Language AI HandbookAll chapters Part 66 of 67Frontier Methods of 2025
Retrieval-Augmented Training › Key ParametersOne section · Sections in this chapter11 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for RETRO-style retrieval-augmented training are:
- chunk_size: The number of tokens per chunk, controlling the granularity of retrieval alignment. RETRO uses 64 tokens. Smaller chunks mean more frequent retrieval and finer alignment, at higher computational cost.
- k: The number of retrieved neighbors per chunk. RETRO retrieves 2 neighbors. More neighbors provide more context but increase the cross-attention cost and may introduce noise.
- neighbor_len: The length of each retrieved neighbor sequence, including the continuation. RETRO uses 128 tokens (64 neighbor + 64 continuation).
- retro_layer_frequency: How frequently CCA layers appear in the stack. More RETRO layers provide more integration opportunities but increase computation.
- database_size: The total number of tokens in the retrieval database. RETRO demonstrates that doubling database size consistently improves perplexity, suggesting diminishing but meaningful returns.