Retrieval-Augmented Training › Why the Scaling Results Are Surprising

The headline result from the RETRO paper is that a 7.5 billion parameter RETRO model achieves lower perplexity on language modeling benchmarks than a 175 billion parameter GPT-3. This is roughly a 25x parameter reduction, which requires some unpacking because raw parameter counts can be misleading.

The RETRO model, while small in parameter count, is supplemented by the 2 trillion token retrieval database. If we ask how much "information capacity" the full RETRO system has, we need to account for both the model weights and the retrieval database. The database itself is large: 2 trillion tokens at roughly 2 bytes per token is about 4 terabytes of raw text. In that sense, RETRO is better described as differently organized than simply more efficient.

The performance advantage also the performance advantage holds even at high numbers of retrieved tokens. The perplexity curves for RETRO as a function of database size and model size show a consistent pattern: doubling the database size improves perplexity more cheaply (in compute) than doubling model parameters. This suggests that much of what large language models store in their weights is effectively a compressed lookup table for world knowledge, and that externalizing this lookup table into a database is more efficient than encoding it in weights.

The comparison also reveals something about the nature of perplexity as a metric. Perplexity on the Pile and similar benchmarks is strongly influenced by the model's ability to predict rare and specific facts: proper nouns, technical terminology, numerical quantities. These are exactly the kinds of tokens where retrieval provides the most benefit, because they are hard to memorize in weights but easy to find in a database. Reasoning and syntax, by contrast, are less directly aided by retrieval because they require computation rather than lookup.