RAG Evaluation: Metrics for Retrieval and Generation Quality › RAG Evaluation
Language AI HandbookAll chapters RAG Evaluation: Metrics for Retrieval and Generation Quality › RAG EvaluationOne section · Sections in this chapter1 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Building a Retrieval-Augmented Generation system involves connecting two complex components: a retriever that fetches relevant context from a knowledge base, and a generator that synthesizes answers from that context. As we explored in earlier chapters on RAG architecture, this two-stage design introduces a unique evaluation challenge. Unlike evaluating a standalone language model or a pure search system, RAG requires us to assess the quality of retrieved documents, the quality of generated text, and how effectively the two components work together. A perfect generator cannot compensate for retrieval that misses necessary information, and flawless retrieval is wasted if the generator hallucinates facts not present in the context.
Think of it like a research team working under a strict deadline. One team member (the retriever) runs to the library and gathers a stack of papers. Another team member (the generator) reads those papers and writes the report. Evaluating only the final report tells you whether the writer is good but nothing about whether the librarian found the right sources. Evaluating only the stack of papers tells you nothing about whether the writer used them correctly. Quality assessment requires examining both tasks and how they interact.
This chapter develops a complete framework for RAG evaluation. We begin with retrieval metrics that measure how effectively our vector search or hybrid retrieval systems surface relevant information. These metrics originate in the information retrieval literature, where researchers have spent decades developing principled ways to measure search engine quality, and we adapt them to the specific constraints of RAG, particularly the limited context window budget. We then examine generation metrics suited for factual synthesis rather than creative writing, explaining why the standard n-gram overlap metrics used in translation and summarization fail for this task and what better alternatives exist.
Finally, we explore end-to-end evaluation approaches that assess whether the system as a whole gives faithful, relevant answers grounded in retrieved evidence. The "RAG Triad" of context relevance, faithfulness, and answer relevance gives a diagnostic framework that lets us pinpoint exactly where a system is failing, whether in retrieval quality, generation integrity, or user-facing utility. Along the way, we implement the RAGAS framework, which uses language models themselves to automate these evaluations at scale, letting the rapid iteration cycles that practical system development requires.
A necessary point deserves emphasis before we dive in: evaluation is an ongoing development tool, not a final validation step. Every design decision in a RAG system, from chunk size to embedding model to retrieval strategy to prompt format, creates tradeoffs that need measurement to optimize. Without rigorous evaluation metrics, you cannot tell which changes help. With them, you can systematically identify bottlenecks and make evidence-based improvements. The investment in evaluation infrastructure pays dividends throughout the system's lifetime.
The metrics we use for RAG evaluation did not emerge from the LLM era but from decades of information retrieval research dating back to the 1960s. Precision and Recall were formalized by Cyril Cleverdon in the Cranfield experiments (1957-1966), which systematically measured indexing effectiveness for scientific literature retrieval. Mean Average Precision became the standard evaluation protocol for TREC (Text REtrieval Conference) competitions starting in 1992. NDCG was proposed by Jarvelin and Kekalainen in 2000 to address the binary nature of earlier metrics. These IR metrics were later adopted by the NLP community for tasks like passage ranking, and they now form the backbone of RAG retrieval evaluation. The generation evaluation side has a different lineage: BLEU emerged from machine translation (Papineni et al., 2002), ROUGE from text summarization (Lin, 2004), and BERTScore from the advent of contextual embeddings (Zhang et al., 2020). RAGAS specifically for RAG evaluation appeared in 2023 (Es et al., 2023). This shows how recently the community recognized that existing metrics were inadequate for this new paradigm.