RAG Evaluation: Metrics for Retrieval and Generation Quality › End-to-End Evaluation

The ultimate goal is evaluating whether the RAG system as a whole answers your queries accurately and faithfully. This requires three distinct dimensions of assessment, often called the "RAG Triad": context relevance, faithfulness (or attribution), and answer relevance. No single dimension captures the full picture. A system could excel at all three independently designed components while still failing users if those components do not integrate effectively.

End-to-end evaluation recognizes that RAG systems are more than the sum of their parts. A retrieval system might reach perfect precision and recall on benchmark queries, and a generator might produce fluent text on all test cases, yet the combination might fail to answer user needs because the retrieval and generation are tuned to different distributions or because the prompt format does not effectively communicate the retrieved context to the generator. Conversely, a system with imperfect retrieval might still succeed if the generator can synthesize partial information effectively. We must therefore evaluate whether the integrated pipeline produces useful answers that are accurate and supported by evidence.

The RAG Triad also is a diagnostic tool. When a system fails to produce a satisfactory answer, the triad helps identify which component is responsible. Low context relevance points to retrieval failures: the system is not finding the right information. Low faithfulness points to generation failures: the system is not using the retrieved information correctly, instead fabricating or misrepresenting it. Low answer relevance points to alignment failures: the system may be giving accurate and grounded information, but it is not the information the user requested. Each failure mode suggests a different remediation strategy. This makes the triad a practical framework for targeted improvement.

Context Relevance measures whether the retrieved chunks contain information necessary to answer the query. Think of it as asking: "Did the librarian bring back books that address my question?" Unlike retrieval precision, which requires human judgments of relevance collected before seeing the generated answer, context relevance for RAG can be assessed by checking whether the generator uses the retrieved information.

Formally, we can measure context relevance through attribution: for each sentence in the generated answer, we determine if it can be attributed to specific sentences in the retrieved context. High context relevance means retrieved documents contributed to the answer; low context relevance suggests the documents were off-topic or the generator ignored them in favor of parametric knowledge from its training data.

This attribution-based approach to context relevance has an important implication: it evaluates whether the retrieved documents contain useful information and whether that information appears in the generated answer. A retrieval system might return a highly relevant document, but if the generator's context window format, prompt design, or attention patterns cause it to ignore that document, context relevance will still be low. Context relevance therefore covers both retrieval quality and generator-retrieval integration, while retrieval precision measures only the former.

Attribution analysis gives a mechanistic understanding of how the generator interacts with retrieved documents. By tracing specific claims in the output back to specific passages in the input, we can determine whether the retrieval component successfully identified useful sources. This is particularly important for debugging: if the generator produces correct answers but shows low attribution to retrieved documents, it may be relying on parametric knowledge rather than the provided context. This creates a false sense of reliability that will fail when the knowledge base contains more recent or specific information than the model's training data.

Faithfulness (also called groundedness or attribution) measures whether the generated answer is factually consistent with the retrieved context. A faithful answer contains only information supported by the evidence, while an unfaithful answer hallucinates facts or contradicts the context.

Evaluating faithfulness requires claim-level verification. Modern approaches use entailment models or LLM prompting to check if claims in the answer are entailed by, contradicted by, or neutral to the retrieved passages. For a claim $c$ and context passage $p$, an NLI model predicts one of three categories:

• Entailment: $p$ supports $c$ (the claim follows logically from the passage)
• Contradiction: $p$ contradicts $c$ (the passage asserts something incompatible with the claim)
• Neutral: $p$ gives no information about $c$ (the passage is unrelated to the claim)

A faithful answer should have all claims entailed by the union of retrieved passages. Any claims classified as contradicted stand for direct factual errors. Claims classified as neutral are ambiguous: they might be general knowledge not present in the corpus, or they might be hallucinations. Many evaluation frameworks flag neutral claims as potential faithfulness violations and require human review for ambiguous cases.

Faithfulness is the cornerstone of trustworthy RAG systems. Unlike general language models that might generate plausible-sounding but unverified information, RAG systems promise to ground their outputs in retrievable evidence. Violations of faithfulness break this contract, creating hallucinations that appear authoritative because they are presented alongside real documents. Natural Language Inference models are particularly well-suited for this task because they are trained specifically to recognize textual entailment relationships, distinguishing between cases where one text supports, contradicts, or is unrelated to another.

The practical difficulty of faithfulness evaluation lies in claim extraction. Before we can verify claims, we must identify the claims made by the generated answer. Some claims are explicit and easily extracted: "The company was founded in 1995." Others are implicit: an answer that says "the technique was later improved" implicitly claims that an earlier version existed. Thorough faithfulness evaluation requires identifying both explicit and implicit claims, which is non-trivial for complex multi-sentence answers. LLM-based claim extractors handle this better than rule-based approaches, but they introduce their own potential for errors and inconsistencies.

Answer Relevance (or answer usefulness) measures whether the generated response addresses your query. A system could retrieve perfect context and cite it faithfully, yet fail to answer the question asked. This failure mode is more common than it might seem: language models sometimes respond to the topic of a question rather than the specific information requested. This gives general background when a specific answer is needed.

This often happens with overly cautious models that give relevant background information without the specific answer requested. Consider a query asking for the CEO of a company. A system might retrieve relevant documents and faithfully report that "The company was founded in 1990 and has grown to employ 50,000 people worldwide," all while never stating who the current CEO is. The answer is faithful and context-relevant but scores zero on answer relevance.

Answer relevance can be evaluated through several approaches:

1. Checking if the answer contains the specific information type requested (a date, a name, an explanation, a comparison).
2. Measuring semantic similarity between the query and answer, with the intuition that a relevant answer should address the same concepts as the question.
3. Using LLM judges to assess whether the answer responds to the question, drawing on the model's understanding of pragmatic discourse structure.

An elegant automated approach from RAGAS works by reverse-engineering the query. Given the generated answer, prompt an LLM to generate questions that the answer would appropriately respond to. Then measure the semantic similarity between these reverse-engineered questions and the original query. If the answer is truly relevant, the generated questions should closely resemble the original query. If the answer talks around the question without addressing it, the generated questions will diverge from the original.

Answer relevance captures the pragmatic dimension of system performance. A query asking for the current stock price of a company might receive a response that explains the company's business model, recent news, and market position, all accurately cited from retrieved documents. Such a response might score perfectly on faithfulness and context relevance while completely failing on answer relevance because it never states the stock price. This metric ensures that the system remains focused on user intent rather than merely generating related content.