RAG Evaluation: Metrics for Retrieval and Generation Quality › The RAGAS Framework

Manually evaluating the RAG Triad for thousands of queries is impractical. The RAGAS (Retrieval-Augmented Generation Assessment) framework automates RAG evaluation using LLMs as judges. This gives metrics that approximate human judgment without requiring reference answers for most metrics. RAGAS was introduced by Es et al. in 2023 specifically to address the evaluation bottleneck in RAG development, recognizing that existing evaluation tools from NLP were designed for different tasks.

The automation of evaluation is important for practical RAG development. As systems iterate through different retrieval models, chunking strategies, embedding models, and prompt templates, developers need rapid feedback on which changes improve performance. Manual evaluation creates a bottleneck that slows iteration; automated metrics enable systematic optimization and regression testing. RAGAS uses the reasoning capabilities of large language models to simulate human judgment at scale, turning the evaluation problem into a collection of natural language tasks that LLMs can perform with reasonable accuracy.

The key insight behind RAGAS is that if you have a model capable enough to evaluate RAG system outputs, you can use it as an automated judge. This requires the judge model to be more capable than the system being evaluated, and it requires careful prompt design to minimize judge biases. When these conditions hold, automated evaluation can scale to hundreds of thousands of query-answer pairs, letting statistical analyses that would be impossible with human annotation.

RAGAS defines four key metrics, each targeting a specific dimension of system quality and computed via carefully designed LLM prompts.

Context Precision measures the signal-to-noise ratio in retrieved chunks. For each chunk in the top-$K$ results, an LLM is asked whether it contains information relevant to answering the query. Precision is then the fraction of chunks judged relevant, computed as a weighted average that accounts for chunk position. The prompts ask the judge model to make binary relevance judgments for each chunk independently, then aggregate these judgments into a single score.

Context Recall estimates coverage by asking an LLM to identify which sentences from a ground-truth answer are supported by the retrieved context. It is the fraction of ground-truth sentences that can be attributed to at least one retrieved chunk. This is the one RAGAS metric that requires reference answers, which makes it more expensive to compute but also more reliable as a measure of retrieval completeness. The distinction between context precision and context recall mirrors the precision-recall tradeoff we discussed earlier, but here both are measured from the perspective of informational coverage rather than document-level judgments.

Faithfulness uses an LLM to extract factual claims from the generated answer, then for each claim, determine if it can be inferred from the retrieved context. The score is the ratio of supported claims to total claims:

$$\text{Faithfulness} = \frac{\text{Number of supported claims}}{\text{Total number of claims}}$$

where:

• $\text{Number of supported claims}$: the count of factual claims in the generated answer that are entailed by the retrieved context
• $\text{Total number of claims}$: the total count of factual claims extracted from the generated answer by the LLM judge

A faithfulness score of 1.0 means every claim in the answer is grounded in the retrieved context. A score of 0.5 means half the claims are hallucinated or unverifiable from the context. This metric does not require reference answers because it only checks consistency between the generated answer and the retrieved context.

Answer Relevancy evaluates how directly the answer addresses the question. The LLM generates several potential questions that the given answer would appropriately respond to, then measures the mean cosine similarity between these generated questions and the original query using sentence embeddings. Low similarity indicates the answer is off-topic or incomplete. This metric also does not require reference answers, which makes it suitable for open-domain evaluation.

The metric definitions reflect a careful decomposition of system quality. Context Precision focuses on retrieval efficiency. This keeps limited context windows are not wasted. Context Recall addresses completeness, verifying that the retrieval system found all necessary information. Faithfulness checks the integrity of the generation process, while Answer Relevancy confirms utility to the end user. Together, these metrics give a complete dashboard for system health without requiring expensive human annotation for most queries.

RAGAS operates through a carefully orchestrated pipeline of LLM calls. For faithfulness evaluation, the process involves three distinct stages that transform a raw query-answer-context triple into a numerical score.

In the first stage, claim extraction, the system prompts the LLM with the generated answer and asks it to decompose the answer into atomic factual statements. An atomic claim is a minimal, self-contained factual assertion: "The Eiffel Tower is located in Paris" rather than "The Eiffel Tower, which is located in Paris, was built in 1889 and stands 330 meters tall." Breaking compound statements into atomic claims ensures that each claim can be independently verified against specific context sentences.

In the second stage, claim verification, the system presents each extracted claim alongside the full retrieved context and asks the LLM to classify whether the context supports, contradicts, or does not address the claim. This requires careful prompt design to prevent the judge model from hallucinating its own judgments or being influenced by the fluency of the generated answer rather than its grounding.

In the third stage, score aggregation, the system counts the fraction of verified (supported) claims and reports this as the faithfulness score. When implementing this pipeline, you should log intermediate results (extracted claims, individual verdicts) to let debugging when scores seem incorrect or inconsistent.

This approach assumes the LLM judge is more capable than the generator being evaluated, and that it can accurately assess entailment without hallucinating its own judgments. The implementation relies on the zero-shot reasoning capabilities of modern language models. By carefully crafting prompts that instruct the model to perform specific evaluation subtasks, RAGAS turns the evaluation problem into a series of classification and generation tasks that capable LLMs perform reliably.

The dependency on LLM judges introduces considerations about judge quality: the evaluating model should ideally be more capable than the system being evaluated, and prompts must be designed to minimize the judge's own biases and hallucinations. Practically, many teams use a stronger model (e.g., GPT-4) to evaluate outputs from a smaller model (e.g., GPT-3.5 or Llama), though this creates cost considerations that must be balanced against the evaluation budget.