RAG Evaluation: Metrics for Retrieval and Generation Quality › Limitations and Challenges

RAG evaluation presents unique challenges that standard NLP benchmarks do not address. The primary difficulty lies in the absence of ground truth for open-domain applications. Unlike closed-domain question answering where answers come from a limited knowledge base with pre-defined answer spans, RAG systems often answer questions where the correct answer depends on the specific documents in the corpus, which makes it impossible to pre-annotate reference answers for all possible queries.

This absence of ground truth fundamentally changes the evaluation paradigm. Traditional machine learning relies on labeled datasets with clear correct answers, letting straightforward calculation of accuracy. In open-domain RAG, the knowledge base might contain millions of documents, and the space of possible queries is in effect infinite. We cannot create a test set covering all possible questions, and the correct answer to any specific question might change as the knowledge base is updated with newer information. This necessitates reference-free evaluation methods or dynamic evaluation pipelines that can assess quality without static gold standard answers.

A second major challenge is error compounding. Retrieval errors and generation errors interact in complex ways that are difficult to disentangle. A retrieval system might fetch the correct document but rank it at position 8, beyond the effective context window. Alternatively, perfect retrieval cannot save a generator prone to hallucination or one that ignores the provided context in favor of outdated parametric knowledge. The multiplicative interaction between retrieval and generation quality, as illustrated in our heatmap, means that small improvements to a weak component produce larger gains than equivalent improvements to an already strong component. But identifying which component is the weak link requires careful controlled experiments.

Understanding these interactions is important for system optimization. If retrieval reaches high NDCG but the generator produces unfaithful answers, the problem lies in generation, suggesting a need for better prompting, fine-tuning, or a stronger base model. If retrieval scores are low but the generator performs well, the model may be relying heavily on parametric knowledge, creating a false sense of security that will fail when queries target information outside the training distribution. Isolating these effects requires controlled ablation studies where components are evaluated independently and in combination.

The third major challenge is evaluator reliability. When using LLMs as judges in frameworks like RAGAS, the quality of the evaluation depends critically on the quality of the judge model and its prompts. LLM judges can be inconsistent across runs due to sampling temperature, biased toward confident-sounding but incorrect answers, sensitive to prompt phrasing, and calibrated differently on different query types. Research has shown that LLM judges correlate reasonably well with human judgment on average but have significant error rates on individual examples, particularly for ambiguous cases or queries requiring domain expertise. This means RAGAS scores should be treated as approximate indicators rather than ground truth, requiring periodic validation against human judgments.

A fourth challenge specific to RAG is the tension between faithfulness and completeness. A system that only generates answers when it finds strong supporting evidence will be highly faithful but may refuse to answer many valid queries. A system that generates answers aggressively will answer more queries but with lower faithfulness. Evaluation metrics typically measure faithfulness and answer relevancy independently, but the right balance depends on the application. In medical or legal contexts, faithfulness should be nearly absolute even at the cost of lower answer coverage. In general-purpose assistants, users may prefer a helpful but potentially less precise answer to no answer at all.

The computational cost of evaluation is also non-trivial. LLM-based evaluation can be as expensive as the generation step itself, especially for frameworks that make multiple LLM calls per query (claim extraction, claim verification, question generation). For large-scale evaluation across thousands of queries, this cost can become significant. Some teams use cheaper models for automated evaluation and reserve expensive models for spot-checking. Others develop lightweight proxy metrics that correlate with RAGAS scores on their specific domain, letting fast evaluation without full RAGAS computation for every change.

These challenges highlight the importance of human-in-the-loop validation. Automated metrics give necessary scalability for development and regression testing, but periodic human evaluation remains needed for validating the automated metrics themselves. If an LLM judge gradually drifts in its assessment criteria, or if prompt templates become ineffective for new types of queries, only human oversight can detect these systematic failures. The field continues to evolve toward hybrid approaches that combine automated screening at scale with human judgment on challenging or borderline cases, using active learning techniques to select the most informative cases for human review.