RAG Evaluation: Metrics for Retrieval and Generation Quality › Evaluating the Generation Component

Once documents are retrieved, the generator must synthesize them into a coherent answer. Traditional NLP generation metrics like BLEU and ROUGE, which we might recall from machine translation and summarization tasks, have significant limitations when applied to RAG. Understanding why these metrics fail, and what better alternatives look like, helps clarify the basic requirements for evaluating factual synthesis.

The generation component faces a distinct challenge compared to open-ended text generation or translation. Rather than creating original content or creating a predetermined target text, the generator must condense information extracted from the provided context and present it accurately. This is closer to reading comprehension than to creative writing or translation. The evaluation paradigm must therefore prioritize semantic correctness over lexical similarity, and factual grounding over stylistic matching.

Think of evaluating RAG generation as checking whether a student's answer on an open-book exam correctly uses the provided materials. You care about whether the student understood and correctly applied the source material, not whether their answer matches a specific model answer word-for-word. A student who writes "France's capital city is Paris" and one who writes "Paris is the capital of France" have given equally correct answers despite using different words. Standard text generation metrics struggle with this kind of evaluation because they are fundamentally built around lexical similarity to a reference text.

BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) rely on n-gram overlap between generated and reference texts. BLEU focuses on precision (how many generated n-grams appear in the reference), while ROUGE focuses on recall (how many reference n-grams appear in the generation).

These metrics were developed for scenarios with well-defined target outputs, such as translating a specific sentence or summarizing a specific article. They operate on the assumption that good generated text will closely resemble a reference text at the lexical level, matching specific phrases and word sequences. For these use cases, this is a reasonable assumption: a translation that shares many n-grams with the professional reference translation is likely high quality, and with enough reference translations (BLEU typically uses 4 references), the metric becomes fairly reliable.

These metrics fail for RAG for three specific and important reasons:

1. Reference dependency: RAG systems often answer open-domain questions where no single reference answer exists. The same factual query can be answered correctly in many ways, using different phrasing, different levels of detail, different amounts of context, and different ways of organizing the information.

2. Semantic equivalence blindness: Paraphrases like "The treaty was signed in 1919" and "In 1919, the parties signed the agreement" share no bigrams but have identical meanings. N-gram metrics would penalize the second phrasing even though it is equally correct.

3. Hallucination blindness: N-gram metrics compare against references, not against the retrieved context, so they cannot detect when a model hallucinates information not present in the source documents. A generated answer could reach high BLEU by matching the reference while containing additional fabricated claims that the reference happened not to include.

The reference dependency problem is particularly acute in knowledge-intensive applications. Consider a query about a company's quarterly earnings. A valid answer might state revenue in millions or billions, might round to different decimal places, might present figures in different orders, or might emphasize different metrics, all while being factually correct. No single reference text can capture this valid variation, making n-gram overlap a poor proxy for quality.

The hallucination blindness problem may be the most serious limitation for RAG specifically. The entire motivation for RAG is to ground model outputs in specific, verifiable source documents. If our evaluation metric cannot detect violations of this grounding property, we are measuring the wrong thing entirely. A system that frequently hallucinates but happens to produce fluent text resembling the reference answer could score well on BLEU while being entirely untrustworthy in deployment.

BERTScore addresses the paraphrase problem by using contextual embeddings. Instead of comparing n-grams, it computes pairwise cosine similarity between token embeddings from the generated and reference texts, using a pre-trained model like BERT or RoBERTa.

BERTScore is a change from lexical matching to semantic matching. By embedding tokens in a high-dimensional vector space where semantically similar words cluster together, it recognizes that "automobile" and "car" are in effect equivalent in most contexts, even though they share no characters. This aligns better with human judgment of text quality, which focuses on meaning rather than exact wording.

For each token $t_g$ in the generated text, we find its most similar token $t_r$ in the reference by computing cosine similarity between their contextual embeddings:

$$\text{Similarity}(t_g, t_r) = \frac{\mathbf{e}_{t_g} \cdot \mathbf{e}_{t_r}}{\|\mathbf{e}_{t_g}\| \|\mathbf{e}_{t_r}\|}$$

where:

• $t_g$: a token from the generated text
• $t_r$: a token from the reference text
• $\mathbf{e}_{t_g}$: the contextual embedding vector for token $t_g$ (from a pre-trained model like BERT), which encodes the token's meaning in context
• $\mathbf{e}_{t_r}$: the contextual embedding vector for token $t_r$
• $\cdot$: the dot product between two vectors
• $\|\mathbf{e}\|$: the Euclidean norm (magnitude) of embedding vector $\mathbf{e}$

The cosine similarity ranges from -1 to 1, with values near 1 showing that the two tokens appear in similar contexts in the training data, suggesting they carry similar meanings. BERTScore then builds precision, recall, and F1 scores based on these token-level similarities, following the same structure as ROUGE but using embedding similarity instead of exact overlap.

BERTScore's precision score assigns each generated token to its most similar reference token and averages those maximum similarities. The recall score does the reverse, assigning each reference token to its most similar generated token. The F1 score is the harmonic mean of these two. This gives an overall semantic similarity measure.

While better than n-gram metrics for capturing semantic equivalence, BERTScore still requires reference answers and does not verify factual grounding in retrieved contexts. The cosine similarity calculation measures the angle between embedding vectors, with values near 1 showing that tokens appear in similar contexts in the training data and thus likely share meanings. This allows BERTScore to recognize synonyms, hypernyms, and paraphrases that n-gram metrics would miss. However, the limitation remains that BERTScore only compares the generated text to a reference, not to the source documents that should ground the answer. A generated text could reach high BERTScore by closely matching a reference while containing hallucinated details or ignoring the retrieved context entirely.

For RAG, we need metrics that compare generated text against the retrieved context, not just against reference answers. This is a fundamentally different evaluation task: instead of asking "does this answer resemble the gold standard answer?", we ask "does this answer accurately stand for what the source documents say?"

QuestEval and similar question-answering-based approaches use question generation and answering to verify consistency between generated text and source documents. The process works as follows: first, generate specific factual questions from the candidate answer (for example, "When was the treaty signed?" from "The treaty was signed in 1919"). Second, attempt to answer those questions using only the retrieved context. Third, check whether the answers obtained from the context match the original claims in the generated text.

If the context cannot support the claims made in the answer, the system flags a potential hallucination or faithfulness violation. This approach avoids the need for reference answers by treating evaluation as a consistency checking task. By generating specific questions about factual claims, then attempting to answer those questions using only the retrieved documents, the system verifies whether the evidence supports the conclusions drawn by the generator.

The question-answering framework has strong theoretical appeal: it is asking, "could a reasonable reader extract this information from the retrieved sources?" If the answer is no, the generated claim is either a hallucination or an unwarranted inference. If yes, the claim is grounded. This framing aligns closely with how domain experts would evaluate answer quality in a fact-checking context.

A related approach uses Natural Language Inference (NLI) models directly. NLI models are trained to classify whether a "hypothesis" sentence is entailed by (logically follows from), contradicted by, or neutral with respect to a "premise" sentence. For faithfulness evaluation, the premise is a retrieved context passage and the hypothesis is a claim extracted from the generated answer. If all claims in the answer are entailed by at least one context passage, the answer is faithful; if any claims are contradicted, the answer contains errors.