RAG Evaluation: Metrics for Retrieval and Generation Quality › Worked Example: Step-by-Step Metric Calculation

Let us walk through a complete concrete example that calculates every metric we have discussed, showing each numerical step. This kind of hands-on calculation is the best way to develop intuition for how the metrics behave and where they agree or disagree in their verdicts.

Setup: A user asks "What carbon pricing mechanism did Canada implement in 2023?"

Our retriever returns five chunks, ranked in order:

• Chunk 1 (rank 1): "The European Union's carbon border adjustment mechanism affects imports from countries without carbon pricing..." (Irrelevant to Canada 2023)
• Chunk 2 (rank 2): "Canada implemented a federal carbon pricing system in 2019 that applies a charge on fossil fuels..." (Relevant but discusses 2019, not 2023)
• Chunk 3 (rank 3): "In 2023, Canada introduced a carbon contract for difference program to guarantee carbon prices for industrial emitters..." (Highly relevant: directly answers the question)
• Chunk 4 (rank 4): "Carbon pricing mechanisms vary across jurisdictions, including carbon taxes, cap-and-trade, and contracts for difference..." (Marginally relevant: explains the mechanism type but not specific to Canada)
• Chunk 5 (rank 5): "Canadian provinces have implemented their own clean fuel standards separate from federal carbon pricing..." (Marginally relevant: related to Canada's climate policy generally)

We assign graded relevance: Chunk 1 = 0, Chunk 2 = 2, Chunk 3 = 3, Chunk 4 = 1, Chunk 5 = 1. Binary relevance: Chunks 2, 3, 4, 5 are relevant (grades > 0), Chunk 1 is irrelevant.

Step 1: Precision@K and Recall@K

At $K=3$: retrieved set is {Chunk1, Chunk2, Chunk3}, relevant set in top 3 is {Chunk2, Chunk3} (2 items).

$$\text{Precision@3} = \frac{|\{C2, C3\}|}{3} = \frac{2}{3} \approx 0.667$$

Total relevant items in corpus (grades > 0): 4. Relevant items in top 3: 2.

$$\text{Recall@3} = \frac{2}{4} = 0.5$$

At $K=5$: all 5 chunks retrieved, 4 are relevant.

$$\text{Precision@5} = \frac{4}{5} = 0.8 \qquad \text{Recall@5} = \frac{4}{4} = 1.0$$

Notice the tradeoff: increasing $K$ from 3 to 5 improved recall from 0.5 to 1.0 and also improved precision from 0.667 to 0.8 in this example (because chunks 4 and 5 happen to be relevant). In general, precision would be expected to decrease or stay flat as K increases.

Step 2: MRR

The first relevant chunk is at rank 2 (Chunk 2 is relevant with grade 2).

$$\text{MRR} = \frac{1}{\text{rank}_1} = \frac{1}{2} = 0.5$$

An MRR of 0.5 shows that the system placed an irrelevant EU document at the top position, making the user "wait" until position 2 to find something useful.

Step 3: NDCG@5

Using the graded relevance scores [0, 2, 3, 1, 1] for ranks 1 through 5:

$$\begin{aligned} \text{DCG@5} &= \frac{2^0 - 1}{\log_2(2)} + \frac{2^2 - 1}{\log_2(3)} + \frac{2^3 - 1}{\log_2(4)} + \frac{2^1 - 1}{\log_2(5)} + \frac{2^1 - 1}{\log_2(6)} \\ &= \frac{0}{1} + \frac{3}{1.585} + \frac{7}{2} + \frac{1}{2.322} + \frac{1}{2.585} \\ &= 0 + 1.893 + 3.5 + 0.431 + 0.387 \\ &= 6.211 \end{aligned}$$

The ideal ranking places scores in descending order: [3, 2, 1, 1, 0].

$$\begin{aligned} \text{IDCG@5} &= \frac{2^3 - 1}{\log_2(2)} + \frac{2^2 - 1}{\log_2(3)} + \frac{2^1 - 1}{\log_2(4)} + \frac{2^1 - 1}{\log_2(5)} + \frac{2^0 - 1}{\log_2(6)} \\ &= \frac{7}{1} + \frac{3}{1.585} + \frac{1}{2} + \frac{1}{2.322} + \frac{0}{2.585} \\ &= 7 + 1.893 + 0.5 + 0.431 + 0 \\ &= 9.824 \end{aligned}$$ $$\text{NDCG@5} = \frac{6.211}{9.824} \approx 0.632$$

The NDCG of 0.632 tells us the system achieved 63.2% of the theoretically achievable DCG. The main losses came from placing the highly relevant Chunk 3 at rank 3 instead of rank 1, and placing the irrelevant EU document at rank 1.

Step 4: End-to-End Metrics

The generator, using all five chunks, produces: "Canada introduced a carbon contract for difference program in 2023 to give price certainty for industrial carbon emitters. This builds on Canada's earlier carbon pricing system established in 2019."

We evaluate the claims in this answer:

• Claim 1 ("Canada introduced a carbon contract for difference program in 2023"): supported by Chunk 3 (direct match).
• Claim 2 ("to give price certainty for industrial carbon emitters"): supported by Chunk 3 ("guarantee carbon prices for industrial emitters").
• Claim 3 ("This builds on Canada's earlier carbon pricing system established in 2019"): supported by Chunk 2 ("Canada implemented a federal carbon pricing system in 2019").

All three claims are supported. Faithfulness = 3/3 = 1.0.

Context Precision at $K=5$: 4 of 5 chunks were relevant (all except Chunk 1). Context Precision = 4/5 = 0.8.

Answer Relevancy: The answer directly addresses the mechanism (carbon contracts for difference) and the year (2023). Answer Relevancy would be high, approximately 0.9 on a 0-1 scale.

Synthesis: This example reveals a fine-grained picture. Retrieval quality is suboptimal (MRR = 0.5, NDCG = 0.632) because the most relevant document was placed at rank 3 and an irrelevant document at rank 1. Yet the generator achieved perfect faithfulness and high answer relevancy by correctly identifying and using the right information from the context. This shows how reliable generation can partially compensate for imperfect retrieval, though ideally both components would perform optimally. In a production system, the low MRR and suboptimal NDCG would motivate improvements to the retrieval component, particularly better discrimination between the EU carbon mechanism (irrelevant) and Canada-specific content.