RAG Evaluation: Metrics for Retrieval and Generation Quality › Code Implementation
Language AI HandbookAll chapters RAG Evaluation: Metrics for Retrieval and Generation Quality › Code ImplementationOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us implement these evaluation metrics from first principles, then show the RAGAS framework approach. We will create a minimal RAG evaluation pipeline and compute all the metrics discussed.
First, we establish our imports and set up the environment:
Now we define the retrieval metrics. We implement Precision@K, Recall@K, MRR, and MAP as standalone functions that operate on lists of document identifiers:
from typing import List, Set
def precision_at_k(retrieved: List[str], relevant: Set[str], k: int) -> float:
"""Calculate Precision@K for binary relevance judgments."""
if k == 0:
return 0.0
retrieved_k = set(retrieved[:k])
relevant_retrieved = retrieved_k.intersection(relevant)
return len(relevant_retrieved) / len(retrieved_k)
def recall_at_k(retrieved: List[str], relevant: Set[str], k: int) -> float:
"""Calculate Recall@K."""
if not relevant:
return 0.0
retrieved_k = set(retrieved[:k])
relevant_retrieved = retrieved_k.intersection(relevant)
return len(relevant_retrieved) / len(relevant)
def mean_reciprocal_rank(
retrieved_list: List[List[str]], relevant_list: List[Set[str]]
) -> float:
"""Calculate MRR across multiple queries."""
rr_sum = 0.0
for retrieved, relevant in zip(retrieved_list, relevant_list):
rank = None
for i, doc in enumerate(retrieved, 1):
if doc in relevant:
rank = i
break
if rank:
rr_sum += 1.0 / rank
return rr_sum / len(retrieved_list) if retrieved_list else 0.0
def average_precision(retrieved: List[str], relevant: Set[str]) -> float:
"""Calculate Average Precision for a single query."""
if not relevant:
return 0.0
hits = 0
sum_precisions = 0.0
for i, doc in enumerate(retrieved, 1):
if doc in relevant:
hits += 1
sum_precisions += hits / i
return sum_precisions / len(relevant) if relevant else 0.0
def mean_average_precision(
retrieved_list: List[List[str]], relevant_list: List[Set[str]]
) -> float:
"""Calculate MAP across multiple queries."""
ap_scores = [
average_precision(ret, rel)
for ret, rel in zip(retrieved_list, relevant_list)
]
return np.mean(ap_scores) if ap_scores else 0.0The MAP implementation illustrates the key design decision: we accumulate precision at each relevant document position and normalize by the total number of relevant documents (not the number retrieved). This means systems that miss relevant documents are penalized even if their retrieved set has perfect precision.
Next, we implement NDCG for graded relevance judgments. This requires relevance scores rather than binary labels:
from typing import Dict, List
def dcg_at_k(relevance_scores: List[float], k: int) -> float:
"""Calculate Discounted Cumulative Gain."""
dcg = 0.0
for i, rel in enumerate(relevance_scores[:k], 1):
dcg += (2**rel - 1) / np.log2(i + 1)
return dcg
def ndcg_at_k(
retrieved: List[str], relevance_map: Dict[str, float], k: int
) -> float:
"""Calculate Normalized DCG@K."""
# Get relevance scores for retrieved items (0 if not in map)
retrieved_scores = [relevance_map.get(doc, 0.0) for doc in retrieved[:k]]
# Calculate actual DCG
actual_dcg = dcg_at_k(retrieved_scores, k)
# Calculate ideal DCG (perfect ranking of all relevant documents)
ideal_scores = sorted(relevance_map.values(), reverse=True)[:k]
ideal_dcg = dcg_at_k(ideal_scores, k)
return actual_dcg / ideal_dcg if ideal_dcg > 0 else 0.0The ndcg_at_k function handles an important edge case: if no relevant documents exist (IDCG is zero), we return 0 to avoid division by zero. The ideal ranking uses all relevant documents across the full corpus, not just those that were retrieved. This keeps systems missing relevant documents are penalized.
Now let us create a simulated RAG evaluation dataset and compute these metrics:
# Simulated evaluation data for the Canada carbon pricing example
# Retrieved document IDs (in order of retrieval rank)
retrieved_docs = [
"doc_eu_carbon", # Rank 1: Irrelevant (EU, not Canada)
"doc_canada_2019", # Rank 2: Relevant (Canada 2019 system)
"doc_canada_2023", # Rank 3: Highly relevant (Canada 2023 contract)
"doc_carbon_mechanisms", # Rank 4: Marginally relevant
"doc_canada_provinces", # Rank 5: Marginally relevant
]
# Binary relevance judgments
relevant_binary = {
"doc_canada_2019",
"doc_canada_2023",
"doc_carbon_mechanisms",
"doc_canada_provinces",
}
# Graded relevance (0-3 scale)
relevance_graded = {
"doc_canada_2019": 2.0, # Relevant but not 2023-specific
"doc_canada_2023": 3.0, # Directly answers the question
"doc_carbon_mechanisms": 1.0, # Background context only
"doc_canada_provinces": 1.0, # Related but not answering
}K=3: Precision=0.667, Recall=0.500, NDCG=0.574 K=5: Precision=0.800, Recall=1.000, NDCG=0.632 MRR: 0.500
We can see that the MRR of 0.5 shows the irrelevant EU document at rank 1, while NDCG captures both this ranking problem and the sub-optimal placement of the most relevant document at rank 3. These metrics together identify the retrieval weaknesses more precisely than any single number could.
Now we implement a simplified faithfulness checker using an entailment heuristic. In practice, this would use NLI models or LLM prompting, but the word-overlap heuristic shows the core logic:
def extract_claims(answer: str) -> List[str]:
"""Simple sentence segmentation to extract claims."""
import re
sentences = re.split(r"(?<=[.!?])\s+", answer)
return [s.strip() for s in sentences if len(s.strip()) > 10]
def check_entailment_simple(claim: str, context: str) -> bool:
"""Simplified entailment check using word overlap.
In production, replace with an NLI model or LLM judge.
Word overlap is a rough proxy that works for clear entailment
and clear contradiction but struggles with paraphrases.
"""
claim_words = set(claim.lower().split())
context_words = set(context.lower().split())
if not claim_words:
return False
overlap = len(claim_words.intersection(context_words))
return overlap / len(claim_words) > 0.7 # 70% word overlap heuristic
def faithfulness_score(answer: str, contexts: List[str]) -> float:
"""Calculate faithfulness as ratio of supported claims."""
claims = extract_claims(answer)
if not claims:
return 0.0
supported = 0
for claim in claims:
# Check if claim is supported by ANY context chunk
if any(check_entailment_simple(claim, ctx) for ctx in contexts):
supported += 1
return supported / len(claims)Faithful answer score: 1.00 Hallucinated answer score: 0.00
The faithfulness checker correctly distinguishes between these cases: the faithful answer shares key terms ("Paris", "capital", "France", "987") with the context, while the hallucinated answer claims 1789 (a date not present in the context). In practice, the 70% word overlap threshold would be replaced by a proper entailment model that handles paraphrasing and implicit entailment.
Now let us show a complete RAGAS-style evaluation pipeline using an LLM judge simulation based on TF-IDF similarity:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
class SimpleRAGASEvaluator:
"""Simplified RAGAS evaluator using heuristic judges.
This class simulates the RAGAS framework using lightweight
heuristics. For production use, replace each method's
similarity computation with actual LLM judge calls.
"""
def context_precision(self, query: str, retrieved: List[str]) -> float:
"""Fraction of retrieved chunks relevant to query."""
# Simulate LLM judgment with keyword overlap
query_words = set(query.lower().split())
relevant_count = 0
for doc in retrieved:
doc_words = set(doc.lower().split())
overlap = len(query_words.intersection(doc_words))
if overlap > 0:
relevant_count += 1
return relevant_count / len(retrieved) if retrieved else 0.0
def answer_relevance(self, query: str, answer: str) -> float:
"""Semantic similarity between query and answer via TF-IDF cosine."""
vectorizer = TfidfVectorizer()
try:
vectors = vectorizer.fit_transform([query, answer])
similarity = cosine_similarity(vectors[0:1], vectors[1:2])[0][0]
return float(similarity)
except Exception:
return 0.0
def faithfulness(self, answer: str, contexts: List[str]) -> float:
"""Check if answer is grounded in contexts using word overlap."""
return faithfulness_score(answer, contexts)RAGAS-style Evaluation: Context Precision: 1.00 Answer Relevance: 0.72 Faithfulness: 1.00 Interpretation: All 3 chunks mention 'France', so context precision is 1.0. The answer directly addresses the query, so answer relevance is high. The answer is grounded in the first retrieved chunk.
The heatmap visualization makes the multiplicative nature of RAG performance viscerally clear. Even perfect generation (right column, generation quality 1.0) cannot rescue poor retrieval: with retrieval quality 0.2, end-to-end performance is capped at 0.18. Similarly, perfect retrieval cannot save poor generation. Both components must work well simultaneously for the system to deliver high-quality answers.