Question Answering › QA Evaluation

Evaluating QA systems requires measuring both answer correctness and the quality of the reasoning or evidence provided. Different metrics suit different QA formats, and no single metric captures everything that matters for a deployed QA system.

A key distinction is between intrinsic evaluation (does the answer match the gold standard?) and extrinsic evaluation (does the answer help the user accomplish their task?). Intrinsic evaluation is easier to automate and is the norm in research. Extrinsic evaluation is what ultimately matters for deployment but requires user studies or production logs. The gap between the two is one reason why models that score well on benchmarks sometimes disappoint in practice.

For span extraction and short-answer generation, two metrics dominate.

Exact Match (EM) scores 1 if the predicted answer exactly matches any of the gold answers (after normalizing whitespace and punctuation), and 0 otherwise:

$$\text{EM} = \mathbb{1}[\text{normalize}(\hat{a}) = \text{normalize}(a^*)]$$

where:

• $\hat{a}$: the predicted answer produced by the model
• $a^*$: the gold (reference) answer from the annotation
• $\text{normalize}(\cdot)$: a function that lowercases, removes punctuation, and strips articles ("a", "an", "the") for a fair string comparison
• $\mathbb{1}[\cdot]$: the indicator function, returning 1 if the condition holds and 0 otherwise

EM is strict: "Albert Einstein" and "Einstein" would not match even if both are correct. Normalization removes articles, punctuation, and lowercases. When multiple gold answers exist (different annotators may have written "the 26th of July 1953" and "July 26th, 1953"), the model is evaluated against each gold answer and the maximum score is taken.

Token-level F1 computes overlap between predicted and gold answer at the token level. Let $P$ be the set of tokens in the predicted answer and $G$ the set in the gold answer. Precision, recall, and F1 are:

$$\text{Precision} = \frac{|P \cap G|}{|P|}, \quad \text{Recall} = \frac{|P \cap G|}{|G|}$$ $$F1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

where:

• $P$: the multiset of tokens in the predicted answer
• $G$: the multiset of tokens in the gold answer
• $|P \cap G|$: the number of tokens appearing in both (counting duplicates appropriately via multiset intersection)
• Precision measures how many predicted tokens are correct; recall measures how many gold tokens were found

F1 is more lenient than EM and handles partial credit. If the gold answer is "the 26th of July, 1953" and the predicted answer is "July 26, 1953", EM scores 0 but F1 may score 0.5 or higher depending on tokenization.

The choice between EM and F1 depends on how strict you want to be. EM is appropriate when exact format matters (a date, a number, a proper name). F1 is more forgiving and is the primary metric on SQuAD because human annotators sometimes write answers at different levels of granularity. Both metrics have a fundamental limitation: they measure surface string similarity, not semantic correctness. An answer can be semantically correct but surface-form different from the gold answer and score poorly on both metrics.

For generative QA producing paragraph-length answers, ROUGE measures n-gram overlap between generated and reference answers. ROUGE-1 counts unigram overlap, ROUGE-2 counts bigram overlap, and ROUGE-L measures the longest common subsequence, a sequence-level measure that captures in-order word matches without requiring exact n-gram alignment.

These metrics were originally developed for summarization (as we covered in the previous chapter) and carry the same limitations when applied to QA. High ROUGE scores can be achieved by an answer that correctly paraphrases the reference, but also by an answer that contains many of the same words while making a completely different claim. If the gold answer is "the experiment failed because the temperature exceeded 300 degrees" and the model generates "the experiment succeeded, and the temperature exceeded 300 degrees," ROUGE would score this highly despite the factual reversal. For questions with objective answers, this gap between overlap and correctness is particularly problematic.

ROUGE is most useful as a coarse filter: an answer with very low ROUGE scores is almost certainly wrong. But high ROUGE scores do not guarantee correctness, and for factual QA tasks, ROUGE is often supplemented with EM/F1 on key facts or with LLM-as-judge evaluation.

BERTScore computes soft token-level similarity using contextual embeddings from a pretrained language model. For each token in the predicted answer, it finds the most similar token in the reference, and vice versa:

$$\text{BERTScore}_F = \frac{2}{\left|P\right|+\left|R\right|} \left( \sum_{p_i \in P} \max_{r_j \in R} \cos(\mathbf{e}_{p_i}, \mathbf{e}_{r_j}) + \sum_{r_j \in R} \max_{p_i \in P} \cos(\mathbf{e}_{r_j}, \mathbf{e}_{p_i}) \right)$$

where:

• $P$: the set of tokens in the predicted answer
• $R$: the set of tokens in the reference (gold) answer
• $\mathbf{e}_{p_i}, \mathbf{e}_{r_j} \in \mathbb{R}^d$: contextual embeddings of tokens $p_i$ and $r_j$ from a pretrained language model (typically RoBERTa)
• $\cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}$: cosine similarity between two embedding vectors
• The first sum computes precision: for each predicted token, find the most similar reference token; the second sum computes recall: for each reference token, find the most similar predicted token

The $\max$ operator performs "soft matching": instead of requiring exact token matches, BERTScore credits a predicted token for being semantically similar to any reference token, even if the surface forms differ.

BERTScore correlates better with human judgments than n-gram metrics because it captures semantic similarity: "automobile" and "car" would receive high similarity under BERTScore but zero overlap under EM or ROUGE. For QA evaluation, BERTScore is useful when answers can be paraphrased in multiple valid ways and you want to reward semantic equivalence without requiring exact string matches.

A practical concern with BERTScore is the choice of pretrained model. Different BERT variants produce different similarity scores, and the metric's correlation with human judgments varies across tasks and model choices. The standard recommendation is to use the same language model family as your task domain when possible.

For complex generative QA, automated reference-based metrics often fail to capture what matters: factual accuracy, relevance, completeness, and coherence. An increasingly common approach uses an LLM as a judge. A powerful model (GPT-4, Claude Opus) is prompted to determine whether the generated answer correctly addresses the question:

"Given the question: [q]. Is the following answer correct, partially correct, or incorrect? Answer: [a]. Rate and explain."

The judge model reads both the question and the generated answer, then produces a rating with a natural language explanation. This is far more flexible than n-gram overlap: the judge can recognize that "the author of Hamlet" and "William Shakespeare" are equivalent answers, can detect factual errors that no reference text would catch, and can assess whether an answer appropriately hedges on uncertain information.

LLM-as-judge correlates well with human judgments on complex QA tasks, particularly for open-ended questions where reference answers are inherently subjective or admit multiple valid aspects. The main concern is systematic biases in the judge model: LLMs may prefer longer, more verbose answers (verbosity bias), may favor answers written in their own training style, and may rate their own outputs more favorably (self-enhancement bias). When using a judge model to compare two systems, using the same judge for both sides helps control for consistent biases, but does not eliminate them.

A practical variant is reference-free evaluation, where the judge is also provided the retrieved context and asked to verify whether the answer is faithful to the context. This "faithfulness" score separates grounding quality (does the answer follow from the provided evidence?) from correctness (is the evidence itself accurate?). Both dimensions matter for reliable QA deployment. A system can be highly faithful to a wrong document, or can be factually correct by drawing on parametric memory rather than the provided context. Measuring both independently gives a more complete picture of system behavior.

In ODQA pipelines, you can separately evaluate retrieval quality. This matters because the reader's performance is bounded by whether the correct passage was ever retrieved.

Two metrics are standard:

• Recall@k: Among the top-$k$ retrieved passages, what fraction contain the answer? Measures whether the correct passage is even retrieved. A Recall@1 of 0.6 means the top-ranked passage contains the answer 60% of the time; Recall@10 is typically much higher because more chances are given.
• Mean Reciprocal Rank (MRR): $\text{MRR} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\text{rank}_q}$, where $\text{rank}_q$ is the position of the first relevant passage for question $q$. Rewards having the relevant passage ranked highest.

Recall@k and MRR diagnose whether errors come from retrieval (the right passage was never fetched) or reading (the right passage was retrieved but the reader still got it wrong). This decomposition guides where to focus improvement efforts. If Recall@5 is high but end-to-end accuracy is low, the problem is reading comprehension: the relevant passage is being retrieved but the reader is not extracting the answer correctly. If Recall@5 is low, improving retrieval is the most effective intervention. This diagnostic approach, measuring retrieval and reading independently, is a practical engineering discipline that distinguishes teams that iterate efficiently from teams that optimize the wrong component.