QA: Extractive, Generative, and Open-Domain QA

Question answering (QA) is one of the most direct tests of language understanding: given a question in natural language, produce a correct answer. Unlike summarization, which we covered in the previous chapter, QA requires locating or generating a specific piece of information rather than compressing a body of text. The task sounds simple but draws on reading comprehension, information retrieval, reasoning, and knowledge representation, challenges that have occupied NLP researchers for decades.

The field has evolved through several distinct paradigms. Early QA systems matched patterns and templates against curated databases. The rise of the internet shifted attention to open-domain QA, where answers could be found anywhere across millions of documents. The transformer era changed QA yet again: first by teaching models to identify answer spans within passages, then by enabling models to synthesize answers from their parametric memory without any retrieved context at all.

Today, QA systems span a wide range of designs. At one end sits extractive QA, where a model reads a passage and highlights the text span that answers the question. At the other end sits generative QA, where a model composes an answer from scratch, potentially drawing on retrieved documents, reasoning chains, or encoded knowledge. In between lies a rich space of hybrid approaches. Understanding the differences (when to extract versus generate, when to retrieve versus rely on memory) is central to building effective QA systems.

This chapter traces that evolution. We cover the mechanics of each approach, how to evaluate QA systems rigorously, and how to build both extractive and generative QA pipelines in code. By the end, you will understand how to plug in a pre-built QA model, why each design decision was made, and what consequences it carries.

The earliest QA systems, like BASEBALL (1961) and LUNAR (1972), were domain-specific and rule-based. BASEBALL could answer questions about American baseball game results by querying a structured database. LUNAR answered questions about moon rocks by translating natural language into database queries. These systems worked well within their narrow domains but required hand-engineered rules and could not generalize beyond the curated knowledge they were built around.

The reason these systems were brittle is fundamental: language is infinitely variable, but a hand-written rule set is finite. The same fact can be expressed hundreds of different ways, and a rule that matches "How many home runs did Babe Ruth hit?" fails on "What was Ruth's career home run total?" even though both questions ask for the same thing. Engineers compensated by writing more and more rules, but coverage never caught up with the diversity of real questions.

The question of how to move beyond curated databases led researchers toward text corpora. If answers exist somewhere in documents, the task becomes: find the right document, then find the answer within it. This two-stage structure, retrieve then read, became the dominant paradigm for open-domain QA and persists in modern retrieval-augmented systems. The insight was deceptively simple: rather than building a knowledge base by hand, use the vast body of text already written by humans as the knowledge source.

The "read" step, however, required models that could comprehend text, not just match keywords. This challenge became formalized as machine reading comprehension (MRC). The release of large reading comprehension datasets crystallized progress: SQuAD (Stanford Question Answering Dataset), released in 2016, provided over 100,000 question-passage pairs where the answer was a span of text from the passage. Models had to select exactly the right span, making evaluation straightforward and enabling rapid, reproducible progress.

SQuAD and its successors (SQuAD 2.0, TriviaQA, Natural Questions, QuALITY) drove a wave of architectures designed to solve span extraction. Bidirectional Attention Flow (BiDAF), QANet, and eventually BERT-based models progressively improved performance. When BERT was fine-tuned on SQuAD in 2018, it matched human-level performance on the benchmark, a milestone that marked the maturation of extractive QA as a solved problem on that particular benchmark, though far harder problems lay ahead.

It is worth pausing to appreciate what "human-level performance on SQuAD" means. The benchmark measures agreement between model output and human-annotated answer spans. A model that scores above the human agreement threshold has, in a narrow technical sense, matched human accuracy at selecting answer spans in Wikipedia passages. This is a concrete benchmark achievement. But it does not mean the model has solved reading comprehension in general: SQuAD passages are from Wikipedia, questions were posed by annotators who had already read the passage, and every question has a guaranteed answer in the text. Harder datasets like adversarial SQuAD, TriviaQA, and Natural Questions revealed that models had partially learned to exploit surface-level patterns rather than deeply understanding the passage.

Extractive QA takes a specific form: given a question $q$ and a context passage $c$, find the substring of $c$ that answers $q$. The answer is not generated. It is identified as a contiguous token span $[i, j]$ within $c$, where $i$ is the start index and $j$ is the end index.

This formulation has several advantages. The answer is guaranteed to come from the passage, making hallucination impossible. Evaluation is easy: compare predicted span to gold span. And the task can be framed as a simple classification problem over token positions, which fits naturally into the fine-tuning paradigm for pretrained language models. The model does not need to invent new words; it only needs to recognize which words in the provided passage are the answer.

The constraint that the answer must be a contiguous span from the passage is also a limitation. Many real questions do not have answers that appear verbatim in text. Counting questions ("How many planets are in the solar system?") may require numerical computation. Comparative questions ("Which country has a larger population, India or China?") require comparing values from possibly different parts of the text. Abstractive questions ("What caused World War I?") require synthesizing information into a narrative. Extractive QA handles these poorly or not at all. But for the enormous class of factoid questions ("Who invented the telephone?", "When did the French Revolution begin?", "What is the capital of Australia?"), span extraction works very well.

Given a tokenized passage of length $n$, extractive QA reduces to predicting two integer positions:

where $n$ is the number of tokens in the passage, $P_{\text{start}}$ is the index of the first answer token, and $P_{\text{end}}$ is the index of the last answer token. Together they define a contiguous span within the passage.

A BERT-based model encodes the concatenated question and passage, then applies two learned linear layers over the output representations to compute start and end logit scores for each token position:

where:

• $\mathbf{h}_i \in \mathbb{R}^d$: the hidden state vector at position $i$ from the encoder, with dimension $d$ matching the model's hidden size
• $\mathbf{w}_s, \mathbf{w}_e \in \mathbb{R}^d$: learned weight vectors, one for start positions and one for end positions
• $s_i, e_j \in \mathbb{R}$: scalar logit scores; higher values mean the model believes position $i$ is the start (or $j$ is the end) of the answer

These linear projections are the only new parameters added on top of the pretrained encoder. Everything else (the self-attention layers, feed-forward networks, layer normalization) carries pretrained weights that are fine-tuned with the QA data. The small number of task-specific parameters means the model adapts quickly with relatively little labeled data, often achieving strong performance with just a few thousand training examples.

The scores are then converted to probabilities via softmax over all positions in the passage:

The denominator sums over all token positions, making sure the probabilities sum to 1 across the passage. The model selects the span $(i, j)$ that maximizes the joint score $P(\text{start} = i) \cdot P(\text{end} = j)$, subject to $i \leq j$ and $j - i < L_{\max}$ (a maximum span length constraint, typically 30-50 tokens, to prevent selecting absurdly long spans).

One important subtlety is that the start and end probabilities are computed independently: the model assigns a probability to each position being the start, and separately assigns a probability to each position being the end. The joint selection enforces the constraint that the end must come after the start, but the model does not directly model the joint distribution $P(\text{start} = i, \text{end} = j)$. This independence assumption is computationally convenient and works well in practice, since the relevant span is usually short and the contextual embeddings carry enough information that start and end predictions are implicitly coordinated.

During training, the model minimizes the sum of cross-entropy losses over start and end positions:

where $i^*$ and $j^*$ are the ground-truth start and end positions annotated in the training data. Minimizing this loss pushes the model to assign maximum probability to the correct span boundaries.

The positional relationship between the question and passage matters. Because BERT uses bidirectional attention, every token in the passage attends to every token in the question, and vice versa. This cross-attention between question and passage allows the model to compute question-conditioned representations of each passage token. The hidden state $\mathbf{h}_i$ for a passage token represents both that token and how the token relates to the question. A token that answers the question will have a hidden state shaped by the question's content, which the linear start/end classifiers can then detect.

SQuAD 1.1 guaranteed that every question had an answer in the passage. SQuAD 2.0 added roughly 50,000 adversarially crafted unanswerable questions, plausible-looking enough that a shallow model might confabulate an answer. Questions like "What year was Einstein born in Paris?" are unanswerable from a passage that mentions Einstein's birth in Ulm, Germany. A model that always extracts some span will get these wrong by design.

This change was motivated by a practical concern: deployed QA systems must know when to abstain. A system that always produces an answer is dangerous if the answer is fabricated. SQuAD 2.0 forced models to reason about whether the question is even answerable from the given passage, not just which span is most likely.

Models trained on SQuAD 2.0 learn to produce a "no answer" prediction when confident. The standard approach adds an "answerability score" (the probability that the question is answerable at all) computed from the [CLS] token or as a separate classification head. If the answerability score falls below a threshold, the model outputs "No answer".

This threshold controls a tradeoff: a low threshold produces fewer false "no answer" predictions but more hallucinated spans; a high threshold produces more "no answer" predictions, even for answerable questions. Calibrating this threshold for a specific application requires evaluation on a held-out set representative of the deployment distribution. In a medical QA system where a wrong answer is more dangerous than no answer, you set the threshold high. In a general search assistant where users prefer a possibly wrong answer over no answer, you set it lower.

The answerability prediction also raises an interesting question about what the model has learned. To correctly identify an unanswerable question, the model must understand what information is in the passage and what information is missing. This is harder than simply finding a matching span, and models that score well on SQuAD 2.0 have developed more reliable passage comprehension than their SQuAD 1.1-only counterparts.

In practice, you rarely have the correct passage pre-selected. In open-domain extractive QA, the system must first retrieve relevant passages from a large corpus (Wikipedia, a document collection, the web) before running the reader.

This retrieval step uses one of several strategies:

• Sparse retrieval: TF-IDF or BM25 matches query terms to document terms. Fast and interpretable, but fails on vocabulary mismatch (the question uses "death" where the document uses "passed away"). BM25 is the standard sparse baseline, and it remains competitive for many retrieval tasks despite its simplicity.
• Dense retrieval: Encode question and passages using a bi-encoder neural network; retrieve by nearest-neighbor search in embedding space. Handles synonymy and paraphrase, but requires building a vector index. Dense retrieval became dominant after Dense Passage Retrieval (DPR) was introduced in 2020.
• Hybrid retrieval: Combine sparse and dense scores. Often outperforms either alone, since sparse retrieval captures exact keyword matches while dense retrieval captures semantic similarity; combining both recovers the strengths of each approach.

The full pipeline is: retrieve top-$k$ passages, run the reader on each passage independently, then select the best answer span across all passages using the span confidence scores. The reader does not know which passage came from which document; it simply processes each passage as an independent context and reports which span it finds most likely to be the answer.

One practical concern is passage length. BERT-based readers typically cap at 512 tokens. Wikipedia passages are usually chunked into 100-word blocks for retrieval to ensure they fit within this window. Longer documents require sliding-window approaches, where the same document is processed in overlapping windows and the best span across all windows is selected.

Extractive QA is constrained by a strong assumption: the answer exists verbatim in the provided text. Many real questions violate this. "What year did Napoleon invade Russia?" has a clean extractable answer, but "Why did Napoleon's Russian campaign fail?" requires synthesizing information from multiple sentences or documents. And many questions about the world don't have a single canonical passage to extract from.

Generative QA lifts the span constraint: instead of identifying a substring, the model generates a token sequence. This allows answers that aggregate information across sentences or documents, answers that require inference or reasoning, and answers expressed in the model's own words, potentially more clearly than the source text.

The tradeoff is that generative models can hallucinate, producing plausible-sounding but incorrect answers. Managing hallucination is the central challenge of generative QA. A model that generates "The French Revolution began in 1799" instead of 1789 produces a fluent, confident, wrong answer that no extractive system would produce, because the correct date appears in most relevant passages as a direct quote.

Understanding when to use generative versus extractive QA is a design decision every practitioner faces. The key question is: does the correct answer appear verbatim in the source text? If yes, extractive QA is safer. If the answer requires synthesis, inference, or natural language composition, generative QA is necessary.

Generative QA divides into two settings based on whether the model has access to external context at inference time.

Closed-book QA queries the model directly with no supporting documents:

Question: What is the capital of France? Answer:

The model must answer from its parametric knowledge, the information encoded in its weights during pretraining. Surprisingly, large language models trained on web-scale text can answer many factual questions accurately this way. Roberts et al. (2020) showed that T5 with 11 billion parameters achieved competitive performance on open-domain QA benchmarks without any retrieval, leading to the "language model as a knowledge base" framing.

The intuition behind closed-book QA is that a language model trained on a trillion tokens of web text has absorbed an enormous amount of factual knowledge. The act of predicting missing words in text requires encoding information about entities, relationships, dates, and events. A model that can accurately predict the next word in "The capital of France is ___" must have encoded the fact that Paris is the capital of France. This implicit knowledge storage is a byproduct of language modeling, not something explicitly designed in.

Closed-book QA is simple (no retrieval infrastructure needed) but brittle. Knowledge encoded in weights reflects training data cutoffs, may be inconsistently distributed across facts, and is hard to update. If a fact changes (a company is acquired, a political leader changes, a scientific consensus shifts), retraining the entire model is the only way to update the knowledge. The model cannot cite sources, making it impossible to verify claims or trace where an answer came from.

Open-book QA provides supporting documents at inference time. The model's task is to read the context and synthesize an answer:

Context: [retrieved passage(s)] Question: [question] Answer:

This is also called retrieval-augmented generation (RAG) when the retrieval step is part of the pipeline. Open-book QA is more reliable for factual questions because the answer is grounded in retrieved evidence. The model acts as a reader and synthesizer, not as a memorized knowledge store. When the answer changes, you update the document corpus, not the model weights.

The distinction between closed-book and open-book QA matters practically. For a customer support chatbot that needs to answer questions about product documentation that changes frequently, closed-book QA is inappropriate: the model's weights would quickly become stale as documentation is updated. Open-book QA over a regularly refreshed document store is the right architecture. For a general trivia assistant where the facts are relatively stable (historical events, scientific constants, geographical facts), closed-book QA from a large LLM can work well with simpler infrastructure.

Before large decoder-only models dominated the field, encoder-decoder architectures like T5 and BART were the standard approach for generative QA. These models treat QA as a sequence-to-sequence task: the encoder reads the question (and optionally a context passage), the decoder generates the answer token by token.

As we discussed in the chapters on encoder-decoder architectures and T5, these models use cross-attention to let the decoder attend to the encoder's output at every generation step. For QA, this means every generated answer token is informed by the full question and context, rather than only the tokens generated so far.

Fine-tuning T5 on a QA dataset is straightforward: format inputs as "question: [q] context: [c]" and targets as the answer text. T5's text-to-text framing makes this natural, since every NLP task becomes a text generation problem. The same model trained on question generation (generating questions from passages) or summarization can be repurposed for QA with different training data and input formatting.

The decoder generates answers autoregressively. At each step $t$, it computes:

where:

• $a_t$: the token generated at step $t$
• $a_{<t}$: all previously generated tokens (the partial answer so far)
• $q, c$: the question and context passed to the encoder
• $\mathbf{h}_t^{\text{dec}} \in \mathbb{R}^d$: the decoder hidden state at step $t$, computed by attending to $a_{<t}$ (via masked self-attention) and to the encoder outputs for $(q, c)$ (via cross-attention)
• $\mathbf{W}_o \in \mathbb{R}^{|V| \times d}$: the output projection matrix that maps the hidden state to logits over the vocabulary of size $|V|$

The softmax converts the logits to a probability distribution, from which the next token is sampled or the most likely token selected (greedy decoding). This continues until the model generates an end-of-sequence token.

The full answer is generated by sampling or beam search until the model produces an end-of-sequence token. Beam search maintains the $B$ most likely partial sequences at each step, where $B$ is the beam size, exploring more candidates than greedy decoding while still being computationally tractable. For QA, where answers are typically short (one to three sentences), beam search with $B = 4$ or $B = 5$ is standard.

The power of this approach is that the answer is not constrained to any span in the input. The model can produce "Paris" even if the passage mentions "the French capital" or "the City of Light." It can infer "steel" from a passage that describes the material's iron and carbon composition without ever using the word "steel." This abstraction and paraphrase capability is what generative QA enables that extractive QA cannot.

Modern instruction-tuned LLMs (GPT-4, Claude, LLaMA with instruction fine-tuning) have largely replaced specialized QA architectures for many applications. These models are trained to follow natural language instructions, so QA becomes a special case of instruction following:

"Answer the following question based on the provided context: [context]. Question: [question]"

The model responds with a generated answer. Prompt engineering (structuring the context, question, and any formatting instructions carefully) becomes the primary lever for improving quality.

Instruction-tuned models excel at multi-step reasoning, clarifying ambiguous questions, acknowledging uncertainty, and explaining their reasoning. A good instruction-tuned model will say "I cannot determine this from the provided context" rather than fabricating an answer, provided the prompt includes an instruction to that effect. This refusal behavior is a form of calibration: the model knows when its confidence is low. They perform well across question types without task-specific fine-tuning, making them attractive for rapid prototyping and for question types that do not fit neatly into extractive or standard generative formats.

The main limitations are cost (large models are expensive to call), latency (generation is slower than span extraction), and the hallucination risk that comes with any generative model. For applications that need to process millions of questions per day at low latency, a fine-tuned smaller model is often preferable to a large general-purpose LLM.

Prompting strategy also has a large effect on quality. Few-shot examples in the prompt (showing the model two or three question-context-answer triplets before the actual question) often improve performance, particularly for formats the model is not familiar with. Chain-of-thought prompting ("Think step by step before answering") improves accuracy on questions that require multi-step reasoning. System prompts that specify the model's role ("You are a helpful assistant that answers questions accurately and cites the passage when possible") encourage faithful answers and appropriate hedging.

Open-domain QA (ODQA) refers to answering questions about virtually any topic, without a pre-selected passage. The system must search a large corpus (typically Wikipedia or the web) to find relevant information before answering.

The challenge has two parts: finding the right information, and extracting or generating the answer from it. Getting either step wrong cascades into a wrong final answer. A perfect reader on the wrong passage returns a wrong answer. A perfect passage in the corpus that was never retrieved might as well not exist. This tight coupling between retrieval and reading quality is what makes ODQA harder than closed-domain QA and why the field has invested so heavily in both retrieval architectures and retrieval-reader joint training.

The classic ODQA architecture, formalized by Danqi Chen et al. (2017) in "Reading Wikipedia to Answer Open-Domain Questions" (DrQA), separates the problem into:

1. Document retrieval: Given question $q$, retrieve top-$k$ relevant passages $\{p_1, \ldots, p_k\}$ from a corpus.
2. Machine reader: Given $q$ and each passage $p_i$, extract or generate the answer $a$.
3. Answer selection: If multiple passages produce candidate answers, select the best one (e.g., by confidence score or answer frequency).

DrQA used TF-IDF retrieval and a BiDAF-based reader. Modern implementations replace both stages with neural alternatives: dense passage retrieval (DPR) for retrieval, and BERT/T5 for reading.

The retrieve-then-read pipeline makes each stage independently optimizable. You can improve retrieval without touching the reader, and vice versa. The modular structure also makes the system transparent: you can inspect which passages were retrieved and why. This transparency is valuable in deployed systems, where understanding failure modes requires knowing whether an incorrect answer came from retrieving the wrong passage (retrieval error) or from misreading the correct passage (comprehension error). The two types of failure require different fixes.

Answer selection from multiple passages introduces an additional aggregation challenge. If three passages each produce a candidate answer and the model returns different spans from each, how do you pick the final answer? The simplest approach uses the reader's confidence score: take the span with the highest score regardless of which passage it came from. A more sophisticated approach uses answer voting: if multiple passages yield the same answer string, that answer is weighted more highly. Answer voting provides a form of cross-passage consistency checking that improves reliability, especially on factoid questions where the same fact appears in many documents.

Dense Passage Retrieval (DPR), introduced by Karpukhin et al. (2020), replaces sparse TF-IDF with dense vector similarity. Two BERT encoders (one for questions, one for passages) are trained so that the question embedding and the corresponding answer passage embedding are close in vector space:

where:

• $E_Q(q) \in \mathbb{R}^d$: the dense embedding of the question, produced by the question encoder (a fine-tuned BERT model)
• $E_P(p) \in \mathbb{R}^d$: the dense embedding of passage $p$, produced by the passage encoder (a separate fine-tuned BERT model)
• $\top$: the transpose, making this a dot product between two vectors of dimension $d$ (typically 768 for BERT-base)

Training uses contrastive learning. For each question, one positive passage (contains the answer) and multiple negative passages (don't contain the answer) are provided. The loss encourages the question to be closer to the positive passage than to all negatives:

where:

• $p^+$: the positive passage, i.e., the passage known to contain the answer to question $q$
• $p_1^-, \ldots, p_m^-$: the $m$ negative passages that do not contain the answer (typically sampled from the corpus or from other questions in the same batch)
• The denominator sums the similarity to the positive and all negatives, normalizing the numerator into a probability

This is the InfoNCE (contrastive) loss: it maximizes the log-probability of the positive passage relative to all negatives. Intuitively, it pushes the question embedding toward the positive passage and away from all negative passages in the shared embedding space.

The choice of negative passages matters enormously for DPR training quality. Random negatives (passages sampled at random from the corpus) are easy: the passage about "apple orchards" is obviously not the answer to "What is the capital of France?" Hard negatives are more useful: passages that are superficially relevant (mention France or mention a capital city) but do not contain the answer. Training with hard negatives forces the model to distinguish between passages that mention the right topic and passages that contain the actual answer, which is a much more useful distinction for retrieval.

At inference time, all passages in the corpus are encoded once and stored in a vector index (FAISS is the standard choice). Retrieval becomes a maximum inner product search: find the passage embeddings with the highest dot product to the query embedding. This is orders of magnitude faster than re-encoding on every query. A corpus of 21 million Wikipedia passages encoded with DPR can be searched in milliseconds using approximate nearest-neighbor search, even on CPU.

DPR significantly outperforms BM25 on standard ODQA benchmarks, particularly for questions where the relevant passage uses different vocabulary than the question. A question about "automobile manufacturing" retrieves passages about "car production" because the dense embeddings capture semantic similarity, not just surface string overlap.

A limitation of the retrieve-then-read pipeline is that the reader processes each passage independently. If the answer requires information from multiple passages (e.g., one passage names the person, another gives their dates), span extraction fails and even generative models may struggle without seeing all relevant passages together.

Fusion-in-Decoder (FiD), introduced by Izacard and Grave (2021), addresses this by processing all retrieved passages jointly. Each passage is independently encoded by a T5 encoder, producing a sequence of encoder hidden states. The decoder then attends over the concatenated hidden states from all passages simultaneously:

where:

• $\mathbf{H}_{p_i} \in \mathbb{R}^{L \times d}$: the matrix of encoder hidden states for passage $p_i$, with $L$ being the passage length and $d$ the hidden dimension
• $[\cdot ; \cdot]$: row-wise concatenation, producing a combined matrix of shape $(k \cdot L) \times d$
• $k$: the number of retrieved passages

The decoder cross-attention operates over $\mathbf{H}_{\text{fused}}$, enabling it to synthesize information from multiple passages when generating the answer. This is the key architectural move: the encoder processes passages independently (which is computationally efficient and allows parallel encoding), but the decoder sees all encoder outputs simultaneously (which enables multi-passage synthesis).

FiD achieved state-of-the-art results on Natural Questions and TriviaQA at the time of publication, showing that reading more passages jointly is better than reading them separately and aggregating. The gains are largest on questions where the answer requires combining evidence from multiple sources, confirming the design motivation.

The computational cost scales linearly with the number of passages $k$: encoding $k$ passages requires $k$ forward passes through the encoder. This is manageable because encoder passes can be parallelized, but inference latency grows with $k$. In practice, $k = 100$ retrieved passages is common for FiD, which would be prohibitive for a pipeline that runs each passage through a full reader independently.

Standard QA benchmarks assume the answer is contained in a single passage. Many real questions require more: following a reasoning chain across documents, combining facts from different sources, or tracking entities and events over a long text. These more demanding question types reveal the limits of single-passage architectures and motivate more sophisticated reasoning approaches.

Multi-hop questions require connecting information from two or more passages. The question cannot be answered by any single passage in isolation, as it requires chaining facts together. For example:

"What country was the director of Inception born in?" requires: (1) finding that the director is Christopher Nolan, (2) finding that Nolan was born in London, England.

Neither fact alone answers the question. A retrieval system that fetches passages mentioning "Inception" may retrieve the director's name, but not their birth country. And a retrieval system prompted with the original question may fetch passages about Inception's plot rather than biographical details of its director. The core difficulty is that the retrieval query at step one cannot be formulated correctly until you have the answer from step one.

This is called the bridge entity problem: the first hop retrieves information about a "bridge entity" (Christopher Nolan) that is needed to formulate the query for the second hop. Single-step retrieval systems cannot handle this because the query must encode knowledge that can only be obtained by reading the first passage. A system that retrieves everything in one shot either needs to be lucky (a single document mentions both facts) or needs to retrieve so broadly that it can find both relevant passages by chance.

Answering multi-hop questions with single-passage retrieval fails unless a single document happens to contain both hops. This failure mode is common in practice: encyclopedic articles about a film may mention the director but link to a separate biography page for their background. Dedicated multi-hop datasets like HotpotQA (Yang et al., 2018), MuSiQue, and 2WikiMultiHopQA benchmark this capability and measure whether systems can follow reasoning chains across documents rather than merely retrieving relevant passages.

Approaches to multi-hop QA include:

• Iterative retrieval: Retrieve a first passage, use it to generate a "bridge" query, retrieve a second passage with the bridge query, then answer. This approach requires a model that can identify what information is still missing and reformulate the query accordingly.
• Chain-of-thought reasoning: Prompt LLMs to think step-by-step, generating intermediate reasoning steps that decompose the multi-hop question. The model produces explicit intermediate conclusions before giving the final answer.
• Subquestion decomposition: Decompose the complex question into simpler sub-questions, answer each independently, then combine the sub-answers into a final response.

Chain-of-thought prompting has proven particularly effective for multi-hop QA with capable LLMs. The model writes out reasoning steps before producing the final answer, making errors visible and enabling verification. When a model states "The director of Inception is Christopher Nolan. Christopher Nolan was born in London, England. Therefore, the country is England," each intermediate step can be checked for correctness, unlike a model that produces only the final answer without showing its work.

The value of explicit reasoning chains extends beyond verification. When a model's chain-of-thought contains an error, it often reveals where the model went wrong, enabling targeted correction. A model that confidently states "The director is Nolan Spielberg" in its chain-of-thought is clearly confused, and a downstream verification step can catch this. A model that produces only the final (wrong) answer provides no such signal.

Modern LLMs with large context windows (GPT-4 supports up to 128k tokens, Claude up to 200k) can read much longer documents than earlier systems. This opens the possibility of passing entire documents, or multiple retrieved documents, directly into the context window rather than chunking and retrieving.

The appeal is obvious: if you can fit a 100-page document into the prompt, you never have to worry about whether the retriever found the right chunk. The model can attend to every part of the document directly. This is especially valuable for question types where the relevant information is spread thin across a long document, or where the question cannot be answered without reading the surrounding context.

Long-context QA has its own challenges. The "lost in the middle" phenomenon (Liu et al., 2023) shows that LLMs tend to attend strongly to information at the very beginning and end of long contexts, while underweighting information buried in the middle. In experiments, performance dropped sharply when the relevant passage was positioned in the middle of a long context, even though the information was present. This effect grows stronger as context length increases, suggesting that raw context capacity does not translate directly into uniform reading capability across the entire context window.

The practical implication is that document ordering matters even in long-context systems. Placing the most relevant documents at the beginning or end of a long prompt tends to produce better answers than placing them in the middle. This "primacy and recency" effect mirrors what we know about human working memory and suggests that transformer attention, despite its theoretical ability to attend uniformly to any position, learns positional biases from training data that concentrate on shorter contexts.

Additionally, long-context inference is expensive in both memory and compute. Attention complexity scales quadratically with sequence length in standard transformers, though architectural innovations like sliding window attention (Longformer, BigBird) and efficient attention approximations mitigate this. Retrieval over a long context can still outperform naive full-context reading for questions that depend on information buried deep in a document, by retrieving the relevant section first rather than asking the model to locate it within a sea of text.

The right approach depends on the application. For a few-hundred-page technical manual where any section might be relevant, chunking and dense retrieval followed by focused reading is often more effective than passing the entire document to a long-context model. For a 20-page research paper where the question requires understanding across the full document, long-context reading may be preferable.

Two additional question types challenge standard QA architectures. Temporal questions ask about facts that change over time: "Who is the CEO of Apple?" has different correct answers depending on when the question is asked. Closed-book QA with a model trained before a leadership change will give the old answer. Open-book QA with an up-to-date document corpus will give the current answer, but only if the retriever fetches a passage with a recent publication date. Handling temporal questions correctly requires retrieving relevant information and prioritizing the most recent relevant information.

Ambiguous questions have multiple valid interpretations, each with a different correct answer. "When did they publish the paper?" requires knowing who "they" refers to and which paper. "What is the best treatment for anxiety?" is ambiguous across types and severities of anxiety. Good QA systems should either resolve ambiguity through clarification (if interacting with a user) or acknowledge ambiguity in the answer (if answering automatically). Models that suppress ambiguity by confidently answering one interpretation are unreliable for these question types.

Evaluating QA systems requires measuring both answer correctness and the quality of the reasoning or evidence provided. Different metrics suit different QA formats, and no single metric captures everything that matters for a deployed QA system.

A key distinction is between intrinsic evaluation (does the answer match the gold standard?) and extrinsic evaluation (does the answer help the user accomplish their task?). Intrinsic evaluation is easier to automate and is the norm in research. Extrinsic evaluation is what ultimately matters for deployment but requires user studies or production logs. The gap between the two is one reason why models that score well on benchmarks sometimes disappoint in practice.

For span extraction and short-answer generation, two metrics dominate.

Exact Match (EM) scores 1 if the predicted answer exactly matches any of the gold answers (after normalizing whitespace and punctuation), and 0 otherwise:

where:

• $\hat{a}$: the predicted answer produced by the model
• $a^*$: the gold (reference) answer from the annotation
• $\text{normalize}(\cdot)$: a function that lowercases, removes punctuation, and strips articles ("a", "an", "the") for a fair string comparison
• $\mathbb{1}[\cdot]$: the indicator function, returning 1 if the condition holds and 0 otherwise

EM is strict: "Albert Einstein" and "Einstein" would not match even if both are correct. Normalization removes articles, punctuation, and lowercases. When multiple gold answers exist (different annotators may have written "the 26th of July 1953" and "July 26th, 1953"), the model is evaluated against each gold answer and the maximum score is taken.

Token-level F1 computes overlap between predicted and gold answer at the token level. Let $P$ be the set of tokens in the predicted answer and $G$ the set in the gold answer. Precision, recall, and F1 are:

where:

• $P$: the multiset of tokens in the predicted answer
• $G$: the multiset of tokens in the gold answer
• $|P \cap G|$: the number of tokens appearing in both (counting duplicates appropriately via multiset intersection)
• Precision measures how many predicted tokens are correct; recall measures how many gold tokens were found

F1 is more lenient than EM and handles partial credit. If the gold answer is "the 26th of July, 1953" and the predicted answer is "July 26, 1953", EM scores 0 but F1 may score 0.5 or higher depending on tokenization.

The choice between EM and F1 depends on how strict you want to be. EM is appropriate when exact format matters (a date, a number, a proper name). F1 is more forgiving and is the primary metric on SQuAD because human annotators sometimes write answers at different levels of granularity. Both metrics have a fundamental limitation: they measure surface string similarity, not semantic correctness. An answer can be semantically correct but surface-form different from the gold answer and score poorly on both metrics.

For generative QA producing paragraph-length answers, ROUGE measures n-gram overlap between generated and reference answers. ROUGE-1 counts unigram overlap, ROUGE-2 counts bigram overlap, and ROUGE-L measures the longest common subsequence, a sequence-level measure that captures in-order word matches without requiring exact n-gram alignment.

These metrics were originally developed for summarization (as we covered in the previous chapter) and carry the same limitations when applied to QA. High ROUGE scores can be achieved by an answer that correctly paraphrases the reference, but also by an answer that contains many of the same words while making a completely different claim. If the gold answer is "the experiment failed because the temperature exceeded 300 degrees" and the model generates "the experiment succeeded, and the temperature exceeded 300 degrees," ROUGE would score this highly despite the factual reversal. For questions with objective answers, this gap between overlap and correctness is particularly problematic.

ROUGE is most useful as a coarse filter: an answer with very low ROUGE scores is almost certainly wrong. But high ROUGE scores do not guarantee correctness, and for factual QA tasks, ROUGE is often supplemented with EM/F1 on key facts or with LLM-as-judge evaluation.

BERTScore computes soft token-level similarity using contextual embeddings from a pretrained language model. For each token in the predicted answer, it finds the most similar token in the reference, and vice versa:

where:

• $P$: the set of tokens in the predicted answer
• $R$: the set of tokens in the reference (gold) answer
• $\mathbf{e}_{p_i}, \mathbf{e}_{r_j} \in \mathbb{R}^d$: contextual embeddings of tokens $p_i$ and $r_j$ from a pretrained language model (typically RoBERTa)
• $\cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}$: cosine similarity between two embedding vectors
• The first sum computes precision: for each predicted token, find the most similar reference token; the second sum computes recall: for each reference token, find the most similar predicted token

The $\max$ operator performs "soft matching": instead of requiring exact token matches, BERTScore credits a predicted token for being semantically similar to any reference token, even if the surface forms differ.

BERTScore correlates better with human judgments than n-gram metrics because it captures semantic similarity: "automobile" and "car" would receive high similarity under BERTScore but zero overlap under EM or ROUGE. For QA evaluation, BERTScore is useful when answers can be paraphrased in multiple valid ways and you want to reward semantic equivalence without requiring exact string matches.

A practical concern with BERTScore is the choice of pretrained model. Different BERT variants produce different similarity scores, and the metric's correlation with human judgments varies across tasks and model choices. The standard recommendation is to use the same language model family as your task domain when possible.

For complex generative QA, automated reference-based metrics often fail to capture what matters: factual accuracy, relevance, completeness, and coherence. An increasingly common approach uses an LLM as a judge. A powerful model (GPT-4, Claude Opus) is prompted to determine whether the generated answer correctly addresses the question:

"Given the question: [q]. Is the following answer correct, partially correct, or incorrect? Answer: [a]. Rate and explain."

The judge model reads both the question and the generated answer, then produces a rating with a natural language explanation. This is far more flexible than n-gram overlap: the judge can recognize that "the author of Hamlet" and "William Shakespeare" are equivalent answers, can detect factual errors that no reference text would catch, and can assess whether an answer appropriately hedges on uncertain information.

LLM-as-judge correlates well with human judgments on complex QA tasks, particularly for open-ended questions where reference answers are inherently subjective or admit multiple valid aspects. The main concern is systematic biases in the judge model: LLMs may prefer longer, more verbose answers (verbosity bias), may favor answers written in their own training style, and may rate their own outputs more favorably (self-enhancement bias). When using a judge model to compare two systems, using the same judge for both sides helps control for consistent biases, but does not eliminate them.

A practical variant is reference-free evaluation, where the judge is also provided the retrieved context and asked to verify whether the answer is faithful to the context. This "faithfulness" score separates grounding quality (does the answer follow from the provided evidence?) from correctness (is the evidence itself accurate?). Both dimensions matter for reliable QA deployment. A system can be highly faithful to a wrong document, or can be factually correct by drawing on parametric memory rather than the provided context. Measuring both independently gives a more complete picture of system behavior.

In ODQA pipelines, you can separately evaluate retrieval quality. This matters because the reader's performance is bounded by whether the correct passage was ever retrieved.

Two metrics are standard:

• Recall@k: Among the top-$k$ retrieved passages, what fraction contain the answer? Measures whether the correct passage is even retrieved. A Recall@1 of 0.6 means the top-ranked passage contains the answer 60% of the time; Recall@10 is typically much higher because more chances are given.
• Mean Reciprocal Rank (MRR): $\text{MRR} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\text{rank}_q}$, where $\text{rank}_q$ is the position of the first relevant passage for question $q$. Rewards having the relevant passage ranked highest.

Recall@k and MRR diagnose whether errors come from retrieval (the right passage was never fetched) or reading (the right passage was retrieved but the reader still got it wrong). This decomposition guides where to focus improvement efforts. If Recall@5 is high but end-to-end accuracy is low, the problem is reading comprehension: the relevant passage is being retrieved but the reader is not extracting the answer correctly. If Recall@5 is low, improving retrieval is the most effective intervention. This diagnostic approach, measuring retrieval and reading independently, is a practical engineering discipline that distinguishes teams that iterate efficiently from teams that optimize the wrong component.

Let's build both an extractive QA system and a generative QA pipeline from scratch, using Hugging Face Transformers and sentence-transformer-based retrieval.

The Hugging Face transformers library provides a pipeline API that wraps fine-tuned QA models, handling tokenization, span extraction, and post-processing automatically.

We will first set up the imports and load a pre-trained extractive QA model.

The model assigns the highest start probability to the token that begins "100,000" and the highest end probability to the token ending "pairs". The answer is extracted by joining the tokens between those indices. Notice how concentrated the probability mass is: the model is very confident about both start and end positions, distributing almost all probability weight onto the correct span.

A key insight into how extractive QA works is to see how the start and end probabilities are distributed over the passage tokens.

The sharp probability peaks around the answer tokens show how the model concentrates its confidence. Tokens far from the answer span receive near-zero probability, while the target span stands out clearly. This sparsity is not explicitly enforced by the loss function (which only requires the correct position to have the highest probability, not that others be near zero), but emerges from training because the model learns that confident, localized predictions generalize better.

Now let's build a minimal open-domain QA pipeline: a set of documents, a retriever that finds the most relevant passage, and the extractive reader.

The pipeline first ranks passages by semantic similarity to the question, then applies the extractive reader to the top candidates. The best answer is selected by reader confidence, producing the most likely correct span. Note that the retrieval score and the reader score are separate: a passage can have a high retrieval score (seems relevant to the question topic) but produce a low reader score (the reader doesn't find a confident answer span). The reader score is the final arbiter because it reflects the model's confidence that a specific span is the answer, conditioned on reading the passage alongside the question.

Understanding how retrieval depth affects answer recovery is essential for tuning ODQA systems. Let's visualize key retrieval quality concepts.

The recall-at-k curve illustrates a fundamental tradeoff in ODQA: retrieving more passages improves the chance of including the answer, but the reader must process more text, increasing latency and the chance of selecting a wrong answer from a distractor passage.

Let's compute EM and F1 for a set of predictions to see how these metrics behave on different answer types.

The benchmark progress chart captures a defining moment in NLP history: BERT-Large crossed the human performance threshold in 2018, just two years after SQuAD was released. This rapid progression, from 51 F1 with logistic regression to 93+ F1 with BERT, illustrates how much the field accelerated once large-scale pretraining became available. The transition from BiDAF (77.3 F1) to BERT-Large (93.2 F1) represents a jump of 15+ F1 points achieved not by rethinking the QA architecture but by replacing the encoder with a pretrained transformer. This is the core insight of the pretrain-then-fine-tune paradigm: the heavy lifting of language understanding is done during pretraining, and fine-tuning on a downstream task is relatively cheap.

The choice between extractive and generative QA depends on the application requirements. Let's visualize the tradeoffs across several quality dimensions.

The radar chart highlights the fundamental tension: extractive QA is safer (grounded in source text, no hallucination) but limited to single-span answers from a single passage. Generative QA is more flexible but requires careful prompting and evaluation to prevent confabulation. In practice, many production systems use extractive QA for factoid questions (where the answer is a specific named entity, date, or number) and generative QA for explanatory or analytical questions.

The main parameters to tune in a QA pipeline are:

• model: The pretrained checkpoint for the reader. deepset/roberta-base-squad2 is a strong general-purpose extractive QA model trained on SQuAD 2.0 with unanswerable question support.
• max_length (tokenizer): Maximum token length for the input. Set to 512 for BERT/RoBERTa. Passages longer than this are truncated.
• top_k (retriever): Number of passages to retrieve before reading. Higher values improve answer recall but increase reader latency. Typical values: 3-10.
• normalize_embeddings (SentenceTransformer): Normalizes embeddings to unit length so dot product equals cosine similarity. Should always be True for retrieval.
• score threshold (pipeline): Minimum confidence score for a span extraction to be accepted as a valid answer. Setting this higher reduces false answers on unanswerable questions.

Question answering systems have matured rapidly, but significant challenges remain in deploying them reliably. Understanding these limitations is essential for building systems that work in production and for interpreting research claims accurately.

Retrieval quality bottleneck. In retrieval-augmented QA, the quality of the final answer is bounded by what the retriever returns. If the relevant document is never fetched, even a perfect reader cannot help. Dense retrieval handles vocabulary mismatch better than sparse methods, but requires significant infrastructure: encoding millions of passages, maintaining a vector index, and keeping the index up-to-date as documents change. A retrieval pipeline that worked well on a static Wikipedia snapshot may degrade silently as the underlying corpus evolves. Index staleness is a real operational concern: a news QA system built on a snapshot from six months ago will give wrong answers to questions whose facts have changed. Keeping the index fresh requires either periodic full re-encoding (expensive) or incremental update strategies.

The retrieval bottleneck also manifests as a coverage gap. Dense retrieval models are trained on specific question-answer pairs and may generalize poorly to question types or topics outside their training distribution. A DPR model trained on Wikipedia QA may retrieve poorly on legal documents, medical literature, or code documentation, because the embedding space was shaped by general-domain training examples. Domain-specific fine-tuning of the retriever, or using hybrid retrieval that combines domain-tuned dense models with BM25, can mitigate this.

Hallucination in generative QA. Generative models can produce fluent, confidently stated answers that are factually wrong. This is particularly dangerous in high-stakes domains like medicine, law, and finance. Grounding answers in retrieved documents mitigates but does not eliminate hallucination. Models can misread passages, infer beyond what the text supports, or blend retrieved content with parametric memory. A model asked "What is the dosage of ibuprofen for adults?" may retrieve a correct passage but add an incorrect qualification not in the passage, mixing its parametric knowledge with the retrieved information in ways that produce errors invisible to a simple faithfulness check.

Calibration, making sure confidence scores correlate with accuracy, remains an open problem. Models typically give high confidence to wrong answers at the same rate they give high confidence to correct ones. Research on uncertainty quantification for language models continues, but no mature solution exists for production deployment. Practical mitigations include: always grounding answers in retrieved sources and telling users which source was used, using abstention thresholds calibrated on held-out data, and building human-in-the-loop review for high-stakes domains.

Evaluation gaps. Standard benchmarks like SQuAD measure a narrow slice of QA capability: single-passage, short-answer, well-posed questions with known answers. Real-world questions are messier: ambiguous, multi-hop, temporally sensitive, or subjective. Models that score well on benchmarks can still fail on practical queries. A model with 90 F1 on SQuAD may fail on 30% of real customer support queries because those queries involve domain-specific terminology, require multi-document synthesis, or are poorly formed. Developing evaluation suites that better reflect deployment conditions, including distribution shift from benchmark to production, is an active area of research. Teams building QA systems should maintain their own evaluation sets from real queries, not rely solely on public benchmarks.

Unanswerable questions and refusal. Knowing when not to answer is as important as knowing the answer. A model that always produces some answer is dangerous if the question cannot be answered from available information. Calibrated confidence scores and explicit "I don't know" outputs reduce downstream harm. SQuAD 2.0 and similar datasets help train models to abstain on unanswerable questions, but calibration under distribution shift is hard to guarantee. In practice, the frequency of unanswerable questions in deployment is often much higher than in training data, because deployed systems receive queries from users who don't know whether the system has the relevant information.

Multi-hop and temporal reasoning. Many real questions require combining information across multiple documents or reasoning about time. Current retrievers typically fetch passages relevant to the original question, not the chain of hops needed to answer it. Iterative retrieval and chain-of-thought prompting help but add latency and complexity. Temporal questions are particularly tricky: knowledge changes over time, but models are static. Combining a large static model with a frequently-updated document store is the practical solution, but requires careful engineering to ensure the model reads from the updated corpus rather than relying on stale parametric knowledge.

Domain adaptation. Models fine-tuned on Wikipedia-style passages may underperform on specialized domains: scientific literature, legal documents, medical records. The vocabulary, reasoning patterns, and answer formats differ significantly. Continuing fine-tuning on domain-specific QA data or using domain-specific retrievers can close this gap, but requires labeled data that may be expensive to collect. Active learning strategies, where the system identifies its most uncertain predictions in the target domain and requests human annotation for those examples, can make domain adaptation more data-efficient.

The combination of these limitations means that production QA systems require ongoing evaluation, monitoring, and iteration. A system that works well at launch will drift in quality as the document corpus changes, user query patterns shift, and the world changes in ways that make training-time knowledge stale. Treating QA system development as a product that requires sustained engineering rather than a one-time model deployment is the right mental model for practitioners.

Question answering spans a wide design space, from simple span extraction to complex multi-hop reasoning with retrieval and generation. The key ideas to carry forward:

• Extractive QA frames answer finding as span selection, predicting start and end token indices in a passage. BERT-based models trained on SQuAD achieve near-human performance on the benchmark and remain widely used for grounded, low-latency QA.
• Generative QA generates free-form answers, allowing synthesis across documents and natural phrasing. Instruction-tuned LLMs handle this with prompting, but hallucination requires careful management.
• Open-domain QA adds a retrieval step before reading, using sparse (BM25) or dense (DPR) retrieval to fetch relevant passages from large corpora. Dense retrieval handles vocabulary mismatch; Fusion-in-Decoder enables multi-passage synthesis.
• Evaluation uses Exact Match and F1 for short answers, ROUGE or BERTScore for longer outputs, and LLM-as-judge for complex generative responses. Retrieval quality should be evaluated separately from reading quality.
• Multi-hop QA requires combining information across documents; chain-of-thought prompting and iterative retrieval are the leading approaches.
• Limitations include retrieval bottlenecks, hallucination in generative systems, evaluation gaps between benchmarks and deployment, and domain adaptation challenges. Reliable QA systems require ongoing evaluation and maintenance, rather than one-time deployment.

The next chapter extends these ideas to information extraction: rather than answering natural language questions, we structure the information in text into typed entities, relations, and events, a more explicit form of knowledge extraction that powers knowledge graphs and structured databases.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about question answering systems.