Question Answering › Generative QA: Composing Answers

Extractive QA is constrained by a strong assumption: the answer exists verbatim in the provided text. Many real questions violate this. "What year did Napoleon invade Russia?" has a clean extractable answer, but "Why did Napoleon's Russian campaign fail?" requires synthesizing information from multiple sentences or documents. And many questions about the world don't have a single canonical passage to extract from.

Generative QA lifts the span constraint: instead of identifying a substring, the model generates a token sequence. This allows answers that aggregate information across sentences or documents, answers that require inference or reasoning, and answers expressed in the model's own words, potentially more clearly than the source text.

The tradeoff is that generative models can hallucinate, producing plausible-sounding but incorrect answers. Managing hallucination is the central challenge of generative QA. A model that generates "The French Revolution began in 1799" instead of 1789 produces a fluent, confident, wrong answer that no extractive system would produce, because the correct date appears in most relevant passages as a direct quote.

Understanding when to use generative versus extractive QA is a design decision every practitioner faces. The key question is: does the correct answer appear verbatim in the source text? If yes, extractive QA is safer. If the answer requires synthesis, inference, or natural language composition, generative QA is necessary.

Generative QA divides into two settings based on whether the model has access to external context at inference time.

Closed-book QA queries the model directly with no supporting documents:

Question: What is the capital of France? Answer:

The model must answer from its parametric knowledge, the information encoded in its weights during pretraining. Surprisingly, large language models trained on web-scale text can answer many factual questions accurately this way. Roberts et al. (2020) showed that T5 with 11 billion parameters achieved competitive performance on open-domain QA benchmarks without any retrieval, leading to the "language model as a knowledge base" framing.

The intuition behind closed-book QA is that a language model trained on a trillion tokens of web text has absorbed an enormous amount of factual knowledge. The act of predicting missing words in text requires encoding information about entities, relationships, dates, and events. A model that can accurately predict the next word in "The capital of France is ___" must have encoded the fact that Paris is the capital of France. This implicit knowledge storage is a byproduct of language modeling, not something explicitly designed in.

Closed-book QA is simple (no retrieval infrastructure needed) but brittle. Knowledge encoded in weights reflects training data cutoffs, may be inconsistently distributed across facts, and is hard to update. If a fact changes (a company is acquired, a political leader changes, a scientific consensus shifts), retraining the entire model is the only way to update the knowledge. The model cannot cite sources, making it impossible to verify claims or trace where an answer came from.

Open-book QA provides supporting documents at inference time. The model's task is to read the context and synthesize an answer:

Context: [retrieved passage(s)] Question: [question] Answer:

This is also called retrieval-augmented generation (RAG) when the retrieval step is part of the pipeline. Open-book QA is more reliable for factual questions because the answer is grounded in retrieved evidence. The model acts as a reader and synthesizer, not as a memorized knowledge store. When the answer changes, you update the document corpus, not the model weights.

The distinction between closed-book and open-book QA matters practically. For a customer support chatbot that needs to answer questions about product documentation that changes frequently, closed-book QA is inappropriate: the model's weights would quickly become stale as documentation is updated. Open-book QA over a regularly refreshed document store is the right architecture. For a general trivia assistant where the facts are relatively stable (historical events, scientific constants, geographical facts), closed-book QA from a large LLM can work well with simpler infrastructure.

Before large decoder-only models dominated the field, encoder-decoder architectures like T5 and BART were the standard approach for generative QA. These models treat QA as a sequence-to-sequence task: the encoder reads the question (and optionally a context passage), the decoder generates the answer token by token.

As we discussed in the chapters on encoder-decoder architectures and T5, these models use cross-attention to let the decoder attend to the encoder's output at every generation step. For QA, this means every generated answer token is informed by the full question and context, rather than only the tokens generated so far.

Fine-tuning T5 on a QA dataset is straightforward: format inputs as "question: [q] context: [c]" and targets as the answer text. T5's text-to-text framing makes this natural, since every NLP task becomes a text generation problem. The same model trained on question generation (generating questions from passages) or summarization can be repurposed for QA with different training data and input formatting.

The decoder generates answers autoregressively. At each step $t$, it computes:

$$P(a_t \mid a_{<t}, q, c) = \text{softmax}(\mathbf{W}_o \mathbf{h}_t^{\text{dec}})$$

where:

• $a_t$: the token generated at step $t$
• $a_{<t}$: all previously generated tokens (the partial answer so far)
• $q, c$: the question and context passed to the encoder
• $\mathbf{h}_t^{\text{dec}} \in \mathbb{R}^d$: the decoder hidden state at step $t$, computed by attending to $a_{<t}$ (via masked self-attention) and to the encoder outputs for $(q, c)$ (via cross-attention)
• $\mathbf{W}_o \in \mathbb{R}^{|V| \times d}$: the output projection matrix that maps the hidden state to logits over the vocabulary of size $|V|$

The softmax converts the logits to a probability distribution, from which the next token is sampled or the most likely token selected (greedy decoding). This continues until the model generates an end-of-sequence token.

The full answer is generated by sampling or beam search until the model produces an end-of-sequence token. Beam search maintains the $B$ most likely partial sequences at each step, where $B$ is the beam size, exploring more candidates than greedy decoding while still being computationally tractable. For QA, where answers are typically short (one to three sentences), beam search with $B = 4$ or $B = 5$ is standard.

The power of this approach is that the answer is not constrained to any span in the input. The model can produce "Paris" even if the passage mentions "the French capital" or "the City of Light." It can infer "steel" from a passage that describes the material's iron and carbon composition without ever using the word "steel." This abstraction and paraphrase capability is what generative QA enables that extractive QA cannot.

Modern instruction-tuned LLMs (GPT-4, Claude, LLaMA with instruction fine-tuning) have largely replaced specialized QA architectures for many applications. These models are trained to follow natural language instructions, so QA becomes a special case of instruction following:

"Answer the following question based on the provided context: [context]. Question: [question]"

The model responds with a generated answer. Prompt engineering (structuring the context, question, and any formatting instructions carefully) becomes the primary lever for improving quality.

Instruction-tuned models excel at multi-step reasoning, clarifying ambiguous questions, acknowledging uncertainty, and explaining their reasoning. A good instruction-tuned model will say "I cannot determine this from the provided context" rather than fabricating an answer, provided the prompt includes an instruction to that effect. This refusal behavior is a form of calibration: the model knows when its confidence is low. They perform well across question types without task-specific fine-tuning, making them attractive for rapid prototyping and for question types that do not fit neatly into extractive or standard generative formats.

The main limitations are cost (large models are expensive to call), latency (generation is slower than span extraction), and the hallucination risk that comes with any generative model. For applications that need to process millions of questions per day at low latency, a fine-tuned smaller model is often preferable to a large general-purpose LLM.

Prompting strategy also has a large effect on quality. Few-shot examples in the prompt (showing the model two or three question-context-answer triplets before the actual question) often improve performance, particularly for formats the model is not familiar with. Chain-of-thought prompting ("Think step by step before answering") improves accuracy on questions that require multi-step reasoning. System prompts that specify the model's role ("You are a helpful assistant that answers questions accurately and cites the passage when possible") encourage faithful answers and appropriate hedging.