Question Answering › Long-Context and Multi-Hop QA

Standard QA benchmarks assume the answer is contained in a single passage. Many real questions require more: following a reasoning chain across documents, combining facts from different sources, or tracking entities and events over a long text. These more demanding question types reveal the limits of single-passage architectures and motivate more sophisticated reasoning approaches.

Multi-hop questions require connecting information from two or more passages. The question cannot be answered by any single passage in isolation, as it requires chaining facts together. For example:

"What country was the director of Inception born in?" requires: (1) finding that the director is Christopher Nolan, (2) finding that Nolan was born in London, England.

Neither fact alone answers the question. A retrieval system that fetches passages mentioning "Inception" may retrieve the director's name, but not their birth country. And a retrieval system prompted with the original question may fetch passages about Inception's plot rather than biographical details of its director. The core difficulty is that the retrieval query at step one cannot be formulated correctly until you have the answer from step one.

This is called the bridge entity problem: the first hop retrieves information about a "bridge entity" (Christopher Nolan) that is needed to formulate the query for the second hop. Single-step retrieval systems cannot handle this because the query must encode knowledge that can only be obtained by reading the first passage. A system that retrieves everything in one shot either needs to be lucky (a single document mentions both facts) or needs to retrieve so broadly that it can find both relevant passages by chance.

Answering multi-hop questions with single-passage retrieval fails unless a single document happens to contain both hops. This failure mode is common in practice: encyclopedic articles about a film may mention the director but link to a separate biography page for their background. Dedicated multi-hop datasets like HotpotQA (Yang et al., 2018), MuSiQue, and 2WikiMultiHopQA benchmark this capability and measure whether systems can follow reasoning chains across documents rather than merely retrieving relevant passages.

Approaches to multi-hop QA include:

• Iterative retrieval: Retrieve a first passage, use it to generate a "bridge" query, retrieve a second passage with the bridge query, then answer. This approach requires a model that can identify what information is still missing and reformulate the query accordingly.
• Chain-of-thought reasoning: Prompt LLMs to think step-by-step, generating intermediate reasoning steps that decompose the multi-hop question. The model produces explicit intermediate conclusions before giving the final answer.
• Subquestion decomposition: Decompose the complex question into simpler sub-questions, answer each independently, then combine the sub-answers into a final response.

Chain-of-thought prompting has proven particularly effective for multi-hop QA with capable LLMs. The model writes out reasoning steps before producing the final answer, making errors visible and enabling verification. When a model states "The director of Inception is Christopher Nolan. Christopher Nolan was born in London, England. Therefore, the country is England," each intermediate step can be checked for correctness, unlike a model that produces only the final answer without showing its work.

The value of explicit reasoning chains extends beyond verification. When a model's chain-of-thought contains an error, it often reveals where the model went wrong, enabling targeted correction. A model that confidently states "The director is Nolan Spielberg" in its chain-of-thought is clearly confused, and a downstream verification step can catch this. A model that produces only the final (wrong) answer provides no such signal.

Modern LLMs with large context windows (GPT-4 supports up to 128k tokens, Claude up to 200k) can read much longer documents than earlier systems. This opens the possibility of passing entire documents, or multiple retrieved documents, directly into the context window rather than chunking and retrieving.

The appeal is obvious: if you can fit a 100-page document into the prompt, you never have to worry about whether the retriever found the right chunk. The model can attend to every part of the document directly. This is especially valuable for question types where the relevant information is spread thin across a long document, or where the question cannot be answered without reading the surrounding context.

Long-context QA has its own challenges. The "lost in the middle" phenomenon (Liu et al., 2023) shows that LLMs tend to attend strongly to information at the very beginning and end of long contexts, while underweighting information buried in the middle. In experiments, performance dropped sharply when the relevant passage was positioned in the middle of a long context, even though the information was present. This effect grows stronger as context length increases, suggesting that raw context capacity does not translate directly into uniform reading capability across the entire context window.

The practical implication is that document ordering matters even in long-context systems. Placing the most relevant documents at the beginning or end of a long prompt tends to produce better answers than placing them in the middle. This "primacy and recency" effect mirrors what we know about human working memory and suggests that transformer attention, despite its theoretical ability to attend uniformly to any position, learns positional biases from training data that concentrate on shorter contexts.

Additionally, long-context inference is expensive in both memory and compute. Attention complexity scales quadratically with sequence length in standard transformers, though architectural innovations like sliding window attention (Longformer, BigBird) and efficient attention approximations mitigate this. Retrieval over a long context can still outperform naive full-context reading for questions that depend on information buried deep in a document, by retrieving the relevant section first rather than asking the model to locate it within a sea of text.

The right approach depends on the application. For a few-hundred-page technical manual where any section might be relevant, chunking and dense retrieval followed by focused reading is often more effective than passing the entire document to a long-context model. For a 20-page research paper where the question requires understanding across the full document, long-context reading may be preferable.

Two additional question types challenge standard QA architectures. Temporal questions ask about facts that change over time: "Who is the CEO of Apple?" has different correct answers depending on when the question is asked. Closed-book QA with a model trained before a leadership change will give the old answer. Open-book QA with an up-to-date document corpus will give the current answer, but only if the retriever fetches a passage with a recent publication date. Handling temporal questions correctly requires retrieving relevant information and prioritizing the most recent relevant information.

Ambiguous questions have multiple valid interpretations, each with a different correct answer. "When did they publish the paper?" requires knowing who "they" refers to and which paper. "What is the best treatment for anxiety?" is ambiguous across types and severities of anxiety. Good QA systems should either resolve ambiguity through clarification (if interacting with a user) or acknowledge ambiguity in the answer (if answering automatically). Models that suppress ambiguity by confidently answering one interpretation are unreliable for these question types.