Question Answering › From Pattern Matching to Reading Comprehension

The earliest QA systems, like BASEBALL (1961) and LUNAR (1972), were domain-specific and rule-based. BASEBALL could answer questions about American baseball game results by querying a structured database. LUNAR answered questions about moon rocks by translating natural language into database queries. These systems worked well within their narrow domains but required hand-engineered rules and could not generalize beyond the curated knowledge they were built around.

The reason these systems were brittle is fundamental: language is infinitely variable, but a hand-written rule set is finite. The same fact can be expressed hundreds of different ways, and a rule that matches "How many home runs did Babe Ruth hit?" fails on "What was Ruth's career home run total?" even though both questions ask for the same thing. Engineers compensated by writing more and more rules, but coverage never caught up with the diversity of real questions.

The question of how to move beyond curated databases led researchers toward text corpora. If answers exist somewhere in documents, the task becomes: find the right document, then find the answer within it. This two-stage structure, retrieve then read, became the dominant paradigm for open-domain QA and persists in modern retrieval-augmented systems. The insight was deceptively simple: rather than building a knowledge base by hand, use the vast body of text already written by humans as the knowledge source.

The "read" step, however, required models that could comprehend text, not just match keywords. This challenge became formalized as machine reading comprehension (MRC). The release of large reading comprehension datasets crystallized progress: SQuAD (Stanford Question Answering Dataset), released in 2016, provided over 100,000 question-passage pairs where the answer was a span of text from the passage. Models had to select exactly the right span, making evaluation straightforward and enabling rapid, reproducible progress.

SQuAD and its successors (SQuAD 2.0, TriviaQA, Natural Questions, QuALITY) drove a wave of architectures designed to solve span extraction. Bidirectional Attention Flow (BiDAF), QANet, and eventually BERT-based models progressively improved performance. When BERT was fine-tuned on SQuAD in 2018, it matched human-level performance on the benchmark, a milestone that marked the maturation of extractive QA as a solved problem on that particular benchmark, though far harder problems lay ahead.

It is worth pausing to appreciate what "human-level performance on SQuAD" means. The benchmark measures agreement between model output and human-annotated answer spans. A model that scores above the human agreement threshold has, in a narrow technical sense, matched human accuracy at selecting answer spans in Wikipedia passages. This is a concrete benchmark achievement. But it does not mean the model has solved reading comprehension in general: SQuAD passages are from Wikipedia, questions were posed by annotators who had already read the passage, and every question has a guaranteed answer in the text. Harder datasets like adversarial SQuAD, TriviaQA, and Natural Questions revealed that models had partially learned to exploit surface-level patterns rather than deeply understanding the passage.