Penn Treebank (1993) › The Data Bottleneck

By the late 1980s, computational linguists had developed sophisticated theories of syntax and powerful parsing algorithms. Chart parsers, based on context-free grammars or more expressive formalisms, could efficiently search through vast spaces of possible syntactic structures. Unification-based grammars like Lexical Functional Grammar (LFG) and Head-Driven Phrase Structure Grammar (HPSG) represented rich linguistic insights about agreement, subcategorization, and feature propagation. Principle-based approaches, inspired by Chomsky's Government and Binding theory, aimed to derive surface syntactic structures from universal principles and language-specific parameters. These frameworks were theoretically elegant, linguistically motivated, and demonstrably capable of analyzing complex sentences when given the right grammar and lexicon.

Yet they struggled with coverage and robustness. Hand-crafted grammars required enormous effort to develop. A linguist might spend months encoding the rules for handling relative clauses, only to discover that real newspaper text contained relative clause constructions the grammar hadn't anticipated.reduced relatives, stacked relatives, relatives with unusual gap patterns. Each new text genre brought new challenges: coordination structures that violated the grammar's assumptions, novel uses of punctuation, sentence fragments, parentheticals, and dislocations. The grammar grew and grew, accumulating special cases and exceptions, becoming increasingly difficult to maintain. Worse, the grammars were brittle. A single unknown word or unanticipated construction could cause parsing to fail entirely, producing no output when a partial analysis might have been useful.

Statistical approaches promised a solution. Instead of writing rules by hand, one could learn them from data. Hidden Markov Models had revolutionized speech recognition by learning acoustic-phonetic patterns from labeled speech corpora. Why not apply similar techniques to syntax? The idea was appealing: gather a large corpus of sentences, annotate them with their correct syntactic structures, and train statistical models to predict these structures for new sentences. The models would naturally handle ambiguity through probabilities, assigning higher likelihood to more frequent structures. They would be robust to novel constructions, backing off to simpler patterns when specific configurations hadn't been seen during training. They would improve automatically as more training data became available, without requiring linguists to painstakingly craft new rules.

But this vision faced a chicken-and-egg problem. Training statistical parsers required treebanks.large corpora of sentences annotated with parse trees. Creating such treebanks required parsing sentences to produce the annotations. If you already had a reliable parser, you could use it to annotate data automatically, but if you had a reliable parser, you wouldn't need to train a statistical one. Manual annotation was possible in principle but seemed prohibitively expensive. Linguists would need to analyze thousands of sentences, drawing syntactic trees for each, maintaining consistency across annotations, resolving ambiguous cases through explicit guidelines. The effort seemed Herculean.

Earlier treebanks existed but were limited in scale or scope. The Lancaster-Oslo/Bergen (LOB) corpus, created in the 1970s, contained a million words of British English tagged with part-of-speech labels but lacked syntactic structure. The London-Lund Corpus provided parsed British English, but its 50,000 words were too few for training robust statistical models. The Brown Corpus, containing a million words of American English from the 1960s, was tagged with parts of speech but not parsed. Smaller syntactically annotated corpora existed for specific purposes, but none was both large and consistently annotated with enough syntactic detail to train and evaluate statistical parsers. The field needed a treebank that was large enough to matter, detailed enough to be useful, and consistent enough to be reliable.