Penn Treebank (1993) › The Penn Treebank Project

In 1989, Mitchell Marcus at the University of Pennsylvania secured funding from DARPA and NSF to build exactly this resource. The Penn Treebank project aimed to create a multi-million-word corpus annotated with part-of-speech tags and skeletal syntactic structures. The choice of "skeletal" was deliberate.rather than attempting to represent every theoretical distinction linguists might care about, the treebank would focus on surface syntactic structure: noun phrases, verb phrases, clauses, their hierarchical organization, and their grammatical functions. This pragmatic decision prioritized consistency and scalability over theoretical completeness. Deep syntactic relationships, semantic roles, and discourse structure would be left for later work or other resources.

The annotation process required careful planning. Marcus, Marcinkiewicz, and Santorini developed detailed annotation guidelines specifying how to analyze every construction the annotators might encounter. These guidelines addressed countless edge cases: How should you bracket "old men and women".is "old" modifying just "men" or both conjuncts? Where should adverbs attach.to the verb phrase they modify or higher in the tree? How should you represent ellipsis, where material is missing but understood? The guidelines ran to hundreds of pages, constantly refined as annotators encountered new problems and reported inconsistencies. The goal was not to resolve every theoretical debate in linguistics but to ensure that different annotators, encountering the same sentence, would produce the same or nearly the same tree structure.

The annotation scheme balanced linguistic sophistication with practical tractability. The treebank distinguished roughly 45 part-of-speech tags, more granular than the traditional schoolroom categories but less fine-grained than some linguistic theories demanded. Common nouns, proper nouns, and pronouns received distinct tags; verbs were distinguished by tense and form; determiners, prepositions, and conjunctions each had dedicated tags. At the phrasal level, the treebank used approximately 15 phrase labels: NP for noun phrases, VP for verb phrases, S for clauses, SBAR for subordinate clauses, PP for prepositional phrases, ADJP for adjective phrases, ADVP for adverbial phrases, and several others. Each phrase could be annotated with functional tags indicating grammatical role: -SBJ for subjects, -TMP for temporal modifiers, -LOC for locatives, -PRD for predicates, among others.

This representation captured significant syntactic information while remaining relatively simple to annotate and process computationally. A sentence like "The company reported earnings that exceeded analysts' expectations" would be analyzed with a tree structure showing that "The company" forms a noun phrase serving as the subject, "reported" is the main verb, "earnings" is the direct object, and "that exceeded analysts' expectations" is a relative clause modifying "earnings." The relative clause itself contains its own subject ("that," standing in for "earnings"), verb ("exceeded"), and object ("analysts' expectations"). This hierarchical structure made explicit the grammatical relationships implicit in the linear word sequence.

Annotators worked primarily with Wall Street Journal articles from 1989, selected for their syntactic variety and practical relevance.business and financial news covered a wide range of topics and employed sophisticated language. The text was first automatically tagged with parts of speech using existing taggers, then human annotators corrected the tags and added skeletal parse trees using specialized annotation tools. The Fidditch partial parser, developed by Donald Hindle at AT&T Bell Labs, provided initial tree structures that annotators could correct and complete, dramatically speeding up the annotation process compared to drawing trees from scratch. Even with these tools, annotation was labor-intensive. A skilled annotator might parse 300-400 words per hour, meaning the full million-word corpus required thousands of hours of annotation effort.

The project treated quality control as a core requirement. Inter-annotator agreement studies measured consistency: two annotators would independently analyze the same sentences, and researchers would calculate how often they agreed on tree structures. The agreement wasn't perfect.syntactic ambiguity and judgment calls ensured some variation.but it was high enough to inspire confidence. When disagreements occurred, the annotation guidelines were clarified and refined. The project maintained a distinction between the training corpus, used for developing statistical models, and separate development and test sets for evaluating those models. This methodological rigor, standard practice today, was relatively novel in early-1990s computational linguistics and would become a defining feature of empirical NLP.