PropBank (2005) › Limitations

PropBank's design also imposed practical limits. Some followed from its choices about coverage and role labels; others reflected the difficulty of annotating semantic knowledge at scale.

PropBank's most obvious limitation was its coverage. Even after substantial annotation efforts, PropBank covered only a fraction of the verbs and predicates that appear in real text. Many common verbs lacked frame files, limiting PropBank's usefulness for unrestricted text processing. The annotation process, while more efficient than FrameNet's, still required creating frame files for each verb and annotating sentences, a labor-intensive process that didn't scale easily to cover the full vocabulary.

The coverage problem was exacerbated by the fact that PropBank focused primarily on verbal predicates. While verbs are central to event structure, many sentences involve nominalizations, adjectives, and other predicates that also have semantic roles. A sentence like "The acquisition of TechStart by Acme Corp" involves an event with roles, but "acquisition" is a noun, not a verb. PropBank's focus on verbs limited its coverage of these alternative predicate types, constraining its usefulness for some applications.

PropBank's use of verb-specific frame files created a tension between verb-specific distinctions and general role patterns. Some researchers argued that PropBank's verb-specific approach missed generalizations that could be captured with more general role labels. For example, many verbs have similar argument structures that could be grouped together, but PropBank's frame files treated each verb separately, potentially missing these patterns.

The verb-specific approach also created challenges for handling novel or rare verbs. If a system encountered a verb that didn't have a frame file in PropBank, it couldn't use PropBank's role definitions. While systems could fall back to general interpretations of Arg0, Arg1, etc., these interpretations were less precise than verb-specific definitions. This limitation affected PropBank's usefulness for domains with specialized vocabulary or for processing text with many rare verbs.

PropBank's annotations, while valuable, reflected the inherent subjectivity in semantic annotation. Different annotators sometimes disagreed about which phrases filled which roles, especially for arguments that were less clearly defined or for verbs with multiple possible argument structures. This annotation inconsistency affected the quality of training data derived from PropBank. Machine learning systems trained on inconsistent annotations learned inconsistent patterns, reducing their accuracy.

The annotation process also faced challenges with ambiguity and underspecification in natural language. Consider "John broke the window." This sentence clearly involves a breaking event, but many semantic roles are unspecified: What instrument was used? When did it happen? Why did John break it? PropBank's annotations marked only the roles that were explicitly realized in the sentence, but this meant that many annotations were incomplete. Systems couldn't always distinguish between roles that were unrealized but implied versus roles that simply weren't relevant to the event.

PropBank's use of numbered arguments rather than named roles was a design choice that enabled scalability but created limitations. Numbered arguments like Arg0 and Arg1 are less interpretable than named roles like Agent and Patient. Researchers and system developers had to consult frame files to understand what each argument represented, adding complexity to using PropBank annotations.

The numbered argument scheme also created challenges for applications that needed human-readable semantic representations. While Arg0 might consistently represent the agent across many verbs, the label "Arg0" doesn't convey this meaning directly. Named roles like Agent are more interpretable, but PropBank's design prioritized scalability over interpretability. This trade-off limited PropBank's usefulness for some applications that needed more transparent semantic representations.

PropBank's annotations were based primarily on general-purpose corpora like the Wall Street Journal texts used in the Penn Treebank. Many applications needed semantic role knowledge in specialized domains like medicine, law, or finance. PropBank's general-purpose annotations didn't always capture domain-specific event structures. A medical frame for diagnosis might require different elements than a general-purpose frame for observation, but creating domain-specific PropBank annotations required new annotation efforts.

As applications moved into specialized domains, they often needed additional annotations or adaptations of PropBank's general-purpose frame files. The original resource did not describe every domain-specific predicate or event structure.

Despite these limitations, PropBank remained widely used because it offered broad semantic annotations aligned with an existing syntactic resource. Its incomplete coverage and occasional annotation disagreements also documented the practical difficulty of building semantic resources at scale.