Semantic Roles and Argument Structure › Limitations and Impact

An SRL score must be interpreted relative to its annotation target. PropBank roles require the selected predicate sense; FrameNet roles require the selected frame. Neither a large label inventory nor an intuitive label name guarantees that a system has analyzed the sentence correctly. In "She ate", the thing eaten can remain unexpressed, and in "The window was broken" the breaker may not be mentioned. A system whose output consists only of overt text spans cannot assign a span to an absent expression. FrameNet explicitly records types of null instantiation, so this limitation of an overt-span task should not be mistaken for a complete absence of implicit-participant information in the resource.

In "She ate", an eater is expressed but the thing eaten need not be. An overt-span output cannot supply a text span for that unexpressed participant; this does not determine whether a reader can identify a particular food. The same problem arises with passives that omit the agent, with middles such as "This book reads easily", and with instructions such as "Insert the tab into the slot", where an addressee is understood rather than explicitly named. A task that asks for such a participant needs an output convention beyond selecting overt spans. FrameNet's null-instantiation conventions are one example of recording an unexpressed role (FrameNet II, Section 3.2.3).

Dataset coverage and language also matter. OntoNotes 5.0 includes English, Chinese, and Arabic, with several annotation layers, including proposition structure (LDC release description). A multilingual experiment must specify which language-specific annotations, role mappings, and data splits it uses. Universal Dependencies describes its basic syntactic annotation as typed dependency relations between words (official guidelines). Those syntactic tree annotations alone do not supply a separate semantic-role inventory's gold labels. Accuracy across languages must be measured for the selected language pairs and protocol rather than inferred from linguistic relatedness alone.

A cross-lingual evaluation should inspect how the source and target languages realize arguments. Relevant differences can include case marking, word order, omitted participants, and the lexical frames allowed by a translated predicate. The same numbered label does not by itself establish an identical role definition across independently annotated resources. Document the mappings and inspect examples of each construction before interpreting a transfer score as evidence about general semantic understanding.

Evaluation protocols should state whether predicates are gold or predicted, whether arguments are heads or full spans, how model tokens align to benchmark words, and which role inventory is scored. A change in any of those settings can change what a score measures. For prompted systems, retain the role descriptions, output schema, model version, decoding settings, and invalid-output policy. Sensitivity to these choices is an empirical question to test, rather than a property established by the toy classifier here.

Training/test overlap is another evaluation concern, particularly when a model's pretraining data are incompletely documented. Check known overlaps, disclose what could not be checked, and distinguish exact or near-duplicate test exposure from legitimate learning of annotation conventions on training examples. Sharing a construction does not by itself demonstrate that a particular test sentence was seen during training. The toy classifier starts from a new scikit-learn estimator and fits only on the training labels. For systems that use pretraining, establish exposure separately from the supervised split.

Predicate-argument structures provide a useful intermediate representation for extraction. They retain a predicate and its participants while separating participant roles from surface subject/object positions. This can reduce the number of surface patterns an application needs to inspect, but the mapping from a role analysis to an event schema is a separate step. The next chapter examines relation and event extraction, including the additional context needed to connect sentence-level arguments into document-level records.

A surface pattern that matches only "X bought Y" misses "X purchased Y" unless the application adds another rule or a shared semantic mapping. Frame or sense information can help identify the common commercial scene. That mapping still needs lexical conditions: acquire is broader than buy, so replacing one with the other does not establish that money changed hands. Faithful recovery from the role representation alone requires preserving the distinctions needed by the downstream task.

Role matching can support question answering, but a wh-word does not uniquely determine a numbered argument. For the worked award.01 example, "Who awarded the prize?" asks for A0 and "Who did the committee award the prize to?" asks for A2; "Who was examined?" can ask for a human A1. Likewise, comparing participant roles can help detect a source/summary mismatch, but matching them does not prove that the sentences have the same truth conditions. "The committee awarded the prize" and "The committee did not award the prize" share the participants and predicate while disagreeing about whether the event occurred.

The explicit role structure also gives error analysis a concrete object to inspect. In the displayed predictions, a reader can compare a role label and its exact span with the reference annotation. This tagger receives the predicate as input, so its displayed predictions can be checked for boundary or role-label disagreements. Predicate identification must be checked separately in a system that predicts predicates. Comparing these outputs does not explain all of the classifier's internal reasoning.