Semantic Roles and Argument Structure › Semantic Role Labeling

Start with the selected-predicate task for contiguous argument spans.

Given tokens $x = (x_0, \dots, x_{n-1})$ and a selected predicate at position $p$ with $0 \leq p < n$, predict a set of labeled spans

$$\hat{A} = \{(i_1, j_1, r_1), \dots, (i_k, j_k, r_k)\}$$

where each $(i, j)$ denotes the half-open range from token $i$ through token $j-1$, with $0 \leq i < j \leq n$, and $r$ is a role drawn from an inventory $\mathcal{R}$ determined by the benchmark. The toy’s gold spans are nonoverlapping and exclude the selected predicate token, as required by its BIO encoding. The set lists which text spans fill which roles for the selected predicate; it does not by itself encode every aspect of sentence meaning. The CoNLL-2012 organizer paper's sample marks the selected predicate with V in its argument column, serialized as (V*) (Table 12 and Figure 3). The toy tag sequence below uses bare V. The toy code below takes the selected token index and uses the lowercased form of that token as a feature.

The inventory $\mathcal{R}$ can be coarse or fine, global or frame-specific. A task that permits discontinuous arguments also needs a way to link separated spans as parts of one argument, such as the continuation convention below. Other protocols predict the predicate instead of supplying it. These decisions specify the output target and evaluation protocol. Their effect on empirical prediction difficulty must be measured for the selected setting.

The problem is usually decomposed into four subtasks:

1. Predicate identification. Which tokens are predicates? An evaluation can supply gold predicates instead of scoring their identification.
2. Frame or sense identification. Which frameset or frame does this predicate instance evoke? For verbal predicates, this includes word sense disambiguation; the selected frameset or frame determines which role definitions apply.
3. Argument identification. Which spans realize roles for this selected predicate, including any adjunct-like modifiers included by the annotation scheme?
4. Argument classification. Which role does each span receive?

The four-way decomposition separates different outputs to inspect. If gold predicates are supplied, predicate identification is not a model decision in that evaluation. Predicting them introduces another possible source of error. A wrong sense or frame can also change which role definitions should apply; no universal severity ranking follows from the decomposition. Argument identification errors determine whether a span is considered at all. Argument classification errors determine which label it receives. A system that gets identification right but classification wrong will produce spans with plausible boundaries and incorrect labels, which is a different failure mode from a system that misses spans entirely.

In a pipeline these decisions run in sequence, so an upstream error can affect downstream labels. He et al. (2018) use shared contextual features to score word-indexed predicate candidates and span-based argument candidates and predict their relations. Their end-to-end setting predicts predicates and argument spans without gold predicates. Given the input, the baseline predicts a role or no relation separately for each pair that survives pruning. Across examples, errors on different pairs can still co-vary when the model relies on the same misleading features. Pruning a candidate rules out all of its pairs.

If a pipeline retains intermediate predictions and reference annotations for its stages, we can compare them to locate predicate, sense, boundary, or role-label disagreements. Such a comparison does not by itself establish which earlier decision caused a later error. A joint model can use evidence from several decisions together. Gold-predicate tagging and full predicate-argument prediction both remain useful experimental settings, so the term end-to-end should be accompanied by an explicit description of which labels were supplied and which were predicted.

In SRL, the labels depend on which predicate you are asking about. In "The chef carved the roast", the span "the roast" is A1 of carved. If the sentence also contained a nominalization, the same span could be an argument of two different predicates with two different roles.

Predicate conditioning identifies which predicate-argument relations the output describes. The same token can receive different labels depending on which predicate it is associated with. A sentence with multiple predicates therefore needs an output convention that preserves predicate identity, for example separate tagging passes or labeled predicate–argument edges.

For predicate-conditioned token tagging, the input or output convention must identify the selected predicate. The tagger below encodes each token's position relative to the supplied predicate, together with its surface form, as features. This convention identifies which predicate-argument relations the tags describe; it does not require every SRL architecture to use an externally supplied predicate index.

Relative position identifies whether a token precedes or follows the selected predicate and records its distance, clipped to six tokens in either direction. The predicate form adds lexical context. These inputs can distinguish the same token under different supplied predicates; they do not prove a general distance ordering between arguments and adjuncts or guarantee that the correct role is recoverable. The later position plot describes only the hand-built corpus.

One encoding converts labeled spans into a per-token tag sequence using the BIO convention we saw in BIO Tagging. In the toy's gold encoding, argument spans exclude the selected predicate token. A span of role A0 starting at token $i$ and ending before token $j$ becomes:

• B-A0 at position $i$,
• I-A0 at positions $i+1$ through $j-1$.

In this gold encoding, the predicate token is V and every remaining token outside the arguments is O. The example uses the tutorial's AM-MNR label for the instrument. Released PropBank carve.01 lists the instrument as ARG2.

The plain BIO scheme used here cannot connect separated pieces as one argument. PropBank-style data can include continuation segments, which need an additional convention such as C-A1 to associate them with the argument's first segment. A list of unlinked B-A1 spans does not express that relationship. CoNLL-2005 explicitly describes noncontiguous arguments (Carreras and Màrquez, 2005, Section 2); the miniature corpus below avoids them.

For one predicate, this BIO scheme can encode a flat set of nonoverlapping contiguous argument spans that exclude the selected predicate token. Discontinuous arguments need a way to link their separated pieces, such as the continuation convention above. The toy converter handles only contiguous spans.

The standard metric is span-level precision, recall, and F1, where a predicted span counts as correct only if both boundaries and the label match the gold annotation exactly:

$$P = \frac{|\hat{A} \cap A|}{|\hat{A}|}, \qquad R = \frac{|\hat{A} \cap A|}{|A|}, \qquad F_1 = \frac{2PR}{P + R}.$$

Here $A$ is the set of gold arguments and $\hat{A}$ the predicted argument set for the same selected predicate. The notation $|S|$ counts elements of set $S$, and the intersection keeps arguments with matching start, end, and role. $P$ is precision, $R$ is recall, and $F_1$ is their harmonic mean. For corpus aggregation, the keys also retain sentence and predicate identity. The formulas assume nonzero denominators. The code returns zero for precision when the prediction set is empty, for recall when the gold set is empty, and for $F_1$ when $P+R=0$.

The formula is the familiar one, but the unit of evaluation is the span, not the token. A prediction is correct only if the start boundary, the end boundary, and the label all match. This is a strict criterion, and it is the right one for applications that need exact argument spans.

Three details make published numbers hard to compare:

• Gold versus predicted predicates. An evaluation can supply gold predicates or require their prediction. He et al. (2018) report these as separate settings; a score from one setting cannot by itself establish performance in the other.
• Head-word versus full-span scoring. Span SRL and dependency SRL use different units: complete argument spans versus argument heads. CoNLL-2005 required both the words spanning an argument and its role to be correct (Carreras and Màrquez, 2005, Section 2.1). Do not compare its span scores directly with dependency SRL scores.
• Annotation inventory and tokenization. PropBank and FrameNet define different semantic labels. OntoNotes is a corpus project that includes PropBank-style annotations, rather than a third independent role theory. Compare the actual resource release and evaluation token boundaries. Two encoders can use different subword tokenizers while their predictions are evaluated against the same original word spans.

These settings change what is evaluated. Reading SRL results requires checking the predicate-identification setting, the annotation unit, the label inventory, and alignment to the benchmark tokenization. Subword tokenization inside an encoder does not itself change the benchmark evaluation unit when outputs are correctly realigned.

The distribution of labels matters too. In CoNLL-2005's training data, A0 and A1 are much more frequent than A4 and A5; Table 1 reports 61,440 A0 instances, 84,917 A1 instances, 2,703 A4 instances, and 68 A5 instances. AM-PNC has 2,289 training instances, so calling all less common roles "a few dozen" would be inaccurate. The counts and their split must be stated when interpreting a per-role score (shared-task paper, Table 1).

Pool exact matches across roles to compute micro-averaged scores. Precision divides the match count by the total predicted count when that count is positive. Recall divides it by the total gold count when that count is positive. Micro F1 is twice the match count divided by the sum of those two totals when that sum is positive. An aggregate does not specify the error rate for a rare role. If an application requires that role, report its support and per-role precision and recall as well as the aggregate. The role’s value depends on the application.

The field's trajectory is a useful case study in how NLP moved from feature engineering to pretraining.

These examples illustrate different modeling choices rather than a sequence in which each newer approach replaces all earlier ones. Parse features expose explicit structural information; recurrent and pretrained encoders learn contextual features; text-to-text approaches serialize annotations into an output string. The representation and decoding choices affect which outputs a model can express and how its errors must be scored.

For a concrete example of pretrained encoders in SRL, Shi and Lin (2019) present BERT-based models for span and dependency tasks. Their evaluation settings and comparisons belong to that paper; they do not establish a timeless performance level for every OntoNotes experiment. To attribute a gain to pretraining rather than syntax features or decoding, compare systems that control those other choices or inspect a relevant ablation.

A generated annotation must satisfy the same structural requirements as any other prediction: spans must align to the input, labels must belong to the chosen inventory, and the selected predicate must be identifiable. Set an invalid-output policy before scoring, check the predictions against these requirements, and apply that policy consistently when computing F1. For example, an output phrase absent from the input cannot be accepted as an exact input span, even if it is a plausible paraphrase of an argument.

Prompted and task-trained systems should be compared on a named dataset, output protocol, and metric, rather than assigned a universal accuracy ranking. The text-to-text formulation itself is compatible with supervised training: Paolini et al. (2021) include SRL among the structured tasks in their Translation between Augmented Natural Languages framework. Serialization is a modeling choice, not evidence that exact structural prediction is impossible.

Take the sentence:

The committee awarded the prize to the young researcher last year.

With awarded as the predicate, compare the following participant analyses. The frame-based description needs a qualification about the resource target.

Thematic roles. The committee is Agent, the prize is Theme, to the young researcher is Recipient, and last year is Time.

PropBank. In award.01, the official descriptors are giver for A0, award for A1, and beneficiary for A2. In this sentence, those labels go on the committee, the prize, and to the young researcher, respectively. Last year is AM-TMP, an adjunct.

Frame semantics. Giving provides a useful comparison: its core FEs are Donor, Theme, and Recipient, and Time is peripheral. Those names describe the transfer participants in a paraphrase with gave. FrameNet 1.7 lists give.v in Giving but does not list award.v there or in its LU index. The transfer analogy is therefore not a released FrameNet annotation of awarded: a corpus analysis must consult the actual lexical unit and release rather than assign a frame from intuition alone.

Proto-roles. On an intentional-awarding reading, the committee has agent-like properties such as volition and causation. The prize is the transferred participant, and the researcher is the recipient. Calling the prize Theme does not establish Dowty's incremental-theme property: that property concerns a separate relation between parts of a participant and parts of an event. A proto-role analysis must justify the predicate entailments rather than derive a score from the names Agent, Theme, or Recipient.

The four analyses are not competing answers to the same question. They address different questions. Thematic roles answer "what kind of participant is this?" PropBank answers "which role does the selected roleset define for this argument?" FrameNet answers "which frame element does this fill in this scene?" Proto-roles answer "how agent-like or patient-like is this participant?" The thematic-role, PropBank, and proto-role analyses use the award sentence. The FrameNet comparison uses a paraphrase with gave. Choose the analysis that fits the application.

Now passivize:

The prize was awarded to the young researcher by the committee last year.

The same participant roles are expressed differently in the passive. A1 covers the prize, A2 covers to the young researcher, and A0 covers by the committee, including by in this worked annotation.

A subject-to-A0 shortcut would mislabel the prize. The passive construction identifies it as the passive subject, and the agentive phrase by the committee points to the awarder. The award.01 role definitions determine the numbers. Compare by noon, which gives a time limit. Gildea and Jurafsky use a voice feature and prepositions as PP heads. The toy tagger below uses local token features, including an auxiliary flag for the current word.

Finally, nominalize:

The committee's award of the prize to the young researcher last year.

The predicate is now the noun award. In this event reading the same participants can be identified. A verb-only target excludes this nominal relation; the PropBank frame files and FrameNet include nominal predicates.

The noun award in this event reading relates an awarder, an award, and an awardee. A verb-only target excludes this nominal relation. Whether a system must support it depends on the extraction task; the example does not measure a corpus-wide coverage gain.

The rest of the chapter implements a small PropBank-style tagger. The corpus is deliberately tiny (31 sentences) so every step is visible. The point is the pipeline, not the accuracy.

The toy system trains on 25 sentences, uses hand-designed features, and predicts BIO tags with logistic regression. Larger systems make different choices about representations and decoding, as the cited examples show. This exercise exposes feature extraction, fitting, decoding, and evaluation; scaling its data alone would not turn it into every model in the timeline.

A miniature annotated corpusLink Copied

Each entry stores the token list, the predicate index, and a list of (start, end, role) spans using half-open indexing.