Relation, Event, and Temporal Extraction › Relations

Our binary relation record contains two entities and a link. Choosing that link involves a schema, an extraction method, and an evaluation protocol.

The apparent simplicity is deceptive. "Acme acquired Bolt" looks like a binary predicate, but the same fact can be expressed as "Bolt was acquired by Acme," "Acme's acquisition of Bolt," or "Acme bought Bolt." The phrase "the Acme-Bolt deal" needs context to identify the relation and its direction. A relation extraction system must decide which of these expressions count as evidence for the same relation, which words are the arguments, and whether the relation is directed. Those decisions depend on the schema, and the schema depends on what the downstream application needs to do.

Relation types have an arity, the number of argument positions they take. The single-label classifier for binary relations below takes a document $D$ and an ordered pair of entity mentions $(e_1,e_2)$. Its class set $\mathcal{R}$ contains the binary relation labels and no_relation. For $r\in\mathcal{R}$, it predicts

$$P(r \mid D, e_1,e_2) = \operatorname{softmax}\!\left(W\,\mathbf{h}(D, e_1,e_2) + \mathbf{b}\right)_r$$

where $\mathbf{h}\in\mathbb{R}^d$ represents the document conditioned on the entity positions, $W$ maps its $d$ coordinates to one logit per class, and $\mathbf{b}$ is the class-bias vector. Softmax normalizes these logits over mutually exclusive labels, including no_relation. This is a single-label formulation; it is not sufficient when a pair can have several relation labels. At inference, a decision rule can select the highest-probability class and optionally abstain below a chosen confidence threshold.

The representation $\mathbf{h}$ can be as simple as a bag-of-words feature vector over the tokens between the two mentions, or as complex as a transformer encoding of the entire document with entity markers. The output formula stays the same across these different representations. The important point is that the decision is conditioned on the selected mention pair. The system is not classifying the document in isolation; it is asking whether this particular pair of mentions stands in this particular relation.

Specify three properties of the schema:

• Direction. (Jane Ruiz, employer_of, Acme) and (Acme, employer_of, Jane Ruiz) are different statements. An unordered pair representation alone cannot distinguish these two argument orders.
• Symmetry. Some relations are symmetric (spouse_of), some antisymmetric (parent_of), some neither. A declared symmetry constraint can be imposed at inference. If a relation is known to be symmetric and the model predicts it for one ordering, the inverse ordering should receive the same prediction.
• Multiplicity. Can one entity pair hold multiple relations at once? A schema could allow both (Jane Ruiz, employee_of, Acme) and (Jane Ruiz, chief_executive_of, Acme). These use the same ordered pair. Co-occurring labels require a multi-label output, such as independent binary decisions, rather than one mutually exclusive softmax decision.

These properties interact. In exhaustive classification under a directed, multi-label schema, there is a binary decision for each relation type and each ordered pair, so the number of decisions grows with both counts. For a symmetric relation, the two orderings of each pair of distinct candidates share one prediction. This halves the number of independent pair decisions for that relation. If the schema has many mutually exclusive relation types, a softmax is appropriate. If the types can co-occur, independent sigmoids are more appropriate. A single categorical decision cannot represent two co-occurring labels; a multi-label output can.

Aggregation combines evidence of different relevance. Suppose a document contains three mentions of Acme and three mentions of Bolt. One sentence says that Acme acquired Bolt. Another lists both companies as competitors. A third says that a former Acme executive joined Bolt. All three sentences mention both organizations, but only the first supports acquisition. The third relates the executive to Bolt. A system can pool their representations or scores, but the aggregation rule needs to distinguish acquisition evidence from co-occurrence and other relations.

Closed-schema extraction and Open IE make different choices about the allowed predicates.

Closed-schema extraction fixes the relation inventory in advance. Examples include the ACE 2005 relation inventory (18 subtypes under six top-level types, including Physical, Part-whole, Personal-Social, and ORG-Affiliation), the TAC KBP slot set, a fixed subset of Wikidata properties, and domain ontologies like the UMLS semantic network for biomedical text. A fixed inventory supplies shared output labels for evaluation and aggregation. It cannot emit an unlisted relation type without changing that inventory, although the system can retain the source text and unmapped predicates. If the inventory has no suitable type for a minority investment, it cannot represent that investment as a canonical relation without changing the schema.

Open information extraction (Open IE) derives predicates from text without a fixed list of relation types. A triple can retain a surface phrase, as in ("Acme Corp", "had acquired", "Bolt Systems"), or use a normalized phrase. Examples include TextRunner (Banko et al., 2007), ReVerb (Fader, Soderland, and Etzioni, 2011), and Stanford OpenIE. Their patterns and output conventions differ. Its models and rules still limit coverage. Surface predicates such as had acquired, bought, purchased, and took over need normalization if the application wants to query one canonical relation.

In the surface-triple form above, arguments are text strings rather than supplied knowledge-base identifiers. A subject such as "the company" still needs a reference decision to identify its organization. Surface triples can be queried as strings. Queries that should join aliases or combine predicate paraphrases need additional identity and predicate mappings; otherwise they can miss records stored under another surface form.

One possible design extracts surface predicates, maps them to canonical relations, and keeps the surface form as provenance. A classifier can learn that mapping from labeled examples. Clustering can group candidate paraphrases, for example by representing predicates with argument-type count vectors and grouping similar vectors. The resulting clusters still need alignment to the application's relation labels. Evaluate extraction and mapping errors separately; combining the stages does not guarantee a recall gain. The stored surface predicate lets an auditor see which phrase was mapped to the canonical relation.

Patterns encode selected constructions directly. Xu et al. (2009) report high precision and reasonable recall for their learned dependency patterns; Hennig et al. (2015) document learned patterns damaged by parsing and supervision errors. Compare precision and recall on the target data when choosing between patterns and learned extractors.

The classic example is Hearst patterns (Hearst, 1992) for hyponymy. The lexico-syntactic frames

are cues that $Y$ is a kind of $X$. The same technique generalizes: "PERSON, (?:the )?(?:CEO|chief executive) of ORG" indicates a ceo_of relation; "ORG (?:acquired|bought|purchased) ORG" indicates an acquisition.

Hearst patterns illustrate how to encode a hypothesis about linguistic evidence. "X such as Y" can indicate a hyponym relation, but a match is not proof: parsing, scope, quotation, and nonliteral uses can change the interpretation. Evaluate precision on the target domain before using a pattern without human review.

The following examples compare dependency conditions with surface patterns.

Patterns can operate over dependency paths, which can handle intervening material and some word-order variation. The active-voice pattern nsubj(acquire, X) ∧ dobj(acquire, Y) covers "Acme acquired Bolt" and can tolerate an intervening phrase. "Bolt was acquired by Acme" needs a separate passive pattern, with the passive subject as the acquired entity and the by-phrase as buyer. Curated patterns and validators can provide a precision-oriented component, but parser errors and unhandled context prevent a general precision guarantee.

A dependency rule exposes syntactic conditions that can be inspected alongside the parse. A surface regex that fires on "Acme acquired Bolt" may also fire on "Acme denied that Acme acquired Bolt," which does not affirm the acquisition. One conservative rule for affirmative acquisitions in the main clause can require the acquisition verb to be the sentence’s dependency root and check it for negation and modal auxiliaries. In the denial example, "acquired" heads the complement of "denied", so it fails the root check. The dependency representation supplies clause-head and modifier relations directly; a regex over raw text does not receive those parse labels. The tradeoff is that dependency patterns depend on parser accuracy, so a parsing error can cause a pattern to miss a valid relation or fire on an invalid one.

Check coverage and portability on the target data. A new domain, language, or writing style can require pattern changes; evaluate existing patterns on the target data before adding rules. And patterns interact: a pattern that fires on "Acme's chief executive, Jane Ruiz" can also fire on "Acme's former chief executive, Jane Ruiz," which asserts something quite different. Contextual modifiers (former, alleged, would) must be handled explicitly, which statistical models try to learn from data.

Pattern-based extraction can serve as a precision-oriented component within a broader system. It can seed a training set, cross-check a learned model's high-confidence predictions, or handle a small set of business-critical relations where the schema is stable and the cost of a false positive is high. Learned components are one way to extend coverage beyond a hand-written inventory; adding patterns is another. Measure both coverage and false positives on the target data.

We'll build a small pattern and dependency-rule extractor in the implementation section.

Mintz et al. (2009) motivate distant supervision with the cost and limited quantity of manually labeled relation data. Their method treats co-mentions of a known relation’s entity pair as possible evidence. Riedel et al. (2010) give a stronger formulation, paraphrased here.

If a knowledge base already asserts that entity $e_1$ stands in relation $r$ to $e_2$, then every sentence mentioning both $e_1$ and $e_2$ expresses $r$.

Riedel et al. found co-mentions that did not express the relation in their manually checked samples. Mintz et al. still obtained large training sets with Freebase supervision. In the Mintz et al. procedure, co-mentioning sentences contribute features to a vector for a known relation–entity-pair tuple; the KB supplies the tuple's positive label. The authors also sample negative training instances. State the negative-label rule rather than assuming that every unlisted fact is false.

The appeal is easy to see. Mintz et al. report 116 million relation instances in the Freebase snapshot they describe. The filtered Wikidata5M dataset contains millions of relation triples. If millions of those instances each yield a usable sentence bag, you have millions of weakly labeled training bags. The labels are inherited from the knowledge base rather than individually adjudicated for each sentence. Missing or incorrect knowledge-base facts can therefore create misleading training labels. This is a risk of the labeling procedure, not a guarantee that every trained model reproduces every labeling error.

The problem is that the assumption is too strong. A sentence mentioning Barack Obama and Michelle Obama might be about their marriage, their children, a state dinner, or a policy disagreement. Labeling all of them spouse_of injects systematic noise. The labeling rule depends on the entity pair. Its errors can therefore be correlated with entity identity and context, and a model can learn a shortcut that fits those training labels while failing on different contexts.

Multi-instance learning (Riedel, Yao, and McCallum, 2010; Hoffmann et al., 2011) softens the assumption to the "at-least-one" form: the bag of sentences mentioning the pair expresses the relation, but not necessarily every sentence in the bag. The bag label is known; the instance labels are latent. Two aggregation choices illustrate the difference.

Average pooling can average sentence representations. For a nonempty bag $B$ with $m$ sentences and encoded sentence vectors $\mathbf{h}_i\in\mathbb{R}^d$, it gives:

$$\mathbf{s}_{\text{avg}} = \frac{1}{m}\sum_{i=1}^{m}\mathbf{h}_i$$

where $\mathbf{s}_{\text{avg}}$ has the same $d$ coordinates as each sentence vector. A classifier then maps that bag representation to relation scores. Averaging probabilities is another possible ensemble, but it is not the representation pooling described here.

Selective attention in Lin et al. (2016) weights the representations before classification. For a queried relation with learned vector $\mathbf{q}_r\in\mathbb{R}^d$ and learned diagonal matrix $A\in\mathbb{R}^{d\times d}$:

$$e_i = \mathbf{h}_i^\top A\mathbf{q}_r,\qquad \alpha_i = \frac{\exp(e_i)}{\sum_{j=1}^{m}\exp(e_j)},\qquad \mathbf{s}_r = \sum_{i=1}^{m}\alpha_i\mathbf{h}_i$$

where $e_i$ is the compatibility score, $\alpha_i$ is a nonnegative sentence weight, and $\mathbf{s}_r$ is the relation-conditioned bag vector. The weights sum to one for each queried relation. Lin et al. apply a learned affine classifier and softmax after this weighted representation; the attention weights themselves are not relation probabilities. The numerical illustration below uses fixed logits to show the weighting operation, without training or reproducing that full model.

If the at-least-one assumption holds for a positive bag, the bag label alone leaves the supporting sentences undetermined when there are several candidates. With only one candidate, that sentence must be the witness. Attention can concentrate on a supporting sentence, but it can also concentrate on a misleading one. If none of the sentences supports the label, normalization still assigns weights summing to one; it does not force those weights to be uniform or spread out. A large weight therefore does not establish that its sentence is valid evidence.

Attention is easiest to understand as a distribution over the bag. The computation below compares fixed-logit attention with uniform weighting for six synthetic sentence instances. A separate score marks an instance as positive if it is at least 0.5, which selects only S1.

Evaluation against inherited labels measures agreement with those labels, which can differ from sentence-level correctness. A majority-label predictor can score highly when that inherited label distribution is skewed. TACRED's re-annotation studies below concern a different source of error: mistakes in a human-annotated benchmark.

Programmatic supervision lets rules supply training signals. In Snorkel (Ratner et al., 2017), labeling functions can encode patterns, heuristics, or existing models; its generative model estimates source accuracies and correlations to combine noisy votes. DeepDive (Zhang et al., 2016) describes declarative extraction and distant-supervision rules feeding a graphical model that learns weights and infers fact probabilities. Snorkel DryBell (Bach et al., 2019) reports three classification case studies at Google and evaluates data representative of those production tasks. Pick the strongest individual labeling function on development data. Score that function and the combined labels on the same held-out gold examples, using the same metric and abstention rule. Report coverage alongside the score. Estimating source accuracies and correlations does not guarantee an improvement.

Neural relation models developed in the mid-2010s changed how systems represented input text, encoded entity positions, and pooled evidence from different parts of a sentence. The following designs expose the candidate arguments to the encoder or preserve their positions during pooling.

Position embeddingsLink Copied

Zeng et al. (2014) give each token two position features: its distance to the head entity and to the tail entity,

$$d_i^{(1)} = i - p_1, \qquad d_i^{(2)} = i - p_2$$

where $p_1, p_2$ are the reference token indices chosen for the first and second relation arguments. Let $\tilde{d}_i^{(k)}$ be $d_i^{(k)}$ clipped to the chosen window (for example, $[-30,30]$). Look up the learned vector for each clipped offset and concatenate the two vectors with the word embedding.

$$\mathbf{x}_i = [\mathbf{w}_i;\, \mathbf{d}^{(1)}_{\tilde{d}_i^{(1)}};\, \mathbf{d}^{(2)}_{\tilde{d}_i^{(2)}}]$$

The lookup supplies a vector for a distance such as three positions to the left of the head entity. A convolution can use those coordinates alongside the word embedding. This CNN design does not require a pretrained sentence encoder. Shaw et al. (2018) also encode relative positions in self-attention, but their distances are between token positions rather than between each token and two selected entities.

The clipping window is a practical compromise. Clipping to $[-30,30]$ gives 61 distance indices for each entity position, regardless of sentence length. Distances 30 and 300 then use the same boundary index. This bounds the table and discards that distance distinction; whether the discarded information affects accuracy is an empirical question.

Piecewise poolingLink Copied

A plain max-pooling convolution collapses the whole sentence into one vector, which conflates what happens before, between, and after the entities. Zeng et al. (2015) introduced piecewise max pooling: split the convolution output $\mathbf{h}_1, \dots, \mathbf{h}_L$ into three segments bounded by the two entities, and max-pool each separately:

$$\mathbf{p}_j = \max_{i \in \text{segment}_j} \mathbf{h}_i, \qquad j \in \{1,2,3\}$$

The coordinatewise maximum assumes each segment contains at least one representation. If the chosen boundary handling can produce an empty segment, the implementation needs an explicit convention for it; the displayed formula does not supply one.

The sentence representation is $[\mathbf{p}_1; \mathbf{p}_2; \mathbf{p}_3]$. This preserves three positional regions instead of pooling them together. Relation evidence can occur inside or outside the middle region; a performance gain must be measured on the chosen data rather than inferred from the segmentation alone.

In "Acme, which was founded in 2011, acquired Bolt," both the relative clause and the acquisition verb lie between the named entities. Piecewise pooling does not remove that background clause. It retains coarse entity-relative position, which the classifier can use alongside the learned convolution features.

Entity markers with pretrained encodersLink Copied

With a pretrained transformer, one way to inject entity information is to insert markers into the input text. Instead of relying on positions in a sequence of subword tokens, you wrap the entity spans:

The classifier can read the [CLS] representation or the concatenated [E1] and [E2] start-token representations. Soares et al. (2019) showed that this approach, combined with training on a large corpus of automatically generated entity-pair instances, produces relation representations that transfer to new relation types with very little supervision. The markers expose the selected pair to the encoder; the [CLS] representation can therefore depend on those boundaries. This input construction does not guarantee that every pair receives a distinct representation.

Entity markers expose the chosen boundaries in the encoder input. A system starting from character spans still needs to insert them at the intended locations, including when tokenization splits a word. In Soares et al.'s marker representation, designated start-marker embeddings identify the two arguments; other boundary conventions need their own tokenization and embedding specification.

Record the marker convention when reproducing a model. If each boundary should occupy one token, use an existing reserved token or register the marker and supply an embedding for it. Other designs can use multi-token boundary strings. Typed markers such as [E1:ORG] expose type information. Zhou and Chen (2022) compared marker formats with BERT and RoBERTa on TACRED and found different F1 scores. Report the dataset, encoder, and training setup when comparing marker designs. Changing a type inventory requires deciding how those new labels will be encoded; it does not invariably require a new tokenizer vocabulary or retraining.

A different formulation is to ask a question. Levy et al. (2017) associated each relation slot with one or more natural-language question templates. For an acquisition, a question could be:

acquire → "What did Acme Corp acquire?"

Levy et al. trained a reading-comprehension model to answer the question from a sentence context. At extraction time, they applied the reader to sentences in the documents of interest. A non-null answer span supplies the other argument of the predicted relation. A correct prediction still requires the question and source context to support that relation.

The formulation can use reading-comprehension models that support unanswerable questions. A new relation can be described by a new template without adding a relation-specific output head, although accurate zero-shot extraction is not guaranteed. Levy et al. evaluated this possibility on a Wikipedia slot-filling task.

The QA formulation also changes the burden of schema design. A question template expresses the relation type in words a human can read and evaluate. A template needs an answer criterion consistent with the relation definition. Ambiguous wording can come from the template even when the schema is precise, so review both the label definition and its verbalization.

The cost depends on how questions are constructed. For 50 entities and 20 types, a design with one verification question per ordered pair of distinct entities and relation type gives $50\times49\times20=49{,}000$ question formulations. That enumeration is not inherent to slot-filling QA. A question such as "What did Acme acquire?" supplies a head and relation and asks the model to find the tail; one question can therefore cover several candidate tails. With one template per type, 50 heads and 20 types give 1,000 question formulations before candidate pruning or application to individual sentences or chunks. Levy et al. (2017) use this head-and-relation slot-filling formulation.

Document-level relation extraction adds cross-sentence evidence and reference resolution to the difficulties of sentence-level extraction. Sentence-level benchmarks still expose errors, including annotation noise and ambiguous relation definitions.

Consider this illustrative paragraph:

Acme Corp announced on 3 March 2019 that it had acquired Bolt Systems. The deal closed two weeks later. Acme's chief executive, Jane Ruiz, said the acquisition would expand the company's cloud business.

The chief-executive sentence directly supports a relation between Jane Ruiz and the surface name "Acme". Mapping that name to "Acme Corp" adds an identity-resolution step. The acquisition sentence uses "it" for its buyer. Resolving that pronoun to Acme Corp uses evidence in the same sentence. Some outputs need document context even when other relations in the same paragraph are locally explicit.

The second sentence, "The deal closed two weeks later," gives a closing event and a relative time without repeating the names Acme or Bolt. A candidate generator restricted to named-entity pairs in that sentence has no Acme–Bolt pair to classify. Connecting "the deal" to the earlier acquisition supplies the event reference that this local candidate set omits. In the worked reading, the earlier announcement also supplies the reference date for "two weeks later".

DocRED (Yao et al., 2019) made this concrete as a benchmark: documents from Wikipedia paired with Wikidata facts, requiring reading comprehension and coreference reasoning, with the authors reporting that over 40 percent of relational facts in their human-annotated corpus require evidence from more than one sentence. Several model families address document-level relations:

• Graph-based models build graphs with token nodes and propagate information with graph convolutions or Graph LSTMs (Peng et al., 2017; Sahu et al., 2019).
• Context aggregation encodes the document once, pools mentions separately for each entity, and combines entity representations for pair classification (Zhou et al., 2021). ATLOP pools mention embeddings with logsumexp and also constructs a localized context representation.
• Structure-aware attention (Xu et al., 2021) models dependencies between mentions of the same entity and between mentions of different entities, injecting structural priors about how mentions corefer and co-occur.

The graph-based approach augments the token sequence with selected syntactic and cross-sentence structure. In the cited token-graph models, edges encode selected syntactic or cross-sentence connections. Message passing transfers representations along those edges; the graph construction determines which paths are available. In the cited context-aggregation design, the document is encoded once, the mentions of each entity are pooled into a separate entity representation, and the head and tail representations are combined for classification. This aggregation does not require the same explicit graph edges. Its accuracy still depends on the encoder, pooling rule, training data, and evaluation protocol.

ATLOP (Zhou et al., 2021) learns a per-pair adaptive threshold rather than comparing every relation score with one fixed cutoff. Its special threshold class separates positive from negative relation scores for a pair; it should not be confused with a calibrated probability that the pair has no relation. Skewed labels can motivate threshold evaluation, but an adaptive threshold is not a universal remedy for imbalance. The loss, score calibration, decision costs, and desired evaluation metric also matter.

The next figure is an illustrative calculation, not an ATLOP measurement. It holds the two synthetic score distributions fixed and changes only positive-class prevalence. On the displayed threshold grid, the F1-maximizing thresholds are 0.50, 0.55, 0.60, 0.60, 0.65, and 0.70 as prevalence decreases. A cutoff of 0.50 is optimal for the most prevalent row and is below the optimum for the other five rows under these particular assumptions. For prevalence 0.25, F1 is 0.887 at threshold 0.50 and 0.897 at threshold 0.55, rounded to three decimals. Real score distributions need their own held-out threshold evaluation.

The problem is not specific to one threshold value. It follows from the interaction between prevalence and decision cutoffs, which the next computation isolates before plotting.

A per-pair threshold supplies a decision boundary for that pair's relation scores. It does not by itself establish probability calibration; threshold quality still needs evaluation under the target label distribution. The same prevalence calculation can be applied to binary role-fill decisions if their score distributions are held fixed.

Relation extraction scores depend on the evaluation protocol.

The basic metrics are micro-averaged precision, recall, and $F_1$ over extracted tuples. Three protocol choices affect the reported score:

• Mention-level vs entity-level. DocRED-style evaluation predicts a set of relations per entity pair using document evidence. TACRED-style evaluation classifies a supplied mention pair. Their scores describe different tasks and datasets.

• Direction of the pair. Suppose the gold acquisition tuple is only (Bolt, acquire, Acme) and the system predicts only (Acme, acquire, Bolt). A directed evaluation counts a false positive for the predicted tuple and a false negative for the missed gold tuple. An evaluation that ignores direction can credit a reversed tuple that a directed evaluation rejects. State which rule is used.

• Label noise. Alt, Gabryszak, and Hennig (2020) inspected roughly 5,000 challenging development and test examples, rather than a random sample of all TACRED examples. More than half of that selected set needed relabeling; average baseline test F1 rose from 62.1 to 70.1 after the corrections. Stoica, Platanios, and Póczos (2021) re-annotated the full dataset for Re-TACRED and reported 23.9 percent incorrect labels. Their Table 1 gives F1 gains of 13.2, 14.0, and 15.6 points for three models, whose mean rounds to 14.3 points. Report the dataset revision and per-relation scores alongside micro-F1.

A fourth protocol choice is how negative examples are constructed. Some evaluations use the closed-world assumption and label an eligible entity pair negative when the test knowledge base lists no relation for it. The knowledge base can omit a true relation, so a correct prediction on that pair can count as a false positive. Sampling negatives changes the evaluated prevalence when the sampling ratio differs from the full candidate set. With true- and false-positive rates held fixed, sampling can change precision without changing recall. Report how negatives were labeled and sampled so readers can interpret prevalence-sensitive metrics such as precision and F1.