Relation, Event, and Temporal Extraction

An article can mention Acme Corp and Bolt Systems without saying that either bought the other. To build an acquisition record, recover the buyer and target when the text identifies them, retain the supporting text, and record timing at the precision the source supplies. Leave missing participants or timing unknown. You also need to distinguish a completed acquisition from a reported intention to acquire. Entity names alone cannot answer those questions.

A list of names can tell you that an article mentions Acme and Bolt, but it cannot tell you whether Acme bought Bolt, whether Bolt bought Acme, whether the two are partners, or whether the article merely compares them. Recovering the statement adds information that a list of names omits: argument roles, the predicate, and whether the text asserts it. An extracted record states a claim. Whether the source supports it, and whether it is true in the world, are separate questions.

That structure comes in three related forms. Each can be studied and evaluated separately: a relation classifier, for example, can predict a label for two supplied entity spans without extracting events or dates. The worked examples here connect their outputs at document level.

• Relations are links between entities: (Acme Corp, acquired, Bolt Systems). The binary examples here represent statements about entities using a typed link. The schema supplies the name of the link, such as acquired, founder_of, or chief_executive_of, and the extraction system decides whether the link is supported by a particular piece of text.
• Event records can represent described occurrences or situations with a trigger expression, arguments filling roles, and attributes such as time, place, and price. Under the ACE annotation guidelines, triggers are often verbs or nominalizations, with other forms allowed. Acquisition(trigger=acquired, Buyer=Acme Corp, Acquired=Bolt Systems, Price=$120M, Time=unknown). Such records can retain more structure than a binary link: multiple roles, event-valued arguments when the schema permits them, modality, and polarity.
• Temporal expressions and relations ground events on a timeline and order them relative to one another: E1 before E2. E3’s date may fall anywhere in 2011 when only the year is known. Temporal relations let us ask which events precede others and retain unresolved orderings.

Document context adds decisions about references and evidence aggregation to these tasks. In Named Entity Recognition and BIO Tagging we labeled spans inside sentences. Here the unit of analysis shifts upward. A relation's two arguments may sit four paragraphs apart. An event's time may be expressed as "two weeks later," which requires a reference time. A pronoun like "it" may be the buyer in the most important sentence of the document.

This shift from sentence-level to document-level extraction is the central theme. As we'll see, it changes the models, the evaluation, and the error analysis. A supplied-pair sentence classifier and a document-level entity-pair extractor have different inputs and prediction units. Compare their scores only after stating those units and the gold annotations. Document-level models can aggregate evidence across mentions and use reference information, as the DocRED examples below illustrate.

Confusing these levels has practical consequences. A model that is excellent at mention-level relation classification may look mediocre when its predictions are aggregated to the entity level, and that drop is not necessarily a modeling failure. Changing the prediction unit and aggregation rule can change F1 even when the underlying mention predictions stay fixed. Similarly, an event mention is not an event: the same acquisition may be mentioned in a headline, in the body of an article, and in a follow-up piece a week later. If several mentions refer to one occurrence, counting each mention as a distinct event overcounts occurrences. Merging those references is the event-coreference task.

The following one-way pipeline illustrates how an early decision can affect later records.

1. Mention detection: find entity, event trigger, and timex spans. One way to do this is sequence labeling, covered in Part VI.
2. Entity linking: map mentions to canonical identifiers in a knowledge base. We'll cover this in the next chapter.
3. Relation and event extraction: build typed structures over those canonical entities.
4. Temporal grounding: attach times and compute orderings.
5. Aggregation: merge structures across mentions, sentences, and documents into a knowledge base.

Steps 3 and 4 are this chapter. Step 2 is Entity Linking and Knowledge Graph Construction, and step 5 is really a data engineering discipline as much as a modeling one.

This is one useful pipeline order, rather than a requirement for every architecture. The explicit span pipeline illustrated above detects mentions before classifying structures over them; joint and generative models can predict these outputs together. Entity linking supports aggregation across aliases, although a system can aggregate surface strings before resolving their identities. Relation and event extraction recover statements, temporal grounding attaches time information, and aggregation must preserve unresolved identities and contradictions rather than silently merging them.

Each stage also has its own error profile. Mention detection misses spans, entity linking conflates entities, relation extraction invents links, event extraction confuses roles, and temporal normalization assigns the wrong date. In a one-way pipeline, an early mistake can survive all the way to the final knowledge base. A missed organization mention in the first stage can cause a relation to disappear, which can cause an event to lose an argument, which can cause a timeline to be incomplete. Keeping provenance lets a reviewer trace a suspicious final record back to its claimed supporting span.

These tasks predict structures anchored in text. Relation arguments and event triggers can identify spans, while temporal normalization also outputs values such as dates and durations. A system can use token labels for recognition and then assemble structured records. That has three consequences.

First, the output space is combinatorial. Enumerating all ordered pairs of distinct entity mentions gives $n(n-1)$ candidates. A document with 40 mentions has 1,560 such pairs; one with 100 has 9,900. Positive-label prevalence depends on the corpus and schema. Candidate pruning can reduce computation, although batching or joint scoring can also handle many pairs without discarding them.

Count the candidate pairs before choosing an exhaustive scoring design. The following computation makes the candidate space explicit before any model scores a pair.

The candidate count sets the number of pair scores in an exhaustive design. Type constraints and distance filters can discard candidates; batching can instead change how those scores are computed. A filter can also discard a true relation, so compare its recall and computational cost.

Second, records can retain span provenance. A record can point back to the text using evidence strings and offsets. These references enable substring and location checks, although a wrong relation can pass both checks when its strings occur in the source. The Evidence and Grounding example shows offsets. The LLM example checks whether selected strings occur in the document. If a system extracts Acme’s report that it acquired Bolt, it should be able to identify the tokens that support that attributed claim. That evidence is useful for debugging, for audit, and for deciding whether to trust a prediction.

Third, the schema is a design decision. Deciding whether Acme acquired Bolt is one relation or an Acquisition event with fields for the buyer, target, price, and time, and whether Acme and Acme Corp are the same node, determines what your downstream system can answer. Open IE leaves the relation phrases open while still choosing output fields, such as subject, predicate, and object. Canonical queries require a mapping between those predicates and the requested relation types.

The three tasks differ in output structure, which constrains what a model must predict and informs its design.

Keeping these apart matters. A paper reporting "F1 on event extraction" may mean trigger classification, argument identification, or argument role classification. These scores can differ within one experiment: the JRNN result discussed below reports 69.3 trigger-classification F1 and 55.4 argument-role F1. Comparing the numbers without checking their prediction units can produce a ranking of systems measured on different tasks.

The same ambiguity exists in relation extraction. Mention-level F1 and entity-level F1 measure different decisions, and a system that looks strong at the mention level may look much weaker when it must aggregate evidence across a document. Temporal extraction has an analogous split between timex recognition, timex normalization, and pairwise temporal relation classification. These are separate prediction targets; report each metric when its corresponding task is in scope.

Our binary relation record contains two entities and a link. Choosing that link involves a schema, an extraction method, and an evaluation protocol.

The apparent simplicity is deceptive. "Acme acquired Bolt" looks like a binary predicate, but the same fact can be expressed as "Bolt was acquired by Acme," "Acme's acquisition of Bolt," or "Acme bought Bolt." The phrase "the Acme-Bolt deal" needs context to identify the relation and its direction. A relation extraction system must decide which of these expressions count as evidence for the same relation, which words are the arguments, and whether the relation is directed. Those decisions depend on the schema, and the schema depends on what the downstream application needs to do.

Relation types have an arity, the number of argument positions they take. The single-label classifier for binary relations below takes a document $D$ and an ordered pair of entity mentions $(e_1,e_2)$. Its class set $\mathcal{R}$ contains the binary relation labels and no_relation. For $r\in\mathcal{R}$, it predicts

where $\mathbf{h}\in\mathbb{R}^d$ represents the document conditioned on the entity positions, $W$ maps its $d$ coordinates to one logit per class, and $\mathbf{b}$ is the class-bias vector. Softmax normalizes these logits over mutually exclusive labels, including no_relation. This is a single-label formulation; it is not sufficient when a pair can have several relation labels. At inference, a decision rule can select the highest-probability class and optionally abstain below a chosen confidence threshold.

The representation $\mathbf{h}$ can be as simple as a bag-of-words feature vector over the tokens between the two mentions, or as complex as a transformer encoding of the entire document with entity markers. The output formula stays the same across these different representations. The important point is that the decision is conditioned on the selected mention pair. The system is not classifying the document in isolation; it is asking whether this particular pair of mentions stands in this particular relation.

Specify three properties of the schema:

• Direction. (Jane Ruiz, employer_of, Acme) and (Acme, employer_of, Jane Ruiz) are different statements. An unordered pair representation alone cannot distinguish these two argument orders.
• Symmetry. Some relations are symmetric (spouse_of), some antisymmetric (parent_of), some neither. A declared symmetry constraint can be imposed at inference. If a relation is known to be symmetric and the model predicts it for one ordering, the inverse ordering should receive the same prediction.
• Multiplicity. Can one entity pair hold multiple relations at once? A schema could allow both (Jane Ruiz, employee_of, Acme) and (Jane Ruiz, chief_executive_of, Acme). These use the same ordered pair. Co-occurring labels require a multi-label output, such as independent binary decisions, rather than one mutually exclusive softmax decision.

These properties interact. In exhaustive classification under a directed, multi-label schema, there is a binary decision for each relation type and each ordered pair, so the number of decisions grows with both counts. For a symmetric relation, the two orderings of each pair of distinct candidates share one prediction. This halves the number of independent pair decisions for that relation. If the schema has many mutually exclusive relation types, a softmax is appropriate. If the types can co-occur, independent sigmoids are more appropriate. A single categorical decision cannot represent two co-occurring labels; a multi-label output can.

Aggregation combines evidence of different relevance. Suppose a document contains three mentions of Acme and three mentions of Bolt. One sentence says that Acme acquired Bolt. Another lists both companies as competitors. A third says that a former Acme executive joined Bolt. All three sentences mention both organizations, but only the first supports acquisition. The third relates the executive to Bolt. A system can pool their representations or scores, but the aggregation rule needs to distinguish acquisition evidence from co-occurrence and other relations.

Closed-schema extraction and Open IE make different choices about the allowed predicates.

Closed-schema extraction fixes the relation inventory in advance. Examples include the ACE 2005 relation inventory (18 subtypes under six top-level types, including Physical, Part-whole, Personal-Social, and ORG-Affiliation), the TAC KBP slot set, a fixed subset of Wikidata properties, and domain ontologies like the UMLS semantic network for biomedical text. A fixed inventory supplies shared output labels for evaluation and aggregation. It cannot emit an unlisted relation type without changing that inventory, although the system can retain the source text and unmapped predicates. If the inventory has no suitable type for a minority investment, it cannot represent that investment as a canonical relation without changing the schema.

Open information extraction (Open IE) derives predicates from text without a fixed list of relation types. A triple can retain a surface phrase, as in ("Acme Corp", "had acquired", "Bolt Systems"), or use a normalized phrase. Examples include TextRunner (Banko et al., 2007), ReVerb (Fader, Soderland, and Etzioni, 2011), and Stanford OpenIE. Their patterns and output conventions differ. Its models and rules still limit coverage. Surface predicates such as had acquired, bought, purchased, and took over need normalization if the application wants to query one canonical relation.

In the surface-triple form above, arguments are text strings rather than supplied knowledge-base identifiers. A subject such as "the company" still needs a reference decision to identify its organization. Surface triples can be queried as strings. Queries that should join aliases or combine predicate paraphrases need additional identity and predicate mappings; otherwise they can miss records stored under another surface form.

One possible design extracts surface predicates, maps them to canonical relations, and keeps the surface form as provenance. A classifier can learn that mapping from labeled examples. Clustering can group candidate paraphrases, for example by representing predicates with argument-type count vectors and grouping similar vectors. The resulting clusters still need alignment to the application's relation labels. Evaluate extraction and mapping errors separately; combining the stages does not guarantee a recall gain. The stored surface predicate lets an auditor see which phrase was mapped to the canonical relation.

Patterns encode selected constructions directly. Xu et al. (2009) report high precision and reasonable recall for their learned dependency patterns; Hennig et al. (2015) document learned patterns damaged by parsing and supervision errors. Compare precision and recall on the target data when choosing between patterns and learned extractors.

The classic example is Hearst patterns (Hearst, 1992) for hyponymy. The lexico-syntactic frames

are cues that $Y$ is a kind of $X$. The same technique generalizes: "PERSON, (?:the )?(?:CEO|chief executive) of ORG" indicates a ceo_of relation; "ORG (?:acquired|bought|purchased) ORG" indicates an acquisition.

Hearst patterns illustrate how to encode a hypothesis about linguistic evidence. "X such as Y" can indicate a hyponym relation, but a match is not proof: parsing, scope, quotation, and nonliteral uses can change the interpretation. Evaluate precision on the target domain before using a pattern without human review.

The following examples compare dependency conditions with surface patterns.

Patterns can operate over dependency paths, which can handle intervening material and some word-order variation. The active-voice pattern nsubj(acquire, X) ∧ dobj(acquire, Y) covers "Acme acquired Bolt" and can tolerate an intervening phrase. "Bolt was acquired by Acme" needs a separate passive pattern, with the passive subject as the acquired entity and the by-phrase as buyer. Curated patterns and validators can provide a precision-oriented component, but parser errors and unhandled context prevent a general precision guarantee.

A dependency rule exposes syntactic conditions that can be inspected alongside the parse. A surface regex that fires on "Acme acquired Bolt" may also fire on "Acme denied that Acme acquired Bolt," which does not affirm the acquisition. One conservative rule for affirmative acquisitions in the main clause can require the acquisition verb to be the sentence’s dependency root and check it for negation and modal auxiliaries. In the denial example, "acquired" heads the complement of "denied", so it fails the root check. The dependency representation supplies clause-head and modifier relations directly; a regex over raw text does not receive those parse labels. The tradeoff is that dependency patterns depend on parser accuracy, so a parsing error can cause a pattern to miss a valid relation or fire on an invalid one.

Check coverage and portability on the target data. A new domain, language, or writing style can require pattern changes; evaluate existing patterns on the target data before adding rules. And patterns interact: a pattern that fires on "Acme's chief executive, Jane Ruiz" can also fire on "Acme's former chief executive, Jane Ruiz," which asserts something quite different. Contextual modifiers (former, alleged, would) must be handled explicitly, which statistical models try to learn from data.

Pattern-based extraction can serve as a precision-oriented component within a broader system. It can seed a training set, cross-check a learned model's high-confidence predictions, or handle a small set of business-critical relations where the schema is stable and the cost of a false positive is high. Learned components are one way to extend coverage beyond a hand-written inventory; adding patterns is another. Measure both coverage and false positives on the target data.

We'll build a small pattern and dependency-rule extractor in the implementation section.

Mintz et al. (2009) motivate distant supervision with the cost and limited quantity of manually labeled relation data. Their method treats co-mentions of a known relation’s entity pair as possible evidence. Riedel et al. (2010) give a stronger formulation, paraphrased here.

If a knowledge base already asserts that entity $e_1$ stands in relation $r$ to $e_2$, then every sentence mentioning both $e_1$ and $e_2$ expresses $r$.

Riedel et al. found co-mentions that did not express the relation in their manually checked samples. Mintz et al. still obtained large training sets with Freebase supervision. In the Mintz et al. procedure, co-mentioning sentences contribute features to a vector for a known relation–entity-pair tuple; the KB supplies the tuple's positive label. The authors also sample negative training instances. State the negative-label rule rather than assuming that every unlisted fact is false.

The appeal is easy to see. Mintz et al. report 116 million relation instances in the Freebase snapshot they describe. The filtered Wikidata5M dataset contains millions of relation triples. If millions of those instances each yield a usable sentence bag, you have millions of weakly labeled training bags. The labels are inherited from the knowledge base rather than individually adjudicated for each sentence. Missing or incorrect knowledge-base facts can therefore create misleading training labels. This is a risk of the labeling procedure, not a guarantee that every trained model reproduces every labeling error.

The problem is that the assumption is too strong. A sentence mentioning Barack Obama and Michelle Obama might be about their marriage, their children, a state dinner, or a policy disagreement. Labeling all of them spouse_of injects systematic noise. The labeling rule depends on the entity pair. Its errors can therefore be correlated with entity identity and context, and a model can learn a shortcut that fits those training labels while failing on different contexts.

Multi-instance learning (Riedel, Yao, and McCallum, 2010; Hoffmann et al., 2011) softens the assumption to the "at-least-one" form: the bag of sentences mentioning the pair expresses the relation, but not necessarily every sentence in the bag. The bag label is known; the instance labels are latent. Two aggregation choices illustrate the difference.

Average pooling can average sentence representations. For a nonempty bag $B$ with $m$ sentences and encoded sentence vectors $\mathbf{h}_i\in\mathbb{R}^d$, it gives:

where $\mathbf{s}_{\text{avg}}$ has the same $d$ coordinates as each sentence vector. A classifier then maps that bag representation to relation scores. Averaging probabilities is another possible ensemble, but it is not the representation pooling described here.

Selective attention in Lin et al. (2016) weights the representations before classification. For a queried relation with learned vector $\mathbf{q}_r\in\mathbb{R}^d$ and learned diagonal matrix $A\in\mathbb{R}^{d\times d}$:

where $e_i$ is the compatibility score, $\alpha_i$ is a nonnegative sentence weight, and $\mathbf{s}_r$ is the relation-conditioned bag vector. The weights sum to one for each queried relation. Lin et al. apply a learned affine classifier and softmax after this weighted representation; the attention weights themselves are not relation probabilities. The numerical illustration below uses fixed logits to show the weighting operation, without training or reproducing that full model.

If the at-least-one assumption holds for a positive bag, the bag label alone leaves the supporting sentences undetermined when there are several candidates. With only one candidate, that sentence must be the witness. Attention can concentrate on a supporting sentence, but it can also concentrate on a misleading one. If none of the sentences supports the label, normalization still assigns weights summing to one; it does not force those weights to be uniform or spread out. A large weight therefore does not establish that its sentence is valid evidence.

Attention is easiest to understand as a distribution over the bag. The computation below compares fixed-logit attention with uniform weighting for six synthetic sentence instances. A separate score marks an instance as positive if it is at least 0.5, which selects only S1.

Evaluation against inherited labels measures agreement with those labels, which can differ from sentence-level correctness. A majority-label predictor can score highly when that inherited label distribution is skewed. TACRED's re-annotation studies below concern a different source of error: mistakes in a human-annotated benchmark.

Programmatic supervision lets rules supply training signals. In Snorkel (Ratner et al., 2017), labeling functions can encode patterns, heuristics, or existing models; its generative model estimates source accuracies and correlations to combine noisy votes. DeepDive (Zhang et al., 2016) describes declarative extraction and distant-supervision rules feeding a graphical model that learns weights and infers fact probabilities. Snorkel DryBell (Bach et al., 2019) reports three classification case studies at Google and evaluates data representative of those production tasks. Pick the strongest individual labeling function on development data. Score that function and the combined labels on the same held-out gold examples, using the same metric and abstention rule. Report coverage alongside the score. Estimating source accuracies and correlations does not guarantee an improvement.

Neural relation models developed in the mid-2010s changed how systems represented input text, encoded entity positions, and pooled evidence from different parts of a sentence. The following designs expose the candidate arguments to the encoder or preserve their positions during pooling.

Position embeddingsLink Copied

Zeng et al. (2014) give each token two position features: its distance to the head entity and to the tail entity,

where $p_1, p_2$ are the reference token indices chosen for the first and second relation arguments. Let $\tilde{d}_i^{(k)}$ be $d_i^{(k)}$ clipped to the chosen window (for example, $[-30,30]$). Look up the learned vector for each clipped offset and concatenate the two vectors with the word embedding.

The lookup supplies a vector for a distance such as three positions to the left of the head entity. A convolution can use those coordinates alongside the word embedding. This CNN design does not require a pretrained sentence encoder. Shaw et al. (2018) also encode relative positions in self-attention, but their distances are between token positions rather than between each token and two selected entities.

The clipping window is a practical compromise. Clipping to $[-30,30]$ gives 61 distance indices for each entity position, regardless of sentence length. Distances 30 and 300 then use the same boundary index. This bounds the table and discards that distance distinction; whether the discarded information affects accuracy is an empirical question.

Piecewise poolingLink Copied

A plain max-pooling convolution collapses the whole sentence into one vector, which conflates what happens before, between, and after the entities. Zeng et al. (2015) introduced piecewise max pooling: split the convolution output $\mathbf{h}_1, \dots, \mathbf{h}_L$ into three segments bounded by the two entities, and max-pool each separately:

The coordinatewise maximum assumes each segment contains at least one representation. If the chosen boundary handling can produce an empty segment, the implementation needs an explicit convention for it; the displayed formula does not supply one.

The sentence representation is $[\mathbf{p}_1; \mathbf{p}_2; \mathbf{p}_3]$. This preserves three positional regions instead of pooling them together. Relation evidence can occur inside or outside the middle region; a performance gain must be measured on the chosen data rather than inferred from the segmentation alone.

In "Acme, which was founded in 2011, acquired Bolt," both the relative clause and the acquisition verb lie between the named entities. Piecewise pooling does not remove that background clause. It retains coarse entity-relative position, which the classifier can use alongside the learned convolution features.

Entity markers with pretrained encodersLink Copied

With a pretrained transformer, one way to inject entity information is to insert markers into the input text. Instead of relying on positions in a sequence of subword tokens, you wrap the entity spans:

The classifier can read the [CLS] representation or the concatenated [E1] and [E2] start-token representations. Soares et al. (2019) showed that this approach, combined with training on a large corpus of automatically generated entity-pair instances, produces relation representations that transfer to new relation types with very little supervision. The markers expose the selected pair to the encoder; the [CLS] representation can therefore depend on those boundaries. This input construction does not guarantee that every pair receives a distinct representation.

Entity markers expose the chosen boundaries in the encoder input. A system starting from character spans still needs to insert them at the intended locations, including when tokenization splits a word. In Soares et al.'s marker representation, designated start-marker embeddings identify the two arguments; other boundary conventions need their own tokenization and embedding specification.

Record the marker convention when reproducing a model. If each boundary should occupy one token, use an existing reserved token or register the marker and supply an embedding for it. Other designs can use multi-token boundary strings. Typed markers such as [E1:ORG] expose type information. Zhou and Chen (2022) compared marker formats with BERT and RoBERTa on TACRED and found different F1 scores. Report the dataset, encoder, and training setup when comparing marker designs. Changing a type inventory requires deciding how those new labels will be encoded; it does not invariably require a new tokenizer vocabulary or retraining.

A different formulation is to ask a question. Levy et al. (2017) associated each relation slot with one or more natural-language question templates. For an acquisition, a question could be:

acquire → "What did Acme Corp acquire?"

Levy et al. trained a reading-comprehension model to answer the question from a sentence context. At extraction time, they applied the reader to sentences in the documents of interest. A non-null answer span supplies the other argument of the predicted relation. A correct prediction still requires the question and source context to support that relation.

The formulation can use reading-comprehension models that support unanswerable questions. A new relation can be described by a new template without adding a relation-specific output head, although accurate zero-shot extraction is not guaranteed. Levy et al. evaluated this possibility on a Wikipedia slot-filling task.

The QA formulation also changes the burden of schema design. A question template expresses the relation type in words a human can read and evaluate. A template needs an answer criterion consistent with the relation definition. Ambiguous wording can come from the template even when the schema is precise, so review both the label definition and its verbalization.

The cost depends on how questions are constructed. For 50 entities and 20 types, a design with one verification question per ordered pair of distinct entities and relation type gives $50\times49\times20=49{,}000$ question formulations. That enumeration is not inherent to slot-filling QA. A question such as "What did Acme acquire?" supplies a head and relation and asks the model to find the tail; one question can therefore cover several candidate tails. With one template per type, 50 heads and 20 types give 1,000 question formulations before candidate pruning or application to individual sentences or chunks. Levy et al. (2017) use this head-and-relation slot-filling formulation.

Document-level relation extraction adds cross-sentence evidence and reference resolution to the difficulties of sentence-level extraction. Sentence-level benchmarks still expose errors, including annotation noise and ambiguous relation definitions.

Consider this illustrative paragraph:

Acme Corp announced on 3 March 2019 that it had acquired Bolt Systems. The deal closed two weeks later. Acme's chief executive, Jane Ruiz, said the acquisition would expand the company's cloud business.

The chief-executive sentence directly supports a relation between Jane Ruiz and the surface name "Acme". Mapping that name to "Acme Corp" adds an identity-resolution step. The acquisition sentence uses "it" for its buyer. Resolving that pronoun to Acme Corp uses evidence in the same sentence. Some outputs need document context even when other relations in the same paragraph are locally explicit.

The second sentence, "The deal closed two weeks later," gives a closing event and a relative time without repeating the names Acme or Bolt. A candidate generator restricted to named-entity pairs in that sentence has no Acme–Bolt pair to classify. Connecting "the deal" to the earlier acquisition supplies the event reference that this local candidate set omits. In the worked reading, the earlier announcement also supplies the reference date for "two weeks later".

DocRED (Yao et al., 2019) made this concrete as a benchmark: documents from Wikipedia paired with Wikidata facts, requiring reading comprehension and coreference reasoning, with the authors reporting that over 40 percent of relational facts in their human-annotated corpus require evidence from more than one sentence. Several model families address document-level relations:

• Graph-based models build graphs with token nodes and propagate information with graph convolutions or Graph LSTMs (Peng et al., 2017; Sahu et al., 2019).
• Context aggregation encodes the document once, pools mentions separately for each entity, and combines entity representations for pair classification (Zhou et al., 2021). ATLOP pools mention embeddings with logsumexp and also constructs a localized context representation.
• Structure-aware attention (Xu et al., 2021) models dependencies between mentions of the same entity and between mentions of different entities, injecting structural priors about how mentions corefer and co-occur.

The graph-based approach augments the token sequence with selected syntactic and cross-sentence structure. In the cited token-graph models, edges encode selected syntactic or cross-sentence connections. Message passing transfers representations along those edges; the graph construction determines which paths are available. In the cited context-aggregation design, the document is encoded once, the mentions of each entity are pooled into a separate entity representation, and the head and tail representations are combined for classification. This aggregation does not require the same explicit graph edges. Its accuracy still depends on the encoder, pooling rule, training data, and evaluation protocol.

ATLOP (Zhou et al., 2021) learns a per-pair adaptive threshold rather than comparing every relation score with one fixed cutoff. Its special threshold class separates positive from negative relation scores for a pair; it should not be confused with a calibrated probability that the pair has no relation. Skewed labels can motivate threshold evaluation, but an adaptive threshold is not a universal remedy for imbalance. The loss, score calibration, decision costs, and desired evaluation metric also matter.

The next figure is an illustrative calculation, not an ATLOP measurement. It holds the two synthetic score distributions fixed and changes only positive-class prevalence. On the displayed threshold grid, the F1-maximizing thresholds are 0.50, 0.55, 0.60, 0.60, 0.65, and 0.70 as prevalence decreases. A cutoff of 0.50 is optimal for the most prevalent row and is below the optimum for the other five rows under these particular assumptions. For prevalence 0.25, F1 is 0.887 at threshold 0.50 and 0.897 at threshold 0.55, rounded to three decimals. Real score distributions need their own held-out threshold evaluation.

The problem is not specific to one threshold value. It follows from the interaction between prevalence and decision cutoffs, which the next computation isolates before plotting.

A per-pair threshold supplies a decision boundary for that pair's relation scores. It does not by itself establish probability calibration; threshold quality still needs evaluation under the target label distribution. The same prevalence calculation can be applied to binary role-fill decisions if their score distributions are held fixed.

Relation extraction scores depend on the evaluation protocol.

The basic metrics are micro-averaged precision, recall, and $F_1$ over extracted tuples. Three protocol choices affect the reported score:

• Mention-level vs entity-level. DocRED-style evaluation predicts a set of relations per entity pair using document evidence. TACRED-style evaluation classifies a supplied mention pair. Their scores describe different tasks and datasets.

• Direction of the pair. Suppose the gold acquisition tuple is only (Bolt, acquire, Acme) and the system predicts only (Acme, acquire, Bolt). A directed evaluation counts a false positive for the predicted tuple and a false negative for the missed gold tuple. An evaluation that ignores direction can credit a reversed tuple that a directed evaluation rejects. State which rule is used.

• Label noise. Alt, Gabryszak, and Hennig (2020) inspected roughly 5,000 challenging development and test examples, rather than a random sample of all TACRED examples. More than half of that selected set needed relabeling; average baseline test F1 rose from 62.1 to 70.1 after the corrections. Stoica, Platanios, and Póczos (2021) re-annotated the full dataset for Re-TACRED and reported 23.9 percent incorrect labels. Their Table 1 gives F1 gains of 13.2, 14.0, and 15.6 points for three models, whose mean rounds to 14.3 points. Report the dataset revision and per-relation scores alongside micro-F1.

A fourth protocol choice is how negative examples are constructed. Some evaluations use the closed-world assumption and label an eligible entity pair negative when the test knowledge base lists no relation for it. The knowledge base can omit a true relation, so a correct prediction on that pair can count as a false positive. Sampling negatives changes the evaluated prevalence when the sampling ratio differs from the full candidate set. With true- and false-positive rates held fixed, sampling can change precision without changing recall. Report how negatives were labeled and sampled so readers can interpret prevalence-sensitive metrics such as precision and F1.

An event record describes an occurrence with participants in specific roles and can include time, place, and factuality attributes. Those fields support queries beyond the binary relation, and each creates a separate correctness condition. A record for a planned or denied event does not assert that it occurred.

A binary relation fits a single edge in a knowledge graph. In a small event subgraph, labeled role edges connect the event record to its arguments. A mention record links the event to its trigger span, and the event record can carry attributes. That extra structure allows a system to answer questions like "who bought whom, for how much, and when?" An evaluation can assess trigger identification, event typing, argument identification, role assignment, and factuality separately.

In the ACE program's formulation, an event mention consists of:

• A trigger: the word or phrase that most clearly evokes the event being described, whether asserted, negated, or hypothetical. It can be a verb ("acquired", "died", "resigned"), a nominalization ("acquisition" in "the acquisition", "resignation" in "his resignation"), or an adjective ("married").
• Arguments: entity, time, or value mentions filling roles allowed by the event type. ACE tags these only when they occur within the event extent.
• Event attributes: polarity (explicit negation versus positive), tense, genericity, and modality (asserted versus other, under ACE's definitions). These describe the source's presentation, rather than independently verifying what happened.

For example, take the sentence "The insurgents attacked the convoy outside Mosul on Tuesday morning." Under the ACE Attack type, selected roles are:

Selected attributes for this event are Polarity=POSITIVE, Tense=PAST, and Genericity=SPECIFIC.

Notice how much of this connects to what you already know. In this ACE example, trigger and argument mentions have spans in the text. Trigger detection can be formulated as sequence labeling, as in NER, covered in Part VI. The role inventory is a semantic-role inventory, close in spirit to the PropBank argument labels we discussed in Semantic Roles and Argument Structure, with the important difference that event roles are event-type-specific (Attacker for Attack, Buyer for Transfer-Ownership) while PropBank reuses numbered core labels such as ARG0 and ARG1, whose meanings are specified by predicate sense; general modifiers use labels such as ARGM-LOC and ARGM-TMP.

The table also shows why event extraction is not just relation extraction with more slots. The Instrument role is unfilled in this example, and that is not an error. The text simply does not mention an instrument. A system that forces every role to be filled will hallucinate arguments, while a system that handles missing roles correctly must distinguish between "not mentioned" and "mentioned but not extracted." That distinction matters when evaluating event extraction.

Trigger ambiguity includes word-sense and schema decisions. "Acquired" can describe purchasing a company, gaining a skill, or contracting an illness. A detector using only that word has no way to distinguish those examples. Arguments and surrounding context provide evidence; a business schema must also define how ownership transfers differ from organization mergers.

The ambiguity is not limited to verbs. "Acquisition" can refer to a business deal or to the process of learning a skill. "Strike" can be a labor action, a military attack, or a baseball pitch. "Deal" can be a business agreement or a turn at dealing cards. The same trigger string can occur with different senses. Its arguments and surrounding context supply evidence for the intended event type. A joint model can use evidence from trigger and argument predictions together, as Li et al. and JRNN illustrate. The words identifying buyers, sellers, or military targets provide context; their presence alone does not establish the event's type or factuality.

Nominal triggers can be nouns inside longer phrases, such as "merger" in "the merger", "layoffs" in "the layoffs", and "announcement" in "the announcement". A nominal mention can leave participants unstated in its local sentence. "The acquisition expanded the cloud business" names the event without naming its buyer.

Nominal triggers also interact with coreference. "The acquisition" may refer to an event mentioned earlier. Resolving that reference requires context beyond the nominal mention, although an event explicitly described in a single sentence can still be extracted at sentence level.

Negative and hypothetical events introduce factuality distinctions. "Acme considered acquiring Bolt" and "Acme denied that it had acquired Bolt" both contain the trigger and both arguments, but neither asserts the event. Polarity, modality, and reporting context are therefore part of the extraction target when an application needs to distinguish assertions, plans, and denials.

The failure is easy to imagine. A financial monitoring system that extracts "Acme acquired Bolt" from a sentence saying "Acme denied that it had acquired Bolt" will report an acquisition that the sentence does not affirm. A representation may identify the participants correctly while mishandling the denial context. Whether that is scored as a relation, factuality, or attribute error depends on the annotation scheme. Systems that skip attribute classification are therefore vulnerable to a whole class of errors that are invisible in trigger-level F1.

Implicit arguments are not syntactically connected to their predicates; some can be recovered from discourse (Cheng and Erk, 2018). For example, "The acquisition closed two weeks later" has no explicit buyer, but the buyer is the buyer of the acquisition the sentence refers to. Recovering that omitted buyer requires connecting this mention to its antecedent event. That is a document-context requirement of this example, rather than of every event mention.

A later sentence can refer to an introduced event with a definite description that omits its participants. "The deal closed," "the merger was approved," and "the acquisition would expand the cloud business" can be used this way when they refer back to an earlier mention. An extractor needs that earlier context to recover the omitted participants in those cases.

A trigger-to-argument pipeline detects triggers first, then classifies arguments for each predicted trigger. Its components can be inspected or replaced separately while their interfaces remain compatible. In a trigger-gated implementation, a missed trigger prevents later role predictions: if trigger detection misses "acquired", no argument extraction happens, no matter how good the argument model is.

A one-way pipeline cannot feed later predicted role labels back into an earlier trigger decision. Its trigger encoder can still inspect the sentence, including words that will become argument candidates. A joint model or iterative pipeline can additionally exchange predicted structural information.

A joint model represents trigger and argument outputs together. The classic formulation (Li, Ji, and Huang, 2013) is a structured prediction over triggers and arguments with global features, and the neural versions followed quickly: JRNN (Nguyen, Cho, and Grishman, 2016) jointly predicted triggers and argument roles from bidirectional sequence representations, with dependency-relation features and memory of earlier decisions; dbRNN (Sha et al., 2018) added dependency-tree shortcuts between syntactically related word states in its bidirectional RNN. DyGIE++ (Wadden et al., 2019) generalized the approach to entities, relations, and events in a single span-graph model with contextualized span representations, which illustrates shared span representations across these extraction tasks.

Joint modeling provides a way to share evidence across trigger and argument predictions. Whether it improves a particular pipeline, and by how much, requires a comparison with the same data, encoder, and evaluation protocol.

Let $x$ be the input and $y^{tr}$ the complete trigger labeling. For this document, fix a set of proposed trigger–argument pairs and let $j$ index those pairs. The variable $y^{arg}_j$ gives the pair’s role label, with a no_role outcome for pairs with no role. Write $y^{arg}$ for the collection of pair labels. To enforce the event schema in this model, give no_role probability one in the conditional factor for each pair whose trigger is labeled absent. For a present trigger, restrict that factor to no_role and the roles allowed by its predicted type. The factored model defines this joint conditional distribution.

A CRF over the complete output structure uses this form.

where $y$ is a complete output structure, $f_k$ are features, $\theta_k$ their weights, and $Z(x)$ sums the unnormalized scores over allowed structures. This follows the CRF construction from Conditional Random Fields. Finite feature weights express preferences. A hard constraint requires excluding structures from the support or assigning them zero probability, for example disallowing roles outside the event type's declared inventory. A target unmentioned in the text does not by itself invalidate an Attack mention.

In the displayed factored model, the pair-level role labels are conditionally independent given $x$ and the complete trigger labeling $y^{tr}$. A CRF can retain this factorization or add features coupling those labels. Its inference cost depends on the allowed structures and factorization; adding a CRF does not by itself establish that inference is slower. A shared encoder can be used in either design. The factored distribution allows both trigger-first and joint decoding. Suppose competing complete trigger labelings A and B have probabilities 0.6 and 0.4, and their best conditional role assignments have probabilities 0.5 and 0.9. Selecting the trigger labeling first picks A. Maximizing the joint probability picks B because $0.4 \times 0.9 = 0.36$ exceeds $0.6 \times 0.5 = 0.30$.

In Nguyen, Cho, and Grishman (2016), Table 3, JRNN reports 69.3 trigger-classification F1 and 55.4 argument-role F1 on its ACE 2005 blind test, a difference of 13.9 points. That comparison concerns one system and protocol. Argument recovery introduces several distinct decisions:

• Multiple mentions, one event. The argument "Acme Corp" may appear as "Acme", "the company", or "it" within the event's scope. Coreference connects mentions that refer to the same entity (see Coreference Resolution).
• Span reuse and event-valued arguments. One span can fill roles for multiple triggers, with a separate variable for each trigger–argument pair. If the schema permits multiple roles for the same pair, extend its variable to range over allowed role sets. Separately, a schema can permit an event as a role filler. For example, the illustrative Announcement schema can represent "Acme announced the acquisition of Bolt" as an Announcement whose Content is an Acquisition event.
• Long-distance roles. In "the company, which was founded in 2011 by Sam Okafor and which had grown to 40 employees, was acquired by a competitor", the company mention and the acquisition trigger "acquired" are separated by two relative clauses.
• Unfilled roles and evaluation. A role can be absent from the text. In set-based argument extraction F1, predicting no argument contributes neither a true positive nor a false positive when the gold set is also empty. It does not earn an extra true positive. A fixed-slot application may additionally report correct empty slots, but that is a separate evaluation choice.

The first bullet concerns reference resolution. In "Acme acquired Bolt. The company later sold it," the second sentence refers to participants by a description and a pronoun. Recovering their canonical identities requires deciding what "the company" and "it" refer to. Merely concatenating the sentences does not supply those decisions.

In the third bullet, two relative clauses intervene in the company subject phrase, while the main clause still links "company" to "was acquired". A window narrower than this surface distance excludes the company mention. A system that uses dependency paths can follow the syntactic link from the argument to the trigger even when the surface distance is large.

The fourth bullet separates extraction F1 from slot-level accuracy. For extraction F1 scored against a gold set, a gold argument with no match is a false negative and a predicted argument with no match is a false positive. Empty gold and predicted sets add no extracted items to either count. If an application reports slot-level accuracy, state the candidate-slot universe and report recall on filled gold slots separately. A large number of correctly empty slots can make slot accuracy high even when the system misses filled arguments.

Two other formulations express event outputs as answers or generated structures.

Event extraction as QA (Du and Cardie, 2020) turns each role into a natural-language question, in the same spirit as QA-based relation extraction. For an Attack event triggered by "attacked", the question "Who was attacked?" is posed against the sentence, and a reading comprehension model extracts the answer span. The answer can be a span elsewhere in the input, and the model includes a no-answer decision for absent arguments.

The QA formulation also gives a natural way to handle absent roles. If the question "Who was attacked?" has no answer in the text, the reading comprehension model can return "no answer," which is exactly the correct behavior for an unfilled role. In Du and Cardie's model, the first token ([CLS]) represents the no-answer case; span candidates are compared with that score. This is also a learned null decision, not a guarantee that empty roles are recognized correctly.

Event extraction as generation (Lu et al., 2021, Text2Event) casts the output as a linearized structure. Here is one way to illustrate such a record. Time is a simplified field in this illustration.

Text2Event trains a sequence-to-sequence model to generate a linearized event structure. The decoder predicts a serialized structure, including the roles present, rather than separately classifying every trigger-argument candidate. Decoding still searches over output sequences. Without structural controls it can emit invalid structures or unsupported arguments; Text2Event uses constrained decoding to restrict the allowed productions. At a role position such as Target in this example, the Text2Event trie limits the decoder to role names allowed by the schema.

Generation can encode event attributes in the output sequence. An output schema extended to include these attributes can serialize Polarity=NEG or Tense=PAST alongside the roles without requiring a separate attribute-output classifier. That is convenient, but it also means that an error in one part of the sequence can cascade to later parts. Constrained decoding helps, but it does not eliminate the need for validation.

An event schema defines the types and roles the system will predict. Several established resources use different inventories:

• FrameNet (Baker, Fillmore, and Lowe, 1998) organizes meaning into frames, each with frame elements. The Commerce_buy frame has Buyer, Goods, Seller, Money. FrameNet describes lexical-semantic frames; the released FrameNet 1.7 index contains 1,221 frames.
• PropBank (Palmer, Gildea, and Kingsbury, 2005) labels predicate arguments with numbered roles (ARG0, ARG1, ...) tied to each verb's frame file, plus adjunct labels (ARGM-TMP, ARGM-LOC). The original project annotates predicate roles in the Penn Treebank and supplies verb-specific framesets.
• ACE 2005 events use eight top-level types and 33 subtypes (Conflict:Attack, Life:Die, Transaction:Transfer-Ownership, Personnel:Start-Position, ...). Text2Event (Lu et al., 2021), for example, calls these 33 labels "event types". The ACE annotation guidelines define their roles and attributes; the corpus includes news and other genres.
• Domain schemas can define application-specific records. The BioNLP 2009 shared task, for example, defines Gene expression, Binding, and Regulation events. A separate illustrative financial schema might define Earnings, Merger, and Guidance; those names are design choices, not labels inherited from ACE.

Evaluate agreement for the actual schema and annotation instructions. A narrower label set changes the task; it does not guarantee higher agreement or better extraction. The TACRED re-annotation results above show why labels themselves need review, even when the output inventory is fixed.

The schema also determines whether an argument can itself be an event. BioNLP 2009 permits an event to fill the Theme or Cause of a Regulation event. Our custom Announcement schema can likewise use an acquisition as Content. This is different from FrameNet's frame-inheritance relations; inheritance alone does not establish that a frame element accepts a nested event. State the allowed argument types when defining a nested representation.

Political event coding reduces news reports to structured records such as (Actor, EventType, Target) with temporal and geographic metadata for statistical analysis. CAMEO's authors describe developing their scheme in research on third-party mediation.

• CAMEO (Conflict and Mediation Event Observations) provides a hierarchical taxonomy of about 300 event types for coding political interactions. CAMEO codes the source and target with the same actor coding scheme.
• TABARI uses verb-phrase patterns and a sparse parse to code political interactions. Its manual describes this pattern-based design; the architecture should not be generalized to every later event coder.
• GDELT (Leetaru and Schrodt, 2013) applies this at global scale, coding news into event records with tone and geography, whose 2013 codebook describes daily update files beginning on 1 April 2013.
• ICEWS provides historical machine-coded political event data; its dataset metadata states that the series was discontinued on 11 April 2023. POLECAT, described by Halterman and colleagues (2023), uses the PLOVER ontology and the Next Generation Event Coder. The authors describe a combination of NLP tools, transformer networks, expert annotations, and actor information from Wikipedia.

Political event coding includes both pattern-based coders and systems such as NGEC that use transformers. For longitudinal analysis, retain the coder and schema versions as well as the event records. A change in coding behavior can alter the measured trend even when the underlying reports stay fixed.

Evaluate that stability explicitly. Recode the same reference collection with each version. Compare records and per-class scores using the same scoring protocol and shared class definitions or a documented mapping. Report unmatched classes separately. A single aggregate F1 score on one version does not measure change between versions. Once labels are aligned, agreement F1 between the two runs can measure record overlap. Neither a rule-based nor a neural architecture guarantees stable outputs when its rules, weights, resources, or inputs change.

Two mentions in different places may refer to the same event. "The deal closed two weeks later" and "the acquisition completed on 17 March" may describe one closing. Event coreference clusters mentions judged to refer to the same occurrence. The ECB+ work (Cybulska and Vossen, 2014) distinguishes event mentions from event denotations and considers action, time, location, and participants.

Events can have names, but they can also be referred to through verbs, nominalizations, and paraphrases. In an acquisition example, "the acquisition", "the deal", and "the transaction" can refer to the same occurrence. Group "the merger" with them only if the source and schema establish that identity. Clustering asks whether the mentions refer to the same occurrence. Participants, time, location, and other evidence in context help answer that question.

Events also stand in relations to each other beyond identity:

• Temporal: before, after, simultaneous, during. Covered in the next section.
• Causal: cause, enable, prevent. BECauSE annotates expressed causal language using its own type and degree distinctions; event-link datasets such as Ning et al.'s (2018) joint temporal–causal dataset use their own causal-link inventory.
• Subevent: a component occurrence is part of a larger event, if the chosen event model defines that part-whole relation. An acquisition appearing as the Content of an announcement is instead an event-valued argument; that sentence alone does not establish a subevent relation. A schema can treat the closing as one stage of a larger transaction process. That process is distinct from the earlier reported acquisition in the worked example.

These links organize event records by showing how the occurrences are connected. Discourse Relations and Coherence is a planned later chapter.

Temporal records can retain calendar values and relative orderings. Normalizing "two weeks later" to a calendar date requires choosing its reference point. This section covers the representation standard, the anchoring problem, and the pairwise ordering task.

Temporal extraction separates surface recognition from semantic interpretation. The string "3 March 2019" can be recognized as a date. Assigning its value, relating it to other dates in the document, and deciding whether an event is represented as a point or an interval are additional decisions. The representation standard gives you a vocabulary for those decisions, and the anchoring problem tells you which context you need to make them.

TimeML (Pustejovsky et al., 2003) defines an annotation language for temporal information in text. ISO 24617-1:2012 specifies ISO-TimeML as a standard for temporal semantic annotation. The following tags are from TimeML 1.2.1:

• TIMEX3: temporal expressions, tagged with a type and either a normalized value or a valueFromFunction reference.
• EVENT: event expressions with an identifier and event class.
• MAKEINSTANCE: event instances linked to EVENT, carrying tense, aspect, and applicable polarity and modality attributes in TimeML 1.2.1.
• SIGNAL: expressions, typically function words, indicating temporal relations or quantification (for example, "before", "after", "during", and "twice").
• TLINK: temporal links between event instances or TIMEX3s. A document creation time participates through its TIMEX3.

TIMEX3 has four types:

TimeML values use ISO 8601 conventions together with task-specific extensions for underspecified dates. Keep those extensions separate from a claim of strict ISO conformance:

• Dates include full values such as 2019-03-03 and reduced-precision values such as 2019-03 (a month) or 2019 (a year). The related TIDES TIMEX2 guide uses 1999-06-XX for an unspecified day in June 1999.
• Times: bare clock-time forms such as T18:00 (6:00 p.m.).
• Durations: P followed by components: P2W (two weeks), P3D (three days), P1Y6M (one year six months).
• A start/end pair can also be written in ISO-style interval notation, such as 2019-03-03/2019-03-17. Do not substitute that string for a TimeML 1.2.1 value without checking its datatype grammar. TimeML can anchor a duration using beginPoint and endPoint references to TIMEX3 identifiers.

The mod attribute captures temporal modifiers: "more than 3 years" gets mod="MORE_THAN", "approximately two weeks" gets mod="APPROX". If only the normalized duration value is kept, that value cannot distinguish an exact duration from the same number with a modifier.

The distinction matters because temporal expressions are not only about dates. "More than three years" and "three years" denote different durations, and the difference is relevant to questions about how long something lasted. A system that normalizes both to P3Y loses the distinction. Keeping the mod attribute preserves it, at the cost of a slightly more complex value representation.

Replacing an unknown date within 2011 with 2011-01-01 invents precision. It can create an unsupported ordering against another event in that year. Preserve both the normalized year and the fact that it only bounds the event's date.

Normalizing "two weeks later" requires deciding what it is two weeks later than, in addition to identifying the expression.

A normalization interface that supports both document-relative and discourse-relative expressions can take the form

where $e$ is the expression, $t_{\text{DCT}}$ is the document creation time, such as the time an email was written, and $t_{\text{ref}}$ is a reference time supplied by context.

Three cases illustrate different reference requirements:

• Fully specified absolute dates need no anchor. "3 March 2019" is 2019-03-03 regardless of when the document was written.
• Deictic expressions need a temporal perspective. In the isolated-expression example below, "yesterday" resolves against the document creation time $t_{\text{DCT}}$. Quoted speech may instead require the quoted utterance time. Substituting the system clock for the intended reference can change the result when those dates differ; absolute dates do not have that dependency.
• Relative expressions anchor to $t_{\text{ref}}$, an event or time supplied by context. "Three days after the merger" names its reference locally; "two weeks later" can require looking back to an earlier sentence. Resolving the reference can involve event extraction or coreference when the antecedent is an event mention.

A system limited to absolute and document-date-relative expressions cannot generally resolve discourse-relative expressions using their intended references. Its coverage depends on the corpus and annotation conventions; no fixed percentage follows from this taxonomy.

The failure is easier to audit when the two candidate anchors are placed on the same axis. The following computation compares two anchors for the closing. In the worked reading, the reference is the announcement on 3 March. Using the publication date instead gives a different result.

The anchor error is subtle because the system can still produce a valid date. For the announcement-anchored reading, substituting the document date produces a well-formed ISO date that disagrees with the intended value. A normalizer that checks only date validity misses this error. Compare normalized values against gold values to test the anchor decision.

For English tense analysis, Reichenbach (1947) distinguished three temporal points:

• Speech time $S$: when the utterance (or document) is produced.
• Event time $E$: the time assigned to the event in the tense interpretation.
• Reference time $R$: the temporal vantage point from which the event is located.

Tense is then a constraint on the ordering of these three. In Reichenbach’s schematic analysis, simple past has $E = R < S$. Present perfect asserts $E < R = S$: the event precedes a reference time that coincides with speech. Past perfect asserts $E < R < S$.

The worked example says, "Acme Corp announced on 3 March 2019 that it had acquired Bolt Systems for $120 million." Within Acme’s report, the past perfect places the acquisition ($E$) before the announcement ($R$). The worked chronology places that announcement before the article’s assumed narration time ($S$). The report leaves the acquisition date unspecified. It could refer to an acquisition earlier that day or on an earlier date.

Reichenbach's framework is useful because it separates two questions that are easy to conflate. The first is what absolute time or interval, if any, the text assigns to the described event. The second is how the text locates the event relative to other events and to the moment of speech. An application that needs both absolute values and discourse order has both prediction targets. If it only answers the second, it can produce a relative ordering without absolute dates. If it only answers the first, it can miss the narrative structure that the tense encodes.

Several temporal normalization systems use grammars and rules:

• SUTime (Chang and Manning, 2012) is a compositional rule-based recognizer built on token regex patterns. Its paper describes temporal objects for times, durations, intervals, and sets. Rules combine recognized components, and a resolution stage uses a reference date for relative expressions.

• HeidelTime (Strötgen and Gertz, 2013) is a rule-based, multilingual tagger with language resources and domain-sensitive normalization strategies. Their 2012 domain comparison evaluates different normalization strategies for news, narratives, colloquial text, and scientific text; the domain can affect which reference-time strategy is appropriate.

• Bethard (2013) pushed the grammar idea further with a synchronous context-free grammar mapping expression words to formal temporal operators that are evaluated to obtain values. The shared derivation ties parsing the supplied expression to its normalization, but it does not guarantee a semantically correct parse or value.

TempEval-3 provides a bounded comparison: on its newswire test data, statistical systems performed best under strict span matching and rule-engineered systems under relaxed matching. All participating normalization components used rule-engineered approaches. Those results do not establish a ranking for every domain or later model.

The lesson is not that neural models are unnecessary. It is that calendar arithmetic and explicit anchor rules can be implemented and checked separately from span recognition. A hybrid system that uses a neural model for recognition and a grammar for resolution can combine these functions. Given a reference time selected from context, the grammar can apply calendar arithmetic and explicit anchor rules. The neural model can recognize varied surface forms.

Temporal relation extraction assigns a relation to a pair of event instances or time expressions. A relative order can be recoverable even when their calendar dates are unknown.

A TLINK records a temporal relation between event instances or times. TimeML 1.2.1 declares the following relType values.

BEFORE, AFTER, INCLUDES, IS_INCLUDED, DURING, SIMULTANEOUS, IBEFORE, IAFTER, BEGINS, BEGUN_BY, ENDS, ENDED_BY, and IDENTITY.

Use the label definitions of the chosen annotation scheme. MATRES's VAGUE label, for example, is absent from the TimeML 1.2.1 list above. Its BEFORE label shares the spelling of a TimeML label.

The following resources use different temporal targets and annotation protocols:

• TimeBank 1.2 and the temporally annotated AQUAINT material used in TempEval-3 are resources for temporal extraction. Do not confuse the original TimeBank with TimeBank-Dense: Chambers and colleagues (2014) describe the original corpus as labeling only a small subset of possible temporal pairs.
• TempEval-1/2/3 (Verhagen et al., 2007, Verhagen et al., 2010, and UzZaman et al., 2013) defined shared tasks. In TempEval-3 Task ABC, systems start from raw text, identify events and time expressions, choose linked pairs, and classify the temporal links.
• TAC KBP temporal slot filling sought temporal bounds on when an entity's slot-filler relation held true, such as a person holding a job title. Its evidence could be distributed across documents.
• MATRES (Ning, Wu, and Roth, 2018) organizes events on temporal axes and compares their start points, rather than attempting full interval-endpoint judgments. The released main-axis data uses before, after, equal, and vague labels. "Multi-axis" does not mean a separate axis for each relation label. The paper reports improved agreement in a two-expert pilot on the main axis.
• Ning and colleagues (2018) evaluated their Temporal and Causal Reasoning (TCR) framework on a newswire corpus annotated with both temporal and causal links. TDDiscourse (Naik et al., 2019) augments TimeBank-Dense with discourse-level temporal annotations, including pairs more than one sentence apart.

The following examples cover feature-based and neural classifiers. Early feature-based classifiers included systems in TempEval 2007. Bethard’s ClearTK-TimeML used event tense, aspect, and constituency paths for same-sentence event pairs. Other classifiers used temporal signal features or VerbNet classes. CAEVO (Chambers et al., 2014) ordered twelve sieves by precision, passed earlier graph constraints to later sieves, and added transitive links after each sieve. Neural examples include bidirectional LSTMs over dependency paths (Cheng and Miyao, 2017) and BERT-based clinical temporal classifiers (Lin et al., 2019) that mark the candidate spans in the input.

Two problems matter when evaluating temporal systems.

Label imbalance and a document-order baseline. Yao and Huang (2018) use the chronological tendency of narrative stories after identifying narrative paragraphs. This does not establish the label distribution of arbitrary documents. If a declared single-label evaluation set has 90 percent BEFORE labels, always predicting BEFORE gives 90 percent accuracy and zero recall for every other class with gold examples. With one prediction per item and all classes included, micro-F1 is also 0.90. Measure the actual baseline and report per-class scores.

Consistency. An independent pairwise baseline predicts each pair separately, so its outputs can be globally inconsistent: $A$ before $B$, $B$ before $C$, and $C$ before $A$. A joint inference constraint or a repair pass can reject such a cycle. A design that predicts one global order can also induce pairwise labels from that order.

A single incorrect label can create such a cycle. For a true order $A<B<C$, predicting $A$ before $B$, $B$ before $C$, and $C$ before $A$ reverses just one pair. The cycle is a clear signal that something is wrong, and it can be used to trigger a repair pass or to flag the document for human review. Independent predictions can form an impossible cycle; they need not do so on every input.

The formal foundation for interval orderings is Allen's (1983) algebra. For proper intervals with $a^- < a^+$ and $b^- < b^+$, thirteen relations exhaust the possible endpoint orderings: six relations, their six inverses, and equality. Write $A = [a^-, a^+]$ and $B = [b^-, b^+]$, where minus and plus denote start and end coordinates:

Six representatives and equality look like this:

• before: A entirely to the left of B.
• meets: A ends exactly where B begins.
• overlaps: A starts first, they share some time, B extends beyond.
• starts: same start, A ends first.
• during: A strictly inside B.
• finishes: same end, A starts later.
• equals: identical extents.

The algebra is closed under composition: from the relation between $A$ and $B$ and between $B$ and $C$, you can derive a set of possible relations between $A$ and $C$. Composition tables let you propagate constraints and detect contradictions.

Two important caveats. First, the point algebra (Vilain and Kautz, 1986) uses instants as its variables, with base relations <, =, > and their disjunctions. Separate start and end variables can encode each individual Allen relation without collapsing the intervals; not every disjunction of interval relations can be expressed as a conjunction of point-algebra constraints on those endpoints. MATRES's start-point comparisons illustrate a point-based choice: start order alone cannot distinguish intervals that meet from ones that overlap. Second, general satisfiability of interval algebra constraint networks is NP-complete (Vilain, Kautz, and van Beek, 1990). This is a worst-case result, not a claim that every network is difficult. Exact checking remains an option for small instances or tractable restricted classes. Path consistency is sound but incomplete for the unrestricted algebra: it detects some contradictions and misses others.

Granularity matters when interpreting the algebra. The calendar-year window contains 3 March, but an event localized only to that window is not thereby known to contain an event on 3 March. Allen relations describe actual interval extents. Uncertain localization requires bounds or sets of possible relations; it cannot be substituted for event duration.

Yao and Huang's narrative-paragraph method uses a tendency for narrated events to follow chronological order. The sequence "we went to the restaurant, we ate, we left" illustrates an order that a reader can infer without an explicit "before" between each clause. Reordering those clauses can change that reading.

Several consequences follow.

• First, document order supplies a candidate weak labeler for the narrative setting. Its accuracy must be measured on the selected texts; the narrative tendency does not make it correct for every event pair.
• Second, not every temporal relation has an explicit signal. An ordering can be expressed without the words "before" or "after". The past-perfect example above uses tense and reference time to constrain the acquisition relative to the announcement.
• Third, flashbacks can reverse mention order, while reported speech can shift the temporal perspective. "Acme announced in 2019 that it had been founded in 2011" puts the later announcement trigger before the earlier founding trigger. Document order alone cannot recover that flashback. Explicit dates, tense, reference structure, or other contextual evidence can supply a different ordering; no particular parser feature is mandatory for every system.

Document order supplies a candidate baseline in the narrative setting, but the flashback above defeats an unqualified BEFORE rule. An unqualified document-order BEFORE rule fails on a pair whose mention order reverses its temporal order, as the flashback example demonstrates. The document-order preference should be defeasible. An explicit feature is one way to let other evidence override it.

Beyond pairwise classification, temporal reasoning asks what follows from the set of extracted constraints: can these events be arranged into a consistent timeline? Which events precede which? Is there a unique total order?

The machinery is constraint propagation over the interval or point algebra. For a pair $A,C$ and an intermediate event $B$, compose the allowed relations from $A$ to $B$ and from $B$ to $C$. Intersect the result with the current set for $A,C$. Cycle through the relevant triples until no set changes. If a sound propagation step empties a pair's allowed relation set, the supplied constraints are jointly contradictory under the representation's assumptions. Recheck the source assertions, extracted relations, normalization, and interval assumptions; the contradiction need not originate in the extractor.

Constraint checks can flag records for review. For an acyclic graph of strict BEFORE constraints, a topological sort returns one linear extension. It does not establish a unique true order, resolve unknown pairs, or enforce interval relations such as overlaps. Keep unresolved relations explicit rather than treating an arbitrary tie-break as evidence.

Constraint propagation can also supplement evaluation. A jointly unsatisfiable set of predicted relations cannot serve as one consistent timeline under the declared assumptions, even when each pair looks plausible. Pairwise scores alone can miss this incompatibility. Recheck source contradictions and factuality as well as extraction: faithfully recording conflicting reports is different from asserting that both reports are true.

Evaluating temporal extraction involves three subtasks with different metrics:

• Timex recognition and normalization is evaluated with span-level precision, recall, and F1 for recognition, and value accuracy (exact match of the normalized temporal value) for normalization, reported separately. A correct span can have an incorrect value, so report both. State the span alignment and the denominator used for value accuracy.

• Temporal relation extraction is evaluated with precision, recall, and F1 over labeled pairs, sometimes with a "reduced" label set that merges fine distinctions. Report class-level behavior alongside micro-F1. In the majority-only example above, one prediction per item and micro-averaging over all classes give 0.90 micro-F1. Every other class with gold examples has zero recall. Macro-F1 and a confusion matrix expose aspects of that result that the micro aggregate alone does not identify.

• Timeline ordering can be assessed against a declared reference order, for example by the fraction of comparable event pairs ordered correctly. Kendall's tau is another option for total orders, with a stated tie convention when needed. These ordering checks do not replace evaluating event identity, extraction, and time normalization.

State which pairs are annotated and evaluated. The original TimeBank labels a subset of possible pairs; TimeBank-Dense and TDDiscourse add annotations under different selection policies. A sampled evaluation set can have a different class balance from all candidate pairs at inference.

The sampling issue is worth emphasizing because it can make a system look better or worse than it is. If the selected pairs have a higher error rate for the evaluated system than the full candidate set, measured accuracy decreases; a lower selected error rate has the opposite effect. Report the selection policy and its label distribution instead of assuming that a named pair category is harder for every model. Papers that report only aggregate F1 without describing the sampling procedure are hard to compare.

Some of the examples require information beyond one sentence. The following effects help determine which document context an application needs.

Document-level effects are not a single problem. Several examples below require information beyond the sentence containing the target mention. For these cases, the missing information can be in a previous sentence, a previous paragraph, or document metadata. A system recovering those outputs must make that information available to its prediction stage.

A sentence-local extractor can miss the identity of an argument expressed through a pronoun or description. In the examples here, "it" and "the company" can refer to entities, while "the deal" can refer to an event. Resolving a reference means identifying what it refers to. In these examples, an earlier mention supplies the antecedent. A description can serve as the antecedent. In “The buyer signed. It later paid,” “the buyer” supplies the antecedent of “it.”

When an argument is expressed through a pronoun or another referring expression, extraction may need coreference to identify its entity. An explicit subject and object can instead support a sentence-level relation without a separate coreference step. Relation information can also provide evidence for resolving mentions. Iterative and joint models can represent this interaction; Coreference Resolution develops the task.

The dependency can be handled in stages. A first pass can resolve easy cases, such as pronouns with a single compatible antecedent in the previous sentence. A second pass can use the extracted relations to disambiguate harder cases, and then a third pass can re-run extraction with the improved coreference. This is an iterative design option. Each pass needs evaluation: an incorrect antecedent can propagate into the next extraction pass.

Two textually different mentions may refer to the same entity: "Acme", "Acme Corp", "ACM", "the Ohio-based manufacturer". Coreference or entity resolution groups references to the same entity; entity linking maps a mention to a knowledge-base identifier. Linking to a shared identifier supports cross-document aggregation, while unresolved aliases may remain separate.

Changing the prediction unit also changes what gets counted. At the mention level, "Acme Corp acquired Bolt Systems" and "Acme acquired Bolt" are separate predictions. Once the mentions have been resolved to the same entities, the two passages can provide evidence for one entity-level prediction. The next chapter covers this in detail.

Entity linking is especially important for aggregation. If the system extracts (Acme, acquire, Bolt) from one document and (Acme Corp, acquire, Bolt Systems) from another, those are two different triples unless the mentions are linked. If aggregation treats these unresolved aliases as distinct identifiers, it can create duplicate nodes or edges and leave a query over one alias incomplete. Entity linking is one way to reconcile the references; an input that already supplies canonical identifiers does not require a separate linking step.

Long documents introduce two practical constraints.

• First, context length. In multi-document QA and key-value retrieval experiments, Liu et al. (2024; preprint 2023) observed sensitivity to where relevant information appeared in long inputs. That result motivates checking evidence position in an extraction evaluation; it does not establish the same degradation for every extraction model. Chunking with overlap and subsequent aggregation is one design option.
• Second, the combinatorial blow-up. If a 10,000-token document contains 100 entity mentions, enumerating ordered pairs of distinct mentions gives 9,900 candidates. Possible filters include entity-type compatibility, sentence distance, and selected keywords. They reduce the candidates scored only by excluding pairs; compare retained-relation recall and runtime against exhaustive or batched scoring.

Chunking with overlap introduces its own problems. An entity mention that falls on a chunk boundary may be split, and a relation whose arguments are in different chunks may be missed. Overlap can put a boundary-spanning mention or pair into a chunk, but it can also produce duplicate extractions that need a merge decision. Design the merge rule together with the chunking strategy, and evaluate the combined output.

An extracted record can retain provenance, including its source sentence, character offsets, and trigger. Those references let an auditor inspect the supporting text and help distinguish an extraction error from a later normalization or aggregation error.

The JSON below shows the spans and offsets for the acquisition candidate attributed to Acme. The worked record that follows also keeps the status and attribution fields, which this projection leaves out.

{
  "head": "Acme Corp",
  "head_span": [0, 9],
  "relation": "acquire",
  "tail": "Bolt Systems",
  "tail_span": [57, 69],
  "evidence": "Acme Corp announced on 3 March 2019 that it had acquired Bolt Systems",
  "evidence_span": [0, 69]
}

Each quoted source span must match the document at its stated character offsets. A mismatch can indicate an incorrect offset, altered surface text, or a fabricated span; it does not by itself identify the cause. A substring check tests whether selected field strings occur somewhere in the document. A strict offset check additionally tests the exact location. The implementation below demonstrates the substring check, which does not establish that the source entails the extracted relation.

Evaluate the tuple and its cited evidence separately. An otherwise correct candidate for Acme’s reported acquisition could cite the paragraph’s chief-executive sentence, which does not support that report. Retaining the evidence span makes this mismatch available for review.

Let's work through a small example with rule extraction, reference heuristics, normalization, and grounding checks. Here is the document:

Acme Corp announced on 3 March 2019 that it had acquired Bolt Systems for $120 million. The deal closed two weeks later. Acme's chief executive, Jane Ruiz, said the acquisition would expand the company's cloud business. Bolt Systems, founded in 2011 by Sam Okafor, employed about 40 people at the time.

For this example, assume the article was created and published on 2019-03-25. For this worked interpretation, resolve "two weeks later" relative to the explicitly dated announcement on 3 March, giving a closing on 17 March. In Acme’s account, the past perfect places the acquisition before the announcement without dating it exactly. We keep that reported acquisition separate from the closing described in the next sentence. The passage gives no independent date for a legal transfer of ownership.

The example contains relations, event-valued arguments, a relative time expression, a year-granularity time, a pronoun, an alias, and a document creation time. Write the expected records by hand before comparing them with the rules below.

Step 1: named mentions. Acme Corp (ORG), 3 March 2019 (DATE), Bolt Systems (ORG), $120 million (MONEY), two weeks later (DATE), Acme (ORG), Jane Ruiz (PERSON), a second Bolt Systems mention (ORG), 2011 (DATE), Sam Okafor (PERSON), and about 40 (CARDINAL). The referring expressions "it", "The deal", "the acquisition", and "the company" are absent from this named-entity list. They still participate in reference decisions.

Step 2: candidate relations.

• (Acme Corp, acquire, Bolt Systems) is the acquisition described in the report "Acme Corp announced on 3 March 2019 that it had acquired Bolt Systems". The reporting clause names Acme Corp and supplies the antecedent of "it". Carry that evidence and the attribution to Acme with the candidate if you use it in an acquisition record. In this example the acquisition has status=reported_complete and claim_source=Acme Corp. If the sentence instead said "Acme Corp announced it intended to acquire Bolt Systems", the same buyer and target would have status=reported_intent and claim_source=Acme Corp. The status describes what Acme reported.
• (Jane Ruiz, chief_executive_of, Acme), from the appositive-possessive construction "Acme's chief executive, Jane Ruiz". The organization tail is "Acme", not "Acme Corp"; linking the alias lets these references share an identifier.
• (Sam Okafor, founder_of, Bolt Systems), from the reduced passive relative clause "founded in 2011 by Sam Okafor".

Step 3: selected events. We retain four records below. E1 is the dated announcement, rather than the earlier acquisition. The acquisition remains the content of that announcement and the deal referred to by E2; its exact date is unknown. This selected table is not an exhaustive event-mention annotation.

Four points worth noticing:

1. E2's argument is an event. "The deal" refers to the acquisition, not to E1's announcement. A record can refer to another event even when that event is not expanded into a separate row in this selected table.
2. E3 has year-level localization. Its date lies within 2011; the text does not say that founding lasted a year.
3. E4 has no exact date. Under the worked simple-past reading, Jane Ruiz spoke before the narration. This ordering leaves her exact day unknown.
4. The past perfect matters. "Had acquired" locates the acquisition before the dated announcement. Assigning 3 March as its exact date asserts precision the sentence does not supply. Collapsing both events to the same date-point without preserving their before relation would lose that precedence distinction.

Step 4: temporal relations.

• E3 before E1 (2011 well before 2019-03-03)
• E1 before E2 (announcement precedes closing)
• E3 before E2 (the founding year precedes the closing date)
• The date-only comparisons leave all three pairs involving E4 undetermined. They use the stored date fields and omit the separate tense-derived ordering relative to narration.

Deriving E2's time is where the anchoring machinery earns its keep. For the worked reading, "two weeks later" is anchored to the announcement:

If we had anchored "two weeks later" to the document creation time (2019-03-25), we would get 2019-04-08, which is wrong by 22 days and would place the closing after the article was published, an obvious contradiction that a consistency check should flag.

The contradiction depends on factuality: this paragraph describes the closing as completed before publication. A future date would be legitimate for an announced plan or prediction. Check event status and evidence before treating a date after publication as an error.

We'll implement dependency rules, reference heuristics, temporal normalization, and interval reasoning, then apply grounding checks to a hand-written response to an extraction prompt.

The dependency rules expose the selected syntactic conditions. The pronoun heuristic supplies a buyer for this document and has explicit failure cases. The temporal calculation compares two anchors. The final checks inspect selected strings and a normalized date in the supplied response.

Run these cells in the book environment with book_plot_style on the Python path and spaCy’s English en_core_web_sm model installed. The worked code also uses Matplotlib, pandas, dateparser, and python-dateutil.

The displayed counts come from the installed spaCy model. Its named mentions include repeated organizations and temporal expressions; these are mentions, rather than a deduplicated entity inventory.

The first rule handles selected subject-verb-object constructions. We build helpers with explicit fallbacks, then iterate over tokens whose lemmas appear in a trigger lexicon. The fallbacks cover the branches shown below, rather than arbitrary parsing variation.

The dependency rules emitted two candidates. The acquisition candidate still has “it” as its head. That is the grammatical subject of “acquired” in the text. Resolving the pronoun to the buyer takes a separate coreference step.

The worked example uses a minimal heuristic to illustrate one antecedent search rule. The toy resolver selects the recognized ORG, PERSON, or GPE mention with the latest end offset at or before the cutoff. Here the caller uses the trigger offset, which can fall after the pronoun. An eligible mention between the pronoun and trigger can then be selected as the antecedent. It does not test compatibility separately for each pronoun.

The display shows candidate tuples. Its acquisition tuple carries neither the reporting clause nor a status field. The resolver chose "Acme Corp", the only recognized mention of an allowed type before this trigger's character cutoff. That answer is correct for this document, but the heuristic cannot distinguish several plausible antecedents or enforce agreement. Broader systems may use learned coreference models or domain rules. Coreference Resolution is a planned later chapter.

Notice also the asymmetry in what the resolver sees: "Bolt Systems" also appears in the text, but after the acquisition trigger, so it falls outside this search cutoff. Direction of resolution matters.

A surface regex can recognize the selected "X's ROLE, Y" construction directly, without a dependency parse. The expression below recognizes two role strings and captures a two-word person name. The capture can be only the prefix of a longer name. Whether it is more accurate or robust than a dependency rule requires a matched evaluation.

The demonstration now emits three relation candidates. Note the alias problem in the last one: the organization tail is "Acme", not "Acme Corp". An aggregation that treats the two surface strings as distinct identifiers will keep them separate unless it reconciles the alias. Entity linking or another canonicalization step can perform that reconciliation.

To use a price numerically, convert the matched amount to a number and apply its scale. This toy regex searches for a dollar sign, at most one whitespace character, digits or commas, and an optional decimal point followed by digits. It allows whitespace before the optional listed scale. It can match a prefix of a longer string. For example, $5millionth uses the million scale and $5 trillion yields 5 with no scale multiplier. The search can skip a leading minus, reading -$5 million as positive 5 million. A comma-only amount such as $, matches. Removing the comma leaves an empty string, so float raises ValueError.

The value is 120000000.0 dollars. Numeric values let you sort, compare, and aggregate prices across documents when the amounts share a currency. Convert other currencies to the common unit first.

Now the temporal core. We set the document creation time, then normalize an absolute expression, then a deictic one, then derive the relative one by arithmetic.

Archive the document creation time when deictic expressions depend on it. Using today's date for "yesterday" in an archived article can produce the wrong result. An absolute date such as "3 March 2019" does not have that dependency.

We assemble the selected event records by hand for the reasoning example. The closing record reuses the date computed above. The time_kind field distinguishes an exact day, an uncertainty window, and an unknown date. The two endpoints of E3 are bounds on localization, not its duration. At this demonstration's day resolution, an exact day uses equal coordinates. These day points are not proper intervals. Disjoint date bounds can establish before or after; an interval label returned for a degenerate input uses the helper's explicit branch-priority convention described below.

The selected records preserve missing information. E2 refers to the acquisition event, E3 stores a year-sized uncertainty window, and E4 has no assigned date. Neither an unknown date nor an uncertainty window should be mistaken for a measured event extent.

The allen helper distinguishes the thirteen endpoint relations for proper intervals. The graph builder below first checks for unknown times and disjoint bounds, then withholds an extent relation for overlapping uncertainty windows. Only the remaining pairs are passed to allen.

For proper intervals, exactly one of the thirteen endpoint relations holds. The helper also accepts equal start/end coordinates, as used for exact days here, and resolves overlapping endpoint predicates by the order of its if branches. For example, the point (2, 2) against (0, 2) returns met-by before reaching the finishes test, although both displayed endpoint predicates hold. That is an implementation convention for points, not a unique Allen relation. Use a point/interval formalism with explicit semantics if those mixed cases matter to the application.