Information Extraction: NER, Relations, Structured Output

Text is the most natural medium for human knowledge, yet it was never designed for machines to query. When you read "Apple acquired Beats Electronics for $3 billion in May 2014," you instantly parse the acquirer, the target, the price, and the date. A database query cannot. Information extraction (IE) is the discipline that connects this gap: it turns free-form natural language into structured facts that software can store, search, aggregate, and reason over.

The need for IE arises wherever valuable knowledge is locked inside text. Consider a healthcare system where clinical notes describe patient diagnoses, medications, and outcomes, yet the electronic health record contains no structured fields for most of this information. Or a financial analyst who needs to track every acquisition announcement across thousands of news articles. Or a legal team that wants to identify all contract clauses containing liability caps. In each case, the raw information exists, but it exists as prose, and prose is inaccessible to the aggregation and query tools that organizations depend on.

The business case for IE is straightforward: the world produces far more text than any team of analysts can read. Financial firms track mergers and acquisitions across thousands of news sources per day. Pharmaceutical companies scan biomedical literature for drug-protein interactions. Intelligence analysts monitor news feeds for geopolitical events. In all of these settings, the alternative to automated IE is a team of human readers who are slower, more expensive, and less consistent than a well-designed extraction system. IE does not replace human judgment on complex questions, but it handles the mechanical task of turning text into structured records that humans can then analyze.

Information extraction has a long history in NLP. Early systems used hand-crafted patterns and regular expressions, which worked well for narrow domains but broke on anything outside their rules. The MUC (Message Understanding Conference) evaluations of the late 1980s and 1990s established the core IE task definitions, standardized evaluation, and drove progress on template-filling tasks like extracting information about terrorist incidents from news reports. Statistical methods in the 1990s and 2000s introduced machine-learned sequence taggers that generalized better but still required annotated training data and struggled with ambiguity.

The transformer era, and especially large language models (LLMs), changed the calculus. A model pretrained on billions of documents carries broad world knowledge and flexible parsing capability. With careful prompting or lightweight fine-tuning, it can extract structured information from nearly any text without domain-specific engineering. This has dramatically lowered the barrier to building IE systems, though the fundamental challenges of ambiguity, coreference, and domain shift remain.

This chapter covers the four main IE tasks, building from the simplest to the most complex:

• Named entity recognition (NER): finding and classifying entity mentions (people, places, organizations, dates, etc.)
• Relation extraction (RE): identifying semantic relationships between pairs of entities
• Event extraction (EE): detecting events and their participants, times, and locations
• Structured output generation: producing complete, schema-compliant JSON or other structured records from unstructured text

For each task we develop the core concepts, work through concrete examples, and implement practical code using both traditional tools and modern LLM-based approaches. By the end you will have a full IE pipeline that can turn a paragraph of news text into a structured record ready for a database or knowledge graph.

Named entity recognition is the entry point for nearly all IE pipelines. Before you can extract a relationship like "Apple acquired Beats," you need to know that "Apple" is an organization and "Beats" is another organization. NER is what provides that grounding: a systematic scan of text that finds and labels every entity mention.

The term "named entity" was introduced in the MUC-6 evaluation in 1996 and originally referred to proper names of specific real-world entities: people, places, organizations. Modern usage has expanded to include temporal expressions, monetary values, percentages, and other categorically meaningful mentions that lack proper names. This expansion reflects practical needs: a financial extraction system needs to identify "$3 billion" as a money amount just as much as it needs to identify "Apple" as an organization.

Why does NER matter so much as a first step? Because almost every downstream IE task depends on knowing what kind of thing each mention refers to. Relation extraction uses entity types to filter candidate pairs: you check for employment relations between PERSON-ORG pairs, not between DATE-MONEY pairs. Event extraction uses entity types to assign argument roles: a PERSON entity near a firing verb is more likely the employee than the organization. Without NER, you are working with anonymous strings, and the combinatorial explosion of possible pairings makes precise extraction nearly impossible at scale.

An NER system scans a piece of text and labels each entity mention with a type. The standard entity categories, established by datasets like CoNLL-2003 and OntoNotes, include:

• PER (Person): "Barack Obama," "Marie Curie," "Elon Musk"
• ORG (Organization): "Apple," "NASA," "European Union"
• LOC (Location): "Mount Everest," "the Amazon River"
• GPE (Geopolitical Entity): "France," "New York City," "the European Union" (when used as a political entity)
• DATE / TIME: "March 15," "9:30 AM," "the 1990s"
• MONEY: "$3 billion," "€200 million"
• PRODUCT: "iPhone 15," "Model 3"

Different frameworks use slightly different taxonomies. spaCy's en_core_web_trf model recognizes 18 entity types following OntoNotes conventions. BERT-based biomedical models may add GENE, DISEASE, and CHEMICAL. The right taxonomy depends on your domain.

The boundary between types is not always clean. "Washington" can be a person (George Washington), a place (Washington state), or a geopolitical entity (Washington D.C.). "March" is usually a month, but in the sentence "The soldiers were ordered to march" it is a verb, not an entity at all. Context resolves these ambiguities, and this is precisely why modern contextual encoders like BERT outperform earlier feature-engineered systems. A static dictionary lookup for "Washington" cannot know which sense applies; a transformer that has seen thousands of sentences with "Washington D.C." and "Washington state" can infer the correct type from the surrounding tokens.

NER is typically framed as a sequence labeling problem. Given a sequence of $n$ tokens $\mathbf{x} = (x_1, x_2, \ldots, x_n)$, the model assigns one tag $y_i$ to each token $x_i$. The most common tagging scheme is BIO (also called IOB2), where each tag belongs to the set:

where:

• $\text{B-TYPE}$ marks the Beginning of an entity of the given type (e.g., B-PER for the start of a person name)
• $\text{I-TYPE}$ marks a token Inside (continuing) an entity of the same type (e.g., I-PER for a second or later token of a person name)
• $\text{O}$ marks a token Outside any entity

For the sentence "Marie Curie won the Nobel Prize":

The BIO scheme lets spans of arbitrary length be represented with only a small tag vocabulary. A system that predicts these tags can locate and type any entity, regardless of how many tokens it covers. Multi-token entities are identified by reading consecutive B and I tags of the same type. The B marker is critical: it distinguishes between a single entity "New York" (B-GPE, I-GPE) and two separate entities "New" and "York" appearing back to back (B-GPE, B-GPE).

Modern transformer-based NER models compute a contextual representation $\mathbf{h}_i$ for each token $x_i$ via the encoder, then pass it through a linear classification head:

where:

• $\mathbf{h}_i \in \mathbb{R}^d$ is the contextual token embedding produced by the transformer encoder for position $i$
• $\mathbf{W} \in \mathbb{R}^{|\mathcal{Y}| \times d}$ is a learned weight matrix mapping from embedding space to tag scores
• $\mathbf{b} \in \mathbb{R}^{|\mathcal{Y}|}$ is a learned bias vector
• $|\mathcal{Y}|$ is the number of possible tags (e.g., $2 \times |\text{types}| + 1$ for BIO)

Because the encoder sees the full sentence bidirectionally, it can resolve ambiguities that left-to-right models would miss: "Washington" near "D.C." is a GPE, while "Washington" near "the Founding Fathers" is a PER. This bidirectional context was the primary advantage that BERT-based NER systems demonstrated over LSTM-CRF systems when they were introduced in 2018 to 2019.

Some systems add a conditional random field (CRF) layer on top of the per-token predictions. The CRF models tag transition probabilities explicitly, making sure that the output sequence is globally consistent: for example, an I-PER tag cannot follow a B-ORG tag because that would imply the same token continues two different entities. In practice, modern large transformers are expressive enough that the CRF layer provides only marginal improvement, but it can still help on smaller datasets or when the training set has inconsistent annotations.

The following visualization shows the typical NER F1 performance of different system classes on the CoNLL-2003 English benchmark, illustrating how accuracy has improved from rule-based systems through statistical models to transformer-based approaches.

spaCy is the most widely used NLP library for production NER. Its transformer-backed en_core_web_trf model achieves state-of-the-art accuracy on standard benchmarks with minimal setup. For this walkthrough we use the smaller en_core_web_sm model, which is faster and sufficient to illustrate the key concepts.

The model identifies "Apple" and "Beats Electronics" as organizations, "$3 billion" as a money amount, "May 2014" as a date, and "Cupertino, California" as a geopolitical entity. Notice that "Tim Cook" is recognized as a person, and the character offsets tell you exactly where in the source text each entity was found. These offsets are valuable for downstream tasks: you can display entities highlighted in the original text, or you can use them to extract the surrounding context for relation extraction.

One subtlety to notice: "Apple's" is correctly resolved to "Apple" as an organization, even though the possessive suffix "'s" is attached. spaCy's tokenizer splits the "'s" into a separate token, and the NER model correctly assigns the entity span to just "Apple". This kind of tokenization-aware entity boundary detection is straightforward for well-resourced languages like English but becomes trickier for languages with rich morphology, where a single word form may encode the entity plus several grammatical markers.

Standard BIO labeling cannot represent nested entities: in the phrase "University of California, Berkeley," both "University of California, Berkeley" and "California" are valid entities, but a flat BIO tagger can only assign each token to one span. More complex schemes like BIOUL (Begin, Inside, Outside, Unit, Last) or nested NER architectures handle this, but for most practical pipelines, flat NER is sufficient. The convention is to label the longest containing span.

The nested entity problem is more common in domains like biomedicine, where gene names, protein complexes, and cell types frequently appear as constituents of longer named phrases. A gene name like "TP53" may appear inside a phrase like "TP53 mutation," which is itself a clinical finding. Dedicated nested NER architectures enumerate all possible spans and classify each one independently, treating the problem as multi-label span classification rather than a sequential tagging problem. This increases computational cost but allows complete annotation of overlapping structures.

A related challenge is entity mention disambiguation. The text "Apple" can refer to the technology company, the fruit, or a music label. NER labels the mention's type but does not link it to a unique knowledge base entry. That step, called entity linking or entity disambiguation, pairs NER with a lookup against a knowledge base like Wikidata or Freebase to produce canonical identifiers. Systems like REL and BLINK train dedicated entity linking models that score candidate knowledge base entries against the surrounding context and return the best match. Entity linking is a prerequisite for knowledge graph population, where the same entity should always map to the same node regardless of how it is mentioned in text. Without it, a system may create duplicate nodes for "Apple Inc.," "Apple," and "AAPL" that should all point to the same company.

Once you have a set of entity mentions, the next question is how they relate. Relation extraction identifies and classifies semantic relationships between entity pairs mentioned in the same text. The output is a set of triples: (subject entity, relation type, object entity).

From the sentence "Apple acquired Beats Electronics for $3 billion in May 2014," relation extraction should produce the triple (Apple, acquired, Beats Electronics). Combined with NER output, the full structured record becomes:

This triple format, borrowed from the Resource Description Framework (RDF), is the basis of knowledge graphs. Systems like Google's Knowledge Graph and Wikidata are populated partly through automated relation extraction at scale. A knowledge graph that connects millions of entities through typed relations supports queries that would be impossible with keyword search alone: "List all AI companies acquired by tech giants for more than $1 billion since 2020."

Relation extraction is harder than NER for a fundamental reason: it is a binary task over pairs of entities, and the number of candidate pairs grows quadratically with the number of entities in a document. A document with 20 entities has $20 \times 19 / 2 = 190$ candidate pairs, most of which express no relation at all. The system must distinguish the small set of true relations from the large set of unrelated entity co-occurrences. This makes precision a particularly difficult metric to optimize: a system that predicts "no relation" for all pairs achieves perfect precision on the "no relation" class but extracts nothing useful.

The most common formulation treats RE as a classification problem. Given a sentence and two marked entity spans, predict the relation type (or "no relation" if the entities are unrelated). The entity types from NER serve as strong features: if both entities are ORG, the system focuses on organization-organization relations (acquired, merged_with, subsidiary_of), while ORG-PER pairs suggest employment or founding relations (founded_by, CEO_of, employed_by).

This type-based filtering is important for efficiency and precision. Restricting to plausible type combinations reduces the search space dramatically and avoids nonsensical pairings like checking whether a date is the CEO of a location. In practice, the mapping from entity type pair to plausible relation types is determined by the schema you are using. For the ACE relation set, the mapping is explicit in the task definition. For open-domain extraction, you can use the relation type definitions themselves to infer which entity types are compatible.

Mathematically, given entity spans $e_1$ and $e_2$ within sentence $\mathbf{x}$, a classifier computes the probability of each relation type $r$ as:

where:

• $r \in \mathcal{R}$ is a candidate relation type from a fixed label set $\mathcal{R}$ (e.g., "acquired," "founded_by," "located_in," "no_relation")
• $\mathbf{h}_{e_1, e_2} \in \mathbb{R}^d$ is a combined representation built from the encoder's [CLS] token embedding and the pooled embeddings of both entity spans
• $\mathbf{W} \in \mathbb{R}^{|\mathcal{R}| \times d}$ and $\mathbf{b} \in \mathbb{R}^{|\mathcal{R}|}$ are learned classification parameters

The entity span representations are typically obtained by pooling the token embeddings within each span, or by using special marker tokens (such as <e1> and </e1>) inserted around the entity mentions before encoding. These markers serve as positional cues that make the entity boundaries salient to the attention mechanism. The TACRED and DocRED datasets are the standard benchmarks for relation extraction. TACRED covers 42 relation types in a news and web context; DocRED extends this to document-level extraction where the subject and object entities may appear in different sentences. Models based on BERT or RoBERTa achieve F1 scores in the mid-70s on TACRED, showing the difficulty of the task.

The chart below compares how relation extraction F1 degrades across different sentence-level distances between entity pairs, illustrating that RE becomes substantially harder when entities are far apart in the sentence.

Annotating relation extraction training data is expensive: every sentence requires a human to identify entity pairs and label their relationship. Distant supervision offers an alternative. The core assumption is: if a knowledge base asserts the triple (Apple, acquired, Beats), and a corpus sentence contains both "Apple" and "Beats," that sentence likely expresses the acquired relation. This auto-generates millions of labeled examples without human annotation.

The assumption is noisy. Many sentences mentioning two entities do not express the expected relation, or express a different one. "Apple and Beats collaborated on audio products before the acquisition" mentions both entities but expresses collaboration, not acquisition. "Beats was founded by Dr. Dre before Apple existed as a company" mentions Apple and Beats but expresses no relation relevant to the acquisition. Models trained on distantly supervised data must therefore be robust to label noise.

Multi-instance learning approaches address this noise by aggregating evidence across multiple sentences about the same entity pair. If a knowledge base asserts (Apple, acquired, Beats), and ten sentences in the corpus mention both entities, at least some of them likely do express the acquisition relation. A model trained to classify the entity pair, using the bag of sentences as input rather than any single sentence, is more robust to individual mislabeled examples. The model learns to identify the most expressive sentence in the bag, ignoring the noisy ones, rather than treating every sentence as equally reliable evidence.

This aggregation idea connects distant supervision to a broader principle in IE: complementary evidence. A single sentence may be ambiguous, but multiple sentences mentioning the same entity pair tend to cluster around the most frequent true relation. Aggregation exploits this statistical regularization without requiring any human labels beyond the knowledge base itself.

Separating NER and RE into pipeline stages introduces error propagation: NER mistakes create bad entity boundaries that make RE fail even with a perfect relation classifier. If NER labels "Beats Electronics" as two separate entities, "Beats" (ORG) and "Electronics" (something else), the relation classifier never sees the correct entity pair and cannot produce the acquisition relation.

Joint models tackle both tasks simultaneously, sharing encoder representations and training with a combined objective. The encoder processes the sentence once, and separate prediction heads for entity typing and relation classification share the same contextual representations. This sharing means the model can learn that the presence of the word "acquired" between two ORG entities is a strong signal for both the entity type classification and the relation type classification, rather than learning each independently.

Modern joint extraction models often frame the problem as span prediction: enumerate all possible spans up to some maximum length, score each as an entity or non-entity, then score all entity-pair combinations for relation types. The table-filling approach represents this as a matrix where rows and columns are tokens and each cell encodes the relationship between the span starting at that row and ending at that column. This matrix representation enables efficient computation and naturally handles overlapping entities. SpERT (Span-based Entity and Relation Transformer) is a representative joint model that operates in this way, processing all spans and entity pairs in a single forward pass through a BERT encoder and achieving strong performance on both the CoNLL-2004 and SciERC benchmarks.

The tradeoff between pipeline and joint models is not always in favor of joint models. Pipeline models are easier to debug, easier to update (you can swap the NER model without touching RE), and easier to train because you can use separate datasets for each task. Joint models require co-annotated data where entities and relations are labeled in the same document, which is harder to obtain. For most production systems where well-labeled data is available and latency matters, a carefully designed pipeline outperforms a joint model despite the error propagation risk.

Events are a richer structure than binary relations. An event is a change of state or occurrence involving multiple participants. Where relation extraction produces a binary triple (Apple, acquired, Beats), event extraction produces a frame with a type, a trigger, and an arbitrary number of typed argument slots.

The ACE (Automatic Content Extraction) framework, developed by NIST for a series of evaluations from 2004 to 2008, remains the dominant formalism. It defines events with:

• Event trigger: the word or phrase that most clearly expresses the event (e.g., "acquired," "exploded," "elected")
• Event type: a category from a fixed ontology (e.g., Business:Merge, Conflict:Attack, Personnel:Elect)
• Arguments: the roles played by entities in the event, each with an argument role label

For the sentence "Apple acquired Beats Electronics for $3 billion in May 2014," the event extraction output is:

• Trigger: "acquired"
• Event type: Business:Merge-Org (in ACE terminology)
• Arguments:
  • Buyer: Apple [ORG]
  • Artifact: Beats Electronics [ORG]
  • Price: $3 billion [MONEY]
  • Time: May 2014 [DATE]

Event extraction is harder than relation extraction for several reasons. First, a single sentence may contain multiple events: "Apple acquired Beats and fired its CEO" contains both an acquisition event and a personnel event. Second, arguments can be implicit: the buyer is not always stated in every sentence that describes an acquisition. Third, the event trigger may be a verb, noun, or adjective, making trigger detection broader than simply identifying action verbs: "acquisition" (noun), "acquired" (verb), and "acquisitive" (adjective) all trigger the same event type. Fourth, coreference means that "the deal" in the next sentence refers to the same acquisition event, and a full system needs to track this cross-sentence linkage.

The framing of event extraction as structured information extraction connects it to semantic role labeling, a task that assigns argument roles to predicates in sentences regardless of whether they describe named events. The difference is that event extraction uses a closed ontology of event types with fixed argument roles (Buyer, Seller, Price for an acquisition), while SRL uses a more flexible, verb-centric argument labeling scheme (Agent, Patient, Instrument, etc.). Both tasks share the insight that the meaning of a predicate cannot be understood without knowing who is doing what to whom.

Trigger detection is framed as a token classification problem: for each token, predict whether it is a trigger and, if so, which event type it initiates. This is structurally identical to NER but at the event level. The model applies the same classification formula:

where:

• $x_i$ is the candidate trigger token at position $i$
• $\mathbf{h}_i$ is its contextual embedding from the transformer encoder, seeing the full sentence $\mathbf{x}$
• $\mathbf{W}_t$ and $\mathbf{b}_t$ are parameters specific to the trigger classification head

The challenge is that trigger words are often ambiguous. "Fired" can mean a Personnel:End-Position event ("fired the CEO") or a Conflict:Attack event ("fired a missile"). "Died" in a clinical context triggers a Life:Die event, while in "the battery died" it is not an event trigger at all. Context is essential. Modern systems use the same transformer encoders as NER, but the label space now includes event types rather than entity types. The ACE 2005 dataset, the standard benchmark, includes 33 event subtypes across 8 main categories, making this a challenging multi-class problem with a large, imbalanced label set: some event types appear thousands of times in training data, while others appear only dozens of times.

One practical difficulty with trigger detection is that the same event type can be triggered by many different words. A Business:Merge event might be triggered by "acquired," "purchased," "bought," "took over," "absorbed," or "merged with," among others. The model must generalize across all of these surface forms. Pre-trained transformers handle this well because they encode semantic similarity in their representations: "acquired" and "purchased" have similar embeddings in context, so the classification boundary that fires on "acquired" also tends to fire on "purchased." But low-frequency trigger words, those appearing only once or twice in training, are learned poorly regardless.

Given a detected trigger, argument role labeling identifies which entity spans fill which roles. This resembles semantic role labeling (SRL), a well-studied NLP task that annotates "who did what to whom." For the trigger "acquired," the system must find the agent (who acquired), the patient (what was acquired), and any additional arguments (price, time, location).

The argument roles are defined per event type in the ontology. A Business:Merge-Org event has Buyer, Seller, and Artifact roles. A Personnel:Elect event has Person, Entity (the organization), and Position roles. The model must learn these type-specific argument patterns from training data. The number of argument roles across all event types in ACE 2005 is 35, so the model must classify which argument role a span fills while conditioning that decision on the event type.

The interaction between trigger detection and argument labeling creates a joint inference problem: the best trigger interpretation depends on which entities are available as arguments, and the best argument assignment depends on the trigger type. A trigger "fired" surrounded by a PER entity (the employee) and an ORG entity (the employer) is more likely to be a Personnel:End-Position event than a Conflict:Attack event. End-to-end systems trained with joint losses handle these dependencies better than sequential pipelines, though joint training is more complex to implement.

An important practical limitation of argument role labeling is argument span coverage. Not every argument role will be filled in every event mention: an acquisition announcement may not state the seller, the price, or the time. Systems must learn to output a partial argument structure when some roles are absent, rather than hallucinating values for unmentioned roles. This is particularly important when using LLMs for event extraction, where the model may generate plausible-sounding but unsupported argument values.

The following visualization shows how precision and recall trade off for event trigger detection compared to argument role labeling on the ACE 2005 benchmark, illustrating why event extraction lags behind NER on standard metrics.

One of the most practical shifts in event extraction over the past few years is the use of LLMs to bootstrap extraction without labeled training data. Traditional ACE-style systems require hundreds of annotated examples per event type. LLMs allow you to describe the event type and argument roles in natural language and extract events immediately.

The approach is called zero-shot or few-shot event extraction. You provide the model with a description of the event ontology in the prompt: "Identify all acquisition events. For each event, extract the Buyer (the acquiring company), the Target (the acquired company), the Price (the monetary value of the deal, or null if not mentioned), and the Date (when the deal was announced or completed, or null if not mentioned)." The model then applies this schema to the input text without any task-specific training.

Zero-shot event extraction performs surprisingly well on well-defined, common event types that appear frequently in training data. Acquisitions, elections, attacks, and natural disasters are all event types that LLMs have seen thousands of times in pretraining text. Performance degrades on rare or domain-specific event types that the model has seen few examples of. For those cases, few-shot prompting, giving two or three annotated examples in the prompt, substantially improves accuracy.

The traditional IE pipeline (NER, RE, EE) is powerful but brittle. Each stage introduces its own errors, and adapting to a new domain or a new schema requires retraining multiple models. Large language models offer a different approach: instruct the model to extract all relevant information in one pass and return a structured JSON object.

This approach uses the LLM's broad world knowledge and flexible reasoning. A model that has read millions of news articles already "knows" that acquisitions involve buyers, sellers, prices, and dates. It understands that "deal," "transaction," and "purchase" are all synonyms for an acquisition event, and that "five billion dollars," "$5B," and "USD 5,000,000,000" all express the same monetary value. You can directly prompt it to structure this knowledge without annotating training data.

The tradeoffs are significant. LLM-based IE is easier to adapt to new schemas and domains, but each inference call costs more than running a fine-tuned sequence tagger, and the model may hallucinate fields that are not present in the source text. For high-volume production workloads processing millions of documents, the cost difference matters. For exploratory work or low-volume high-complexity extraction, LLM-based approaches are often the practical choice. A common production pattern is to use an LLM to validate and develop the schema on a sample of documents, then train a fine-tuned model for high-volume extraction, and use rule-based post-processing to enforce hard constraints on both.

The key to reliable structured extraction is a precise, schema-defining prompt. The schema should specify the exact fields expected, their types, and what to include when information is absent. Vague prompts produce inconsistent output. A prompt that says "Extract the key information" may return a narrative summary; a prompt that defines a specific JSON schema returns a consistently structured object.

Equally important is the instruction for handling missing information. If you say "Extract the deal value," a model that finds no deal value in the text may either return null or invent a plausible number. Explicitly instructing the model to "Set any field to null if the information is not present in the text" substantially reduces hallucination, because the model is given a valid, expected output for the case where information is absent.

The extracted JSON is machine-readable and maps directly to a database schema. This one-pass approach is particularly powerful when the schema is complex or when fields are expressed in varied natural language patterns that would require many separate pattern rules to capture. Notice that the LLM has normalized "May 2014" to a proper ISO date format ("2014-05-01"), converted "$3 billion" to an integer number of USD, and added the implied title "CEO" for Tim Cook, all without being explicitly instructed to do so.

This normalization behavior is one of the most practically valuable properties of LLM-based extraction. Traditional rule-based systems require explicit normalization rules for every surface form: separate patterns for "May 2014," "05/2014," "the fifth month of 2014," and "Q2 2014." An LLM understands all of these as referring to the same time period and normalizes them to ISO format if the schema specifies an ISO date field. The same applies to monetary values expressed as "$3B," "$3 billion," "three billion dollars," or "USD 3,000,000,000."

Modern LLM APIs provide structured output features that guarantee syntactically valid JSON regardless of what the model generates. OpenAI's function calling mechanism and structured output mode, and similar features in Anthropic's API and Google's Gemini, work by constraining the model's token sampling to only produce outputs that conform to a provided JSON Schema.

The mechanism is elegant: during decoding, the sampling process is constrained so that only tokens compatible with the current parse state of the expected JSON Schema can be generated. If the schema says the "amount" field must be an integer, the model cannot generate a string in that position. This is implemented via constrained decoding with a finite state machine derived from the JSON Schema. As we saw in the chapter on constrained decoding, this approach guarantees syntactic validity while allowing the model's full semantic capability to operate within the constraint.

The practical benefit is that you no longer need to handle JSON parse errors in your application code. Without structured output mode, a simple retry loop catches the occasional malformed JSON. With structured output mode, every response is guaranteed to be valid JSON that conforms to the schema. This is especially valuable for production systems where a malformed response that crashes a parsing pipeline at 2 AM is a reliability incident.

When structured output mode is not available, adding "Return only valid JSON. Do not include any explanation." to the prompt and validating with json.loads() with a retry on parse failure achieves similar reliability. A simple retry loop that re-prompts with "The previous response was not valid JSON. Try again." handles most failure cases within two attempts. For the rare case where the model consistently fails to produce valid JSON for a particular input, logging the failure and falling back to a rule-based extraction or manual review is the appropriate response.

Many real documents contain multiple events or multiple entities of the same type. A single news article may mention five acquisitions. The schema must accommodate lists, and the prompt must be explicit about including all instances.

The challenge is completeness: LLMs tend to focus on the most prominent event in a document and undercount secondary events. Explicitly asking for "a list of ALL events" and specifying that the list may contain multiple items reduces this problem. For critical applications, running a separate extraction pass per candidate entity pair (found by an NER scan) and then deduplicating is more reliable than a single all-events extraction.

The list-based schema forces the model to enumerate all events explicitly. This is more reliable than asking for a "summary" because the structure makes completeness auditable: you can check whether the number of events in the JSON matches the number of acquisition verbs in the source text. Including a "status" field ("completed" vs. "abandoned") captures negation, which is a common failure mode for rule-based systems that trigger on keywords like "acquisition" without detecting the negative context "was later abandoned."

Negation handling deserves special attention. A sentence like "Adobe's attempted acquisition of Figma was later abandoned" contains an acquisition trigger but describes an event that did not complete. Rule-based systems often miss this because they match keywords without understanding the modifier "attempted" and the completion marker "abandoned." LLMs handle this naturally because they understand the sentence semantically, not just lexically. Fine-tuned classifiers can also learn negation patterns if they are represented in training data, but they require negative examples that explicitly mark abandoned, failed, or hypothetical events.

Let us now combine NER, relation extraction, and structured output into an end-to-end pipeline. The pipeline takes raw text, identifies entities, and generates a structured record. This is the pattern used in production IE systems: NER provides entity boundaries and types, a relation classifier or LLM determines relationships, and a post-processing step validates and normalizes the output.

The pipeline we build here is intentionally simple: it uses spaCy for NER and a rule-based relation classifier. In production, the relation classifier would be replaced by a fine-tuned BERT model or an LLM call, and the post-processing step would validate against a JSON Schema. But the overall structure, the composition of modular stages, remains the same.

The pipeline correctly identifies the acquisition relationship between Apple and Beats Electronics. The rule-based classifier here is illustrative: production systems use fine-tuned BERT-based classifiers trained on labeled relation examples, which generalize to many more surface forms. The classifier receives the entity types from NER as features alongside the sentence context, making the combination of NER and RE complementary rather than independent. Notice also that the classifier operates only on ORG-ORG pairs: if we had also checked PER-ORG pairs, we might find an employment relation between Tim Cook and Apple. The type-based filtering keeps the search focused and reduces the false positive rate.

The key parameters and design choices for an IE pipeline are:

• NER model: en_core_web_sm (fast, lower accuracy) vs en_core_web_trf (slower, higher accuracy). Use the transformer model when precision matters.
• Entity type filter for RE: restricting which entity type pairs to consider (ORG-ORG for acquisitions, PER-ORG for employment) reduces false positives.
• Relation classifier threshold: in probabilistic classifiers, lowering the confidence threshold increases recall but also false positives.
• Schema strictness: required fields in JSON Schema enforce that the LLM always populates critical slots; optional fields reduce false "null" values.

The pipeline approach described above has an important architectural property: each component is independently swappable. If a better NER model becomes available, you can swap it in without touching the relation classifier. If you need to add a new relation type, you can update the classifier's label set without retraining the NER model. This modularity is valuable in production systems that evolve over time.

The cost of modularity is error propagation. An entity boundary error in NER creates a bad input to the relation classifier, which may then either miss the correct entity pair entirely or classify a wrong pair. In practice, the most common NER errors are at entity boundaries, not entity types: the classifier knows that "Beats Electronics" is an organization, but may label the span as "Beats" instead of "Beats Electronics." The relation classifier then sees "Beats" as one entity and tries to relate it to other ORG spans, possibly failing to find the correct pair.

One mitigation is to use soft entity boundaries: instead of passing hard entity spans to the relation classifier, pass the NER model's uncertainty along with the span boundaries. A relation classifier that is aware that "Beats Electronics" might be "Beats" with some probability can account for this uncertainty in its prediction. This fuzzy pipeline approach is less common in practice because it complicates the interface between components, but it can improve end-to-end metrics when NER errors are a significant bottleneck.

Evaluating IE systems requires task-specific metrics. The evaluation philosophy is the same across all IE tasks: partial credit is not given. Either a prediction matches the gold annotation exactly, or it counts as an error.

For NER, a prediction is correct only if both the span boundaries and the entity type match exactly. The standard metrics are entity-level precision, recall, and F1. To understand why these are defined at the entity level rather than the token level, consider a system that correctly identifies the tokens "New" and "York" but labels them as two separate entities instead of one "New York" span: token-level accuracy would be high, but entity-level evaluation would mark this as an error because the correct span boundary is not reproduced.

Precision measures how many predicted entities are correct:

Recall measures how many gold entities the model successfully predicted:

The harmonic mean of precision and recall, F1, provides a single balanced score:

where:

• $|\text{TP}|$ is the count of True Positives: predicted spans that match a gold span in both token boundaries and entity type
• $|\text{FP}|$ is the count of False Positives: predicted spans with no matching gold span (incorrect prediction or wrong type)
• $|\text{FN}|$ is the count of False Negatives: gold spans that the model failed to predict entirely

The harmonic mean is used rather than the arithmetic mean because it penalizes imbalance between precision and recall more heavily. A system with precision 1.0 and recall 0.0 would have an arithmetic mean of 0.5 but an F1 of 0.0, correctly reflecting that the system is useless because it extracts nothing.

For relation extraction, evaluation is similarly span-based: a predicted triple (subject, predicate, object) is correct only when all three components match the gold annotation. This means RE F1 scores are always lower than NER F1 scores on the same document: a correct relation requires a correct subject entity AND a correct object entity AND a correct relation type. If the NER system has F1 of 0.90, the upper bound on RE F1 is approximately $0.90^2 \approx 0.81$, even with a perfect relation classifier. This compounding effect is why end-to-end IE pipelines are evaluated separately from their individual components, and why improvements in NER propagate through to RE.

For structured output, schema validation is a minimum bar (did the model produce valid JSON with all required fields?). Beyond that, field-level accuracy measures what fraction of fields are correctly populated, and document-level accuracy measures what fraction of complete records are perfect. Document-level accuracy is a demanding metric: a record with nine correct fields and one wrong field scores zero. For applications where a partially correct record is still useful (e.g., database entries that can be manually corrected), field-level accuracy is more informative.

Evaluating LLM-based structured extraction introduces additional complexity beyond traditional NER and RE metrics. The output is a structured record rather than a sequence of labels, so evaluation must compare records field by field.

The simplest approach is exact string match per field: the extracted company name "Apple Inc." does not match the gold "Apple" even though they refer to the same entity. This motivates normalized evaluation, where both the gold and predicted values are normalized before comparison. For monetary values, normalization converts all forms to a canonical integer; for dates, it converts all forms to ISO 8601; for company names, it maps common abbreviations and suffixes to a canonical form.

A second challenge is evaluating null fields: when the model correctly returns null for a field that has no value in the source text, that is a true negative, and it should contribute positively to precision but not to recall. Traditional precision and recall formulas do not naturally handle true negatives in structured extraction, because F1 is defined over positive predictions. A common pragmatic solution is to exclude null fields from the denominator when computing field-level accuracy, and to separately track the null prediction error rate (how often the model predicts null when a value is present, or predicts a value when the text contains none).

Information extraction systems face a set of recurring challenges that no current approach fully resolves. Understanding these limitations helps you design systems that degrade gracefully when the unexpected occurs.

Schema coverage. Any IE system is constrained by its output schema. A system trained or prompted for acquisition events will not capture employment events unless explicitly asked. As the world produces new event types (e.g., large language model releases, regulatory actions against AI systems), schemas need to evolve. LLM-based systems adapt more gracefully because the model's world knowledge extends beyond any fixed label set, but this comes at the cost of harder evaluation and less predictable behavior. Schema evolution also creates a data management problem: historical records extracted under the old schema may need to be re-extracted under the new one, which is expensive if you have millions of documents.

Coreference and cross-sentence extraction. Most IE approaches work sentence by sentence, yet facts are often spread across multiple sentences. "Apple made the announcement on Tuesday. The deal, which values the company at $20 billion, is expected to close in Q3." Resolving "the deal" and "the company" requires coreference resolution, a separate and difficult NLP task. Full document-level IE that tracks entities and events across paragraphs remains an active research area, with neural coreference resolvers like the SpanBERT-based system of Joshi et al. (2020) achieving strong performance on standard benchmarks but still struggling on long documents and rare entity types. In practice, most production IE systems handle cross-sentence extraction either by processing paragraphs as single units (relying on the LLM's or transformer's long-context capability) or by running a separate coreference resolution step before extraction.

Hallucination in LLM extraction. When prompted to extract a field like "deal value," LLMs sometimes invent a plausible-sounding number if no value is present in the text. This is a critical failure mode in fact extraction. Mitigation strategies include requiring the model to quote the exact text span that supports each extracted value, using temperature 0 for deterministic extraction, and cross-validating against the source text programmatically. Attribution-based extraction, where each extracted value is paired with its source quote, makes hallucination immediately detectable. If the model claims the deal value is "$3 billion" and the quoted span does not contain that value, you have a hallucination. This pattern, extract with attribution and verify the attribution, is becoming a standard practice in production LLM extraction pipelines.

Domain shift. A NER model trained on news performs poorly on clinical notes, legal documents, or social media. Entity types differ, writing styles differ, and even the same words carry different meanings. "Discharge" in a medical context triggers a clinical event; in a news context it may trigger a Personnel:End-Position event. "Parties" in a legal document refers to the signatories of a contract; in news it may refer to political parties or social gatherings. Fine-tuning on domain-specific data is the standard solution, but annotating domain data is expensive. Few-shot prompting with LLMs is often a more practical first step for new domains, and domain-adaptive pretraining, continuing pretraining on domain text before fine-tuning, bridges the gap when labeled data is available.

Evaluation against human annotation. Even human annotators disagree on entity boundaries and relation types. Annotation agreement (measured by inter-annotator agreement, often reported as Cohen's kappa) for complex event extraction tasks can be surprisingly low, on the order of 0.6 to 0.8. Model performance comparisons are only meaningful if both systems are evaluated against the same gold standard, ideally one with high annotation agreement. When agreement is low, it is worth examining whether the disagreements cluster on specific entity types or event roles, which can reveal ambiguities in the annotation guidelines. The practical implication is that reported F1 scores on benchmarks like ACE 2005 or CoNLL-2003 represent performance relative to one set of annotation decisions, not absolute accuracy relative to some objective ground truth. Different annotation teams might produce different gold standards from the same text.

Scalability and cost. For high-volume extraction workloads processing millions of documents per day, the cost difference between fine-tuned sequence taggers and LLM-based extraction becomes significant. A fine-tuned BERT NER model processes thousands of sentences per second on a GPU, while an LLM call to a commercial API costs several cents per call and introduces network latency. The appropriate architecture depends on volume, latency requirements, and the complexity of the extraction task. Very complex, low-volume extraction (e.g., extracting structured records from legal agreements) often justifies LLM costs. High-volume, simpler extraction (e.g., tagging entities in a news stream) often favors fine-tuned models.

Information extraction turns free-form text into structured data. We covered the four main tasks in the IE stack:

• Named entity recognition identifies and classifies entity mentions in text using BIO sequence labeling. Modern transformer-based NER models like spaCy's en_core_web_trf achieve high accuracy with minimal setup. The key insight is that contextual representations from bidirectional transformers resolve entity type ambiguities that static features cannot.
• Relation extraction identifies semantic relationships between entity pairs, producing (subject, predicate, object) triples suitable for knowledge graphs. The pipeline approach applies a relation classifier to entity pairs identified by NER, with entity type filtering to reduce the candidate pair space. Distant supervision enables training at scale without expensive manual annotation.
• Event extraction captures structured descriptions of occurrences, including the event trigger, type, and argument roles. It is the most complex IE task because a single sentence may contain multiple events with shared arguments, triggers are ambiguous, and arguments can be implicit or spread across sentences.
• Structured output generation uses LLMs with precise schema-defining prompts to extract complete, schema-compliant JSON records in a single pass. JSON Schema validation and function calling APIs enforce output structure. Attribution-based extraction, where each value is paired with its source span, mitigates the hallucination risk.

The choice of approach depends on your requirements. Rule-based and traditional ML systems offer interpretability and low latency but require domain-specific engineering. Fine-tuned transformer models achieve the best accuracy on well-defined tasks with annotated training data. LLM-based extraction is fastest to adapt to new schemas and domains, at the cost of higher inference cost and hallucination risk. In practice, production IE systems often combine all three: LLMs for initial schema exploration and low-volume complex extraction, fine-tuned models for high-volume extraction, and rule-based post-processing to enforce hard constraints on both. The key evaluation metric across all approaches is entity-level F1, which gives no partial credit and therefore accurately reflects the all-or-nothing nature of most downstream applications.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about information extraction.