Information Extraction › Limitations and Practical Considerations

Information extraction systems face a set of recurring challenges that no current approach fully resolves. Understanding these limitations helps you design systems that degrade gracefully when the unexpected occurs.

Schema coverage. Any IE system is constrained by its output schema. A system trained or prompted for acquisition events will not capture employment events unless explicitly asked. As the world produces new event types (e.g., large language model releases, regulatory actions against AI systems), schemas need to evolve. LLM-based systems adapt more gracefully because the model's world knowledge extends beyond any fixed label set, but this comes at the cost of harder evaluation and less predictable behavior. Schema evolution also creates a data management problem: historical records extracted under the old schema may need to be re-extracted under the new one, which is expensive if you have millions of documents.

Coreference and cross-sentence extraction. Most IE approaches work sentence by sentence, yet facts are often spread across multiple sentences. "Apple made the announcement on Tuesday. The deal, which values the company at $20 billion, is expected to close in Q3." Resolving "the deal" and "the company" requires coreference resolution, a separate and difficult NLP task. Full document-level IE that tracks entities and events across paragraphs remains an active research area, with neural coreference resolvers like the SpanBERT-based system of Joshi et al. (2020) achieving strong performance on standard benchmarks but still struggling on long documents and rare entity types. In practice, most production IE systems handle cross-sentence extraction either by processing paragraphs as single units (relying on the LLM's or transformer's long-context capability) or by running a separate coreference resolution step before extraction.

Hallucination in LLM extraction. When prompted to extract a field like "deal value," LLMs sometimes invent a plausible-sounding number if no value is present in the text. This is a critical failure mode in fact extraction. Mitigation strategies include requiring the model to quote the exact text span that supports each extracted value, using temperature 0 for deterministic extraction, and cross-validating against the source text programmatically. Attribution-based extraction, where each extracted value is paired with its source quote, makes hallucination immediately detectable. If the model claims the deal value is "$3 billion" and the quoted span does not contain that value, you have a hallucination. This pattern, extract with attribution and verify the attribution, is becoming a standard practice in production LLM extraction pipelines.

Domain shift. A NER model trained on news performs poorly on clinical notes, legal documents, or social media. Entity types differ, writing styles differ, and even the same words carry different meanings. "Discharge" in a medical context triggers a clinical event; in a news context it may trigger a Personnel:End-Position event. "Parties" in a legal document refers to the signatories of a contract; in news it may refer to political parties or social gatherings. Fine-tuning on domain-specific data is the standard solution, but annotating domain data is expensive. Few-shot prompting with LLMs is often a more practical first step for new domains, and domain-adaptive pretraining, continuing pretraining on domain text before fine-tuning, bridges the gap when labeled data is available.

Evaluation against human annotation. Even human annotators disagree on entity boundaries and relation types. Annotation agreement (measured by inter-annotator agreement, often reported as Cohen's kappa) for complex event extraction tasks can be surprisingly low, on the order of 0.6 to 0.8. Model performance comparisons are only meaningful if both systems are evaluated against the same gold standard, ideally one with high annotation agreement. When agreement is low, it is worth examining whether the disagreements cluster on specific entity types or event roles, which can reveal ambiguities in the annotation guidelines. The practical implication is that reported F1 scores on benchmarks like ACE 2005 or CoNLL-2003 represent performance relative to one set of annotation decisions, not absolute accuracy relative to some objective ground truth. Different annotation teams might produce different gold standards from the same text.

Scalability and cost. For high-volume extraction workloads processing millions of documents per day, the cost difference between fine-tuned sequence taggers and LLM-based extraction becomes significant. A fine-tuned BERT NER model processes thousands of sentences per second on a GPU, while an LLM call to a commercial API costs several cents per call and introduces network latency. The appropriate architecture depends on volume, latency requirements, and the complexity of the extraction task. Very complex, low-volume extraction (e.g., extracting structured records from legal agreements) often justifies LLM costs. High-volume, simpler extraction (e.g., tagging entities in a news stream) often favors fine-tuned models.