Named Entity Recognition › Limitations and Practical Considerations

NER systems are useful, but they come with important limitations that practitioners must understand before deploying them.

Domain shift is the dominant failure mode. A model trained on Reuters newswire will perform well on similar financial and political news, but will degrade substantially on social media posts, clinical notes, legal contracts, or scientific papers. The vocabulary of entity mentions, the syntactic patterns surrounding them, and even the entity type taxonomy may all differ. Before deploying a general-purpose NER model in a specialized domain, you should evaluate it on in-domain annotated examples. The gap between out-of-domain and in-domain performance is often 15-25 F1 points.

Rare and emerging entities are systematically missed. NER models, even transformer-based ones, rely heavily on the memorized associations between surface forms and entity types learned during pretraining. A new company that did not exist during pretraining, a recently coined term, or a person with an unusual name will be missed or mislabeled more often than familiar entities. This is particularly problematic for news applications where the most newsworthy entities are often the least familiar ones.

Annotation inconsistencies propagate to system outputs. Many NER benchmarks contain annotation disagreements, especially for borderline cases (is "the Internet" a MISC entity or not?). Systems trained on inconsistently annotated data learn the pattern of inconsistency, not a principled definition of entity types. When two annotators would disagree on a case, the model's prediction on that case is essentially random.

Evaluation scores can be misleading. Exact match F1 on CoNLL 2003 is a useful benchmark for comparing research systems, but it correlates imperfectly with downstream task performance. A question-answering system that uses NER as a preprocessing step may tolerate boundary errors well but be sensitive to type errors. Reporting system performance using the CoNLL F1 benchmark without also evaluating on the target domain is a common source of overoptimism.

Computational cost of large models. While transformer-based NER achieves the best accuracy, it requires substantially more computation than CRF-based systems. A BERT-base NER model processes roughly 50-100 sentences per second on a modern CPU, while a CRF can process thousands. For high-throughput applications (processing millions of documents daily), the computational cost of transformer-based NER is a real constraint, and practitioners often use lighter models (DistilBERT, smaller spaCy models) with acceptable accuracy tradeoffs.

Despite these limitations, NER has proven enormously useful in practice. It enables structured information extraction at scale: monitoring news for mentions of specific companies, extracting clinical concepts from medical records, building knowledge graphs from text corpora, and improving search engine query understanding. Transformer-based NER models are accurate enough for many production applications, and the field continues to develop better approaches to few-shot learning, cross-domain generalization, and nested entity recognition.