Named Entity Recognition › NER Datasets and Benchmarks

Several datasets have become standard reference points for NER system comparison. Understanding their origins, domains, and annotation conventions is essential for interpreting published results.

CoNLL 2003 is the most widely used English NER benchmark. It was released as part of the 2003 Conference on Computational Natural Language Learning shared task and consists of newswire text from the Reuters Corpus, annotated with four entity types: PER, ORG, LOC, and MISC.

The dataset contains approximately 22,000 sentences (roughly 300,000 tokens) split into training, development (testa), and test (testb) sets. The test set F1 score on this dataset is the standard comparison point for NER systems across a decade of research, enabling direct comparisons between systems published years apart.

A critical property of CoNLL 2003 is that its annotation guidelines are relatively conservative: abbreviations and ambiguous cases tend to be left unannotated rather than annotated with uncertain labels. This makes the dataset cleaner but also means systems tuned on it may not generalize well to noisier domains.

An important caveat about CoNLL 2003 scores: subsequent research has shown that the test set contains a significant number of annotation errors, and that near-human performance may already have been reached for this particular benchmark. Models reporting 94+ F1 on CoNLL 2003 may be partially benefiting from overfitting to specific annotation idiosyncrasies. This does not diminish the value of the dataset as a long-term comparison point, but it does suggest that further gains on this specific benchmark may not translate into equivalent gains on real-world NER tasks.

OntoNotes 5.0 is a larger and more linguistically diverse corpus covering newswire, broadcast news, broadcast conversation, web text, magazine articles, and telephone conversations. It uses an 18-class entity taxonomy (the same one underlying spaCy's default model), including CARDINAL, ORDINAL, QUANTITY, and LAW in addition to the standard PER/ORG/LOC/DATE/MONEY/PERCENT types.

Because OntoNotes includes spoken language transcripts and web text, it is harder and more representative of real-world variation than CoNLL 2003. State-of-the-art F1 on OntoNotes is roughly 5-8 points lower than on CoNLL 2003 for equivalent-sized models.

The Automatic Content Extraction (ACE) datasets were developed by NIST for the information extraction community. They annotate entities, relations, and events, making them useful for systems that need to extract structured knowledge rather than just entity mentions. ACE uses a nested entity annotation scheme, capturing both inner and outer entity spans.

The biomedical NLP community has developed specialized datasets for gene/protein NER (BioCreative II Gene Mention task), disease NER (NCBI Disease corpus), and chemical/drug NER (BC5CDR corpus). These datasets use expert annotators (biologists, pharmacists) and are substantially harder than general-domain NER because entity boundaries in biomedical text are often ambiguous even to human experts.

NER system performance on CoNLL 2003 has improved dramatically over time, driven by architectural advances. Here is how major model families compare:

The jump from CRF to BiLSTM-CRF marks the move to neural representations: instead of hand-engineered features (capitalization, prefix/suffix patterns, gazetteers), the model learns distributed representations of words and their contexts. The jump from BiLSTM-CRF to BERT marks the move to pre-trained language model representations, which encode much richer contextual information about entity mentions.

Not all entity types are equally easy. Let's look at a simulated breakdown to understand the pattern: