Named Entity Recognition › A Brief History of NER Systems

NER has a rich history spanning four decades and three major paradigm shifts. Understanding this history clarifies why modern systems are designed the way they are and what problems each generation of approaches was solving.

The earliest NER systems were built from hand-written rules and gazetteers. A gazetteer is a list of known entity names, for instance a list of country names, a list of major corporations, or a list of common first and last names. Rules then check whether a token appears in one of these lists, or whether contextual patterns like "Mr. [WORD]" or "[WORD] Inc." indicate an entity.

Rule-based systems have real advantages: they are interpretable, they never hallucinate, and they can be updated by simply editing the rules or adding entries to gazetteers. They remain useful in domains where the entity vocabulary is small and stable, such as a financial system that only needs to recognize a fixed list of stock tickers. Early systems like Proper Noun Analyzer (PROLEX) and FASTUS (1995) achieved surprisingly good performance on newswire NER using these techniques.

Their fundamental weakness is coverage and maintainability. Languages are productive: new entities are created constantly (new companies, new public figures, new events), and hand-written rules cannot keep pace. Rules also interact in complex ways as the rule set grows, making maintenance error-prone. By the late 1990s, the community was actively exploring statistical alternatives.

The first statistical NER systems used Hidden Markov Models (HMMs). An HMM models the joint probability of a token sequence and its label sequence using emission probabilities (how likely is this word given this label?) and transition probabilities (how likely is this label given the previous label?). Training requires only annotated data, which removes the need for manual rule writing. We covered HMMs in detail in the previous chapter on sequence labeling.

HMMs have a significant limitation for NER: they rely on each token being independent given the current label. In practice, NER decisions depend on rich context: whether a word is capitalized, whether it is preceded by a title, whether it appears in a list of known organizations, and dozens of other features. Capturing these features within an HMM requires increasingly complex emission probability models.

Conditional Random Fields (CRFs), introduced by Lafferty, McCallum, and Pereira in 2001, addressed this limitation directly. A linear-chain CRF models the conditional distribution $P(\mathbf{y} | \mathbf{x})$ directly rather than the joint distribution, which means it can condition on arbitrary features of the entire input sequence without needing to model the input distribution. A typical CRF feature function for NER might fire when the current token is capitalized and the previous label was B-PER, assigning that configuration a learned weight.

The standard feature set for CRF-based NER includes:

• Word identity: the exact lowercased word form
• Word shape: capitalization pattern (Xxxx for Title Case, XXXX for all caps, xxxx for all lower)
• Prefix and suffix patterns: the first and last 2-4 characters
• Gazetteer membership: whether the word appears in a person/organization/location list
• Part-of-speech tag: the syntactic category
• Preceding and following words: context window of 1-2 tokens on each side
• Preceding and following labels: the predicted labels for surrounding tokens

CRFs with these features achieved state-of-the-art NER performance through most of the 2000s, reaching around 86-89 F1 on CoNLL 2003. The primary bottleneck was feature engineering: designing the right feature templates required substantial domain expertise and tuition through experimentation.

Let's demonstrate what a simple feature function for CRF-style NER looks like:

Each token generates a dictionary of binary and string-valued features. The CRF then learns a weight for each (feature, label) combination. At inference time, the Viterbi algorithm finds the label sequence with the highest total weighted score, taking into account both feature weights and label transition weights.

The shift from hand-engineered features to learned representations began with word embeddings. Instead of encoding "does this word appear in a gazetteer?", neural NER systems encode each word as a dense vector that captures its distributional meaning. This frees the model from the need for manual feature engineering and allows it to generalize to unseen words based on semantic similarity to known words.

The architecture that defined this era was the BiLSTM-CRF, introduced by Lample et al. (2016) and Chiu and Nichols (2016). The model has three components working together:

First, a character-level convolutional network or BiLSTM reads each word's characters and produces a character-level embedding. This component captures morphological patterns, such as recognizing that words ending in "-son" (Dickson, Jackson, Peterson) tend to be surnames, or that fully capitalized words are often abbreviations. This matters enormously for NER because named entities frequently contain unusual words that are out of vocabulary for word-level models.

Second, the character embedding is concatenated with a pre-trained word embedding (GloVe or fastText) to form a combined token representation. The combined representation captures both morphological patterns and semantic associations.

Third, a bidirectional LSTM processes the sequence of token representations, producing a context-sensitive encoding for each token. At each position, the BiLSTM combines information from the left context (how has the text built up to this point?) and the right context (what comes after this token?). This bidirectional view is essential for NER: recognizing "Times" as part of an organization name requires knowing it follows "New York", and recognizing "Bank" as part of "Bank of America" requires knowing what follows it.

The BiLSTM outputs feed into a linear layer that produces logit scores for each possible BIO tag. These scores feed into a CRF layer that applies the Viterbi algorithm during decoding. The CRF layer enforces label transition constraints, such as the rule that an I-PER tag cannot immediately follow an O tag, which improves output consistency compared to independent token-level classification.

BiLSTM-CRF systems achieved around 90-91 F1 on CoNLL 2003, an improvement of 4-5 points over classical CRFs. The gain came almost entirely from the learned representations replacing hand-engineered features. The core sequence labeling logic, including the CRF decoding, remained the same.

The introduction of BERT in 2018 created the current dominant paradigm for NER. Rather than training task-specific representations from scratch, BERT provides deeply contextual word representations pre-trained on large text corpora. Fine-tuning BERT for NER requires only adding a linear classification head on top of the token representations and training on labeled NER data.

The key advance over BiLSTM models is the quality of the representations. BERT uses self-attention (covered in detail in later chapters) to produce representations that integrate long-range context much more effectively than LSTMs. For NER, this means BERT can use cues from many positions away to determine entity type and boundaries. A sentence like "He said it was the Microsoft founder's decision" requires integrating information about the subject, the verb, and the noun phrase to correctly identify "Microsoft" as an ORG entity in that context.

With BERT-base fine-tuned on CoNLL 2003, NER performance jumped to around 92 F1. Larger models (BERT-large, RoBERTa-large) reached 93-94 F1. These numbers had seemed nearly impossible a few years earlier.

The transformer architecture also changed how practitioners think about NER for new domains. With classical CRFs or BiLSTM-CRFs, adapting to a new domain required substantial annotated training data to learn new feature weights or BiLSTM representations. With pre-trained transformers, the representations already encode rich world knowledge and linguistic patterns from large corpora. Fine-tuning on even a few hundred labeled examples in a new domain can produce a usable system, because the model is not learning representations from scratch.