Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Named Entity Recognition › CRF-Based NER: A Working Implementation

Michael BrenndoerferUpdated January 30, 20263 min read

Language AI HandbookAll chapters

Part 6 of 67Sequence Labeling

Chapter 37Named Entity Recognition › CRF-Based NER: A Working Implementation

Reading section 6 of 17 · CRF-Based NER: A Working Implementation · 3 minSection 6 of 17 · 3 min

Section 6 of 17 · 3 min read

Named Entity Recognition › CRF-Based NER: A Working ImplementationOne section · Sections in this chapter6 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's build a complete CRF-based NER pipeline to understand the mechanics. We will use the sklearn-crfsuite package, which provides a clean Python interface to CRF training.

In[14]:

Code

import subprocess
import sys

# Install sklearn-crfsuite if needed
try:
    import sklearn_crfsuite
except ImportError:
    subprocess.run(
        [sys.executable, "-m", "pip", "install", "sklearn-crfsuite", "-q"],
        check=True,
    )

In[15]:

Code

def sent_to_features(sent):
    """
    Convert a sentence (list of (word, pos, chunk, ner) tuples)
    to a list of feature dicts, one per token.
    """
    return [word_features_for_crf(sent, i) for i in range(len(sent))]


def word_features_for_crf(sent, i):
    word = sent[i][0]
    postag = sent[i][1]

    features = {
        "bias": 1.0,
        "word.lower()": word.lower(),
        "word[-3:]": word[-3:],
        "word[-2:]": word[-2:],
        "word.isupper()": word.isupper(),
        "word.istitle()": word.istitle(),
        "word.isdigit()": word.isdigit(),
        "postag": postag,
        "postag[:2]": postag[:2],
    }
    if i > 0:
        word1 = sent[i - 1][0]
        postag1 = sent[i - 1][1]
        features.update(
            {
                "-1:word.lower()": word1.lower(),
                "-1:word.istitle()": word1.istitle(),
                "-1:word.isupper()": word1.isupper(),
                "-1:postag": postag1,
                "-1:postag[:2]": postag1[:2],
            }
        )
    else:
        features["BOS"] = True

    if i < len(sent) - 1:
        word1 = sent[i + 1][0]
        postag1 = sent[i + 1][1]
        features.update(
            {
                "+1:word.lower()": word1.lower(),
                "+1:word.istitle()": word1.istitle(),
                "+1:word.isupper()": word1.isupper(),
                "+1:postag": postag1,
                "+1:postag[:2]": postag1[:2],
            }
        )
    else:
        features["EOS"] = True

    return features


def sent_to_labels(sent):
    return [label for _, _, label in sent]

In[16]:

Code

# Small synthetic training set:
# Each sentence is a list of (word, pos_tag, ner_label)
train_sents = [
    [
        ("Apple", "NNP", "B-ORG"),
        ("is", "VBZ", "O"),
        ("based", "VBN", "O"),
        ("in", "IN", "O"),
        ("Cupertino", "NNP", "B-LOC"),
        (".", ".", "O"),
    ],
    [
        ("Tim", "NNP", "B-PER"),
        ("Cook", "NNP", "I-PER"),
        ("leads", "VBZ", "O"),
        ("Apple", "NNP", "B-ORG"),
        (".", ".", "O"),
    ],
    [
        ("Google", "NNP", "B-ORG"),
        ("announced", "VBD", "O"),
        ("new", "JJ", "O"),
        ("offices", "NNS", "O"),
        ("in", "IN", "O"),
        ("London", "NNP", "B-LOC"),
        (".", ".", "O"),
    ],
    [
        ("Satya", "NNP", "B-PER"),
        ("Nadella", "NNP", "I-PER"),
        ("runs", "VBZ", "O"),
        ("Microsoft", "NNP", "B-ORG"),
        (".", ".", "O"),
    ],
    [
        ("Paris", "NNP", "B-LOC"),
        ("is", "VBZ", "O"),
        ("the", "DT", "O"),
        ("capital", "NN", "O"),
        ("of", "IN", "O"),
        ("France", "NNP", "B-LOC"),
        (".", ".", "O"),
    ],
]

X_train = [sent_to_features(s) for s in train_sents]
y_train = [sent_to_labels(s) for s in train_sents]

# Train CRF
crf = CRF(
    algorithm="lbfgs",
    c1=0.1,
    c2=0.1,
    max_iterations=100,
    all_possible_transitions=True,
)
crf.fit(X_train, y_train)

In[17]:

Code

# Test on a new sentence
test_sent = [
    ("Amazon", "NNP"),
    ("CEO", "NN"),
    ("Andy", "NNP"),
    ("Jassy", "NNP"),
    ("visited", "VBD"),
    ("Berlin", "NNP"),
    (".", "."),
]


def predict_ner(model, sent_with_pos):
    """Predict NER labels for a sentence given as (word, pos) pairs."""
    # Temporarily package as (word, pos, dummy_label)
    packaged = [(w, p, "O") for w, p in sent_with_pos]
    feats = sent_to_features(packaged)
    return model.predict([feats])[0]


predicted_labels = predict_ner(crf, test_sent)

Out[18]:

Console

CRF NER predictions on new sentence:

Token        POS      Predicted Label
-----------------------------------
Amazon       NNP      B-ORG
CEO          NN       O
Andy         NNP      O
Jassy        NNP      O
visited      VBD      O
Berlin       NNP      B-LOC
.            .        O

The CRF correctly identifies "Andy Jassy" as a PER entity (relying on capitalization and the context of a preceding ORG), "Amazon" as an ORG, and "Berlin" as a LOC. Despite the extremely small training set, the model generalizes because the features (capitalization, POS tags, context words) generalize across entity types.

One of the most useful diagnostic tools for CRF models is inspecting the learned transition weights. High transition weights for B-PER to I-PER confirm the model learned that person names commonly have multiple tokens. High negative weights for O to I-PER confirm the model learned that an inside tag cannot appear without a preceding begin tag.