CRF Training › Feature Template Design
Language AI HandbookAll chapters CRF Training › Feature Template DesignOne section · Sections in this chapter6 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
CRF features are the heart of the model. Well-designed features capture the patterns that distinguish correct label sequences from incorrect ones. Feature templates define how to extract features from the input, and the CRF learns which features matter most.
A feature template is a pattern that generates binary features from observations. For example, the template "current word = X" generates one feature for each unique word X in the vocabulary. Templates let you define feature types; the actual features are instantiated from training data.
The distinction between a feature template and a feature instance matters. A template like "current word = X" is a recipe that generates thousands of concrete features, one for each vocabulary word. You define the template once; the feature extraction code instantiates all the concrete features automatically when it encounters each word in the training data. This design is elegant because it allows the feature set to scale with the vocabulary without requiring you to enumerate every possible feature in advance.
Feature design for CRFs is both an art and an empirical practice. The goal is to encode enough signal that the CRF can distinguish between correct and incorrect label assignments, without being so redundant or noisy that the model overfits to training artifacts. A useful mental model is to think about what a linguistically informed human annotator would notice when deciding on a label. If a feature captures something that a human would use as evidence, it is probably worth including. If it captures something accidental about the training data, it will hurt generalization.
Common Feature Types
Features for sequence labeling typically fall into several categories:
- Lexical features: The word itself, lowercase form, word shape (capitalization pattern)
- Contextual features: Surrounding words in a window
- Morphological features: Prefixes, suffixes, presence of digits or punctuation
- Part-of-speech features: POS tags from a tagger (if available)
- Transition features: Combinations of previous and current labels
Each category captures a different kind of evidence. Lexical features are the most direct: the word "London" is almost always a location. But lexical features are also the most data-hungry: to learn that "London" is a location, the model needs to see it in training data. Morphological features generalize better because they fire for unseen words: a word ending in "-shire" is probably a British county even if the specific word never appeared in training. Context features capture relational information: a capitalized word immediately after "President" is likely a person's name.
The most powerful features are often combinations: "the previous word is 'Dr.' and the current word is capitalized" is a much stronger signal for a person name than either feature alone. CRFs can represent such combinations as conjunctive features, and many feature engineering libraries automatically generate pairwise combinations of base features to capture these interactions.
def extract_features(tokens, position):
"""
Extract features for a token at a given position.
Returns a dictionary of feature_name: value pairs.
Features are designed for NER tagging.
"""
features = {}
word = tokens[position]
# Current word features
features["word.lower"] = word.lower()
features["word.isupper"] = word.isupper()
features["word.istitle"] = word.istitle()
features["word.isdigit"] = word.isdigit()
# Word shape
shape = ""
for char in word:
if char.isupper():
shape += "X"
elif char.islower():
shape += "x"
elif char.isdigit():
shape += "d"
else:
shape += char
features["word.shape"] = shape
# Prefix/suffix
features["word.prefix2"] = (
word[:2].lower() if len(word) >= 2 else word.lower()
)
features["word.suffix2"] = (
word[-2:].lower() if len(word) >= 2 else word.lower()
)
features["word.prefix3"] = (
word[:3].lower() if len(word) >= 3 else word.lower()
)
features["word.suffix3"] = (
word[-3:].lower() if len(word) >= 3 else word.lower()
)
# Context features
if position > 0:
prev_word = tokens[position - 1]
features["prev.word.lower"] = prev_word.lower()
features["prev.word.istitle"] = prev_word.istitle()
else:
features["BOS"] = True # Beginning of sentence
if position < len(tokens) - 1:
next_word = tokens[position + 1]
features["next.word.lower"] = next_word.lower()
features["next.word.istitle"] = next_word.istitle()
else:
features["EOS"] = True # End of sentence
return features# Example: extract features for a sample sentence
sample_sentence = [
"John",
"Smith",
"works",
"at",
"Google",
"in",
"California",
".",
]
sample_labels = ["B-PER", "I-PER", "O", "O", "B-ORG", "O", "B-LOC", "O"]
all_features = []
for i in range(len(sample_sentence)):
features = extract_features(sample_sentence, i)
all_features.append(features)Feature Extraction Example ============================================================ Sentence: John Smith works at Google in California . Labels: ['B-PER', 'I-PER', 'O', 'O', 'B-ORG', 'O', 'B-LOC', 'O'] Features for selected tokens: ------------------------------------------------------------ Token: 'John' (Label: B-PER) BOS: True next.word.istitle: True next.word.lower: smith word.isdigit: False word.istitle: True word.isupper: False word.lower: john word.prefix2: jo word.prefix3: joh word.shape: Xxxx word.suffix2: hn word.suffix3: ohn Token: 'Google' (Label: B-ORG) next.word.istitle: False next.word.lower: in prev.word.istitle: False prev.word.lower: at word.isdigit: False word.istitle: True word.isupper: False word.lower: google word.prefix2: go word.prefix3: goo word.shape: Xxxxxx word.suffix2: le word.suffix3: gle Token: 'California' (Label: B-LOC) next.word.istitle: False next.word.lower: . prev.word.istitle: False prev.word.lower: in word.isdigit: False word.istitle: True word.isupper: False word.lower: california word.prefix2: ca word.prefix3: cal word.shape: Xxxxxxxxxx word.suffix2: ia word.suffix3: nia
The feature extractor captures patterns that are predictive of entity labels. Title-cased words are often names. Words ending in "-ia" might be locations. Words preceded by "at" might be organizations. The CRF learns which of these patterns are reliable predictors.
One thing to notice about the feature extractor above is that it uses the raw word itself as one of the features. This creates one binary feature per unique word in the vocabulary, and those features will have zero weight for any word not seen in training. This is the "unknown word" problem, and it is one of the fundamental challenges in classical NLP. Morphological features like prefixes and suffixes partially address it: even an unseen word shares its suffix with many seen words, allowing the model to make reasonable guesses. Treating unknown words as a special token is another common approach, at the cost of losing information about what the unknown word looks like.
Feature Templates in Practice
Real NER systems use extensive feature templates. The richness of the feature set often matters more than the choice of optimizer or regularization strength. McCallum and Li's influential 2003 paper on CRF-based NER used over 100,000 features on the CoNLL 2003 benchmark, carefully designed to capture capitalization, word shapes, gazetteers of known names, and contextual patterns from several words before and after each token. At the time, this was state of the art for English NER.
The sklearn-crfsuite library provides a convenient way to train CRFs with custom features:
# Install sklearn-crfsuite for CRF training
# uv pip install sklearn-crfsuitedef prepare_sentence_features(tokens, labels):
"""Convert a sentence to CRF training format."""
sentence_features = []
for i in range(len(tokens)):
features = extract_features(tokens, i)
# Convert to string features for sklearn-crfsuite
str_features = {k: str(v) for k, v in features.items()}
sentence_features.append(str_features)
return sentence_features, labels
# Create a small dataset for demonstration
sentences = [
(
["John", "Smith", "works", "at", "Google", "."],
["B-PER", "I-PER", "O", "O", "B-ORG", "O"],
),
(
["Mary", "visited", "Paris", "last", "summer", "."],
["B-PER", "O", "B-LOC", "O", "O", "O"],
),
(
["Apple", "announced", "a", "new", "iPhone", "."],
["B-ORG", "O", "O", "O", "O", "O"],
),
(
["Dr.", "Jones", "from", "MIT", "gave", "a", "talk", "."],
["O", "B-PER", "O", "B-ORG", "O", "O", "O", "O"],
),
(
["The", "Eiffel", "Tower", "is", "in", "France", "."],
["O", "B-LOC", "I-LOC", "O", "O", "B-LOC", "O"],
),
]
# Prepare training data
X_train = []
y_train = []
for tokens, labels in sentences:
features, lbls = prepare_sentence_features(tokens, labels)
X_train.append(features)
y_train.append(lbls)import sklearn_crfsuite
# Train CRF model
crf = sklearn_crfsuite.CRF(
algorithm="lbfgs",
c1=0.1, # L1 regularization
c2=0.1, # L2 regularization
max_iterations=100,
all_possible_transitions=True,
)
crf.fit(X_train, y_train)CRF Training Complete ================================================== Number of features: 122 Number of labels: 6 Labels: ['B-PER', 'I-PER', 'O', 'B-ORG', 'B-LOC', 'I-LOC'] Top positive features by label: -------------------------------------------------- B-PER: word.prefix2:jo: +1.0393 word.shape:Xxxx: +0.9872 word.lower:jones: +0.3844 I-PER: word.lower:smith: +0.4325 word.prefix2:sm: +0.4325 word.suffix2:th: +0.4325 O: word.istitle:False: +2.4189 word.isupper:False: +1.5046 word.isdigit:False: +0.7216 B-ORG: word.suffix2:le: +1.2615 word.lower:mit: +0.4305 word.isupper:True: +0.4305 B-LOC: word.shape:Xxxxxx: +0.6492 word.lower:paris: +0.3769 word.prefix2:pa: +0.3769 I-LOC: word.lower:tower: +0.4375 word.prefix2:to: +0.4375 word.prefix3:tow: +0.4375
The learned feature weights reveal what the CRF considers important. High positive weights for "word.istitle=True" on B-PER indicate that title-cased words are strong signals for person names. The CRF automatically discovers these patterns from the training data.
Inspecting feature weights is one of the most valuable diagnostic tools for CRF practitioners. Unlike neural networks, where the learned representations are distributed and opaque, CRF weights are directly interpretable: each weight has a straightforward meaning as the log-odds contribution of that feature to that label. A large positive weight for "word[-3:]=ian" on I-PER means the model has learned that words ending in "-ian" (like "Iranian" or "Brazilian") often appear inside person entity spans. A large negative weight for the same feature on B-ORG means such words are unlikely to start organization spans. Debugging a CRF that makes systematic errors often starts with examining the weights of features that seem relevant to the error type.