Hidden Markov Models › Evaluating the Tagger
Language AI HandbookAll chapters Hidden Markov Models › Evaluating the TaggerOne section · Sections in this chapter8 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
How well does our HMM tagger perform? To answer this rigorously, we need to evaluate on data the model hasn't seen during training. We'll split the Brown corpus into 90% training and 10% test, re-estimate all parameters on the training portion, and measure accuracy on the held-out test sentences.
We'll also track performance separately for known words (those seen during training) versus unknown words. This distinction matters because the model handles these cases very differently: known words have learned emission probabilities, while unknown words must rely entirely on transition patterns and smoothed probabilities.
from sklearn.model_selection import train_test_split
# Split data into train and test
train_sents, test_sents = train_test_split(
tagged_sents, test_size=0.1, random_state=42
)
# Re-estimate parameters on training data only
pi_train, A_train, B_train, _ = estimate_hmm_parameters(train_sents)
B_train_smoothed, vocab_train = estimate_smoothed_emissions(train_sents)
# Create tagger with training parameters
test_tagger = SimpleHMMTagger(pi_train, A_train, B_train_smoothed, vocab_train)def evaluate_tagger(tagger, test_sentences):
"""Evaluate tagger accuracy on test sentences."""
correct = 0
total = 0
known_correct = 0
known_total = 0
unknown_correct = 0
unknown_total = 0
for sentence in test_sentences:
words = [word for word, tag in sentence]
gold_tags = [tag for word, tag in sentence]
pred_tags = tagger.tag_greedy(words)
for word, gold, pred in zip(words, gold_tags, pred_tags):
total += 1
is_known = word.lower() in tagger.vocab
if gold == pred:
correct += 1
if is_known:
known_correct += 1
else:
unknown_correct += 1
if is_known:
known_total += 1
else:
unknown_total += 1
return {
"overall": correct / total if total > 0 else 0,
"known": known_correct / known_total if known_total > 0 else 0,
"unknown": unknown_correct / unknown_total if unknown_total > 0 else 0,
"unknown_rate": unknown_total / total if total > 0 else 0,
}
results = evaluate_tagger(test_tagger, test_sents)Training sentences: 51,606 Test sentences: 5,734 Unknown word rate: 2.00% | Metric | Accuracy | |:-------|--------:| | Overall | 92.5% | | Known words | 93.8% | | Unknown words | 25.7% |
: HMM tagger accuracy on the Brown corpus test set. The model achieves high accuracy on known words where emission probabilities provide strong signals. Unknown word accuracy drops substantially because the model must rely solely on transition probabilities and smoothed emissions. {#tbl-accuracy}
The accuracy gap between known and unknown words is striking. For known words, the model performs reasonably well because emission probabilities provide strong signals. For unknown words, the model must rely entirely on transition probabilities, which provide weaker disambiguation.
This evaluation uses greedy decoding, which makes locally optimal choices. The Viterbi algorithm (next chapter) finds the globally optimal sequence and typically improves accuracy by 2-5 percentage points, especially for difficult cases where local and global optima differ.
The accuracy numbers here may seem lower than expected for a model praised as a major advance. Remember that we are evaluating with the Universal tagset, which has only 12 coarse categories. With fewer categories, random chance already accounts for a reasonable baseline. HMM taggers evaluated on the Penn Treebank's 36-tag set regularly exceeded 95% overall accuracy, a figure that remained the benchmark for over a decade. The gap between known-word and unknown-word accuracy also motivates the morphological suffix features mentioned earlier: with better handling of unknowns, overall performance rises considerably.
To better understand where the model's disambiguation power comes from, let's visualize the emission probability profiles for several ambiguous words. The degree to which these profiles differ across tags is what determines how much the emission matrix helps versus hinders disambiguation.
The log-scale y-axis is necessary here because emission probabilities span several orders of magnitude. A word that appears frequently as a NOUN will have emission probability on the order of , while the same word appearing rarely as a VERB might have probability on the order of . Both are small in absolute terms, but the ratio between them is what the model exploits for disambiguation. When the bars for NOUN and VERB are nearly the same height (indicating similar emission probabilities from both tags), the model must rely on the transition matrix to pick the right tag.