Chunking › Evaluating Chunkers
Language AI HandbookAll chapters Chunking › Evaluating ChunkersOne section · Sections in this chapter9 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Chunking evaluation follows the same precision-recall-F1 framework used elsewhere in NLP, but applied at the chunk level rather than the token level. A predicted chunk is correct only if it exactly matches a gold chunk in both its boundaries and its type.
Chunking evaluation counts a predicted chunk as correct only when it exactly matches a gold chunk in both span (start and end positions) and type (NP, VP, PP). A prediction that captures the right words but assigns the wrong type, or gets the right type but with an off-by-one boundary, receives no credit. There is no partial credit.
This strict metric has an important consequence: precision and recall trade off differently than in token-level evaluation. A chunker that consistently overshoots chunk boundaries will have high recall (it covers the gold span) but lower precision (it includes extra tokens). A chunker that only identifies the core noun in every NP will have high precision but low recall.
def evaluate_chunker(chunker_obj, test_trees, max_sents=500):
"""
Evaluate a chunker at the chunk level.
Returns precision, recall, F1, and counts.
"""
tp = fp = fn = 0
for tree in test_trees[:max_sents]:
# Gold chunks: set of (type, word_tuple) pairs
gold = set()
for subtree in tree:
if hasattr(subtree, "label"):
words = tuple(w for w, _ in subtree.leaves())
gold.add((subtree.label(), words))
# Predicted chunks
tagged = list(tree.leaves())
pred_tree = chunker_obj.parse(tagged)
pred = set()
for subtree in pred_tree:
if hasattr(subtree, "label"):
words = tuple(w for w, _ in subtree.leaves())
pred.add((subtree.label(), words))
tp += len(gold & pred)
fp += len(pred - gold)
fn += len(gold - pred)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f1 = (
2 * precision * recall / (precision + recall)
if (precision + recall) > 0
else 0.0
)
return {
"precision": precision,
"recall": recall,
"f1": f1,
"tp": tp,
"fp": fp,
"fn": fn,
}
# Evaluate the regex chunker defined earlier
regex_eval = evaluate_chunker(chunker, test_sents)Regex Chunker Evaluation (on 500 test sentences) ================================================== Precision: 75.75% Recall: 72.29% F1 Score: 73.98% True Positives: 3,699 False Positives: 1,184 False Negatives: 1,418
The F1 score for the regex chunker on CoNLL-2000 typically falls in the 50-70% range, depending on how many constructions the grammar covers. Machine learning approaches push this to 90%+.