BIO Tagging › Annotation Guidelines and Span Definition Challenges
Language AI HandbookAll chapters BIO Tagging › Annotation Guidelines and Span Definition ChallengesOne section · Sections in this chapter11 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BIO tags are only as consistent as the annotation guidelines that define what counts as an entity and where its boundaries lie. Ambiguous guidelines produce inconsistent tags, which in turn produce models that learn unreliable boundary conventions.
Several boundary ambiguities recur across NER projects. One common case involves possessive suffixes: should "IBM's" be tagged as B-ORG and the apostrophe-s excluded, or should the full string "IBM's" be the entity? Different datasets make different choices, and a model trained on one convention will have lower precision or recall when evaluated on data annotated with a different one. Tokenization choices interact here: if the tokenizer splits "IBM's" into ["IBM", "'s"], the question is whether "'s" gets I-ORG or O.
Another frequent ambiguity involves titles and honorifics. In "President Biden visited Paris," should "President Biden" be the person entity (including the title) or just "Biden"? The CoNLL-2003 English dataset typically excludes titles, but other datasets include them. Models trained on CoNLL-2003 data will therefore systematically miss the title token when applied to data where titles are included in the span.
Prepositional attachment creates similar challenges. "Bank of America" is clearly a single ORG entity. But "University of California at Berkeley" might be annotated as one span or as "University of California" with "Berkeley" as a separate LOC, depending on how the annotation guidelines treat prepositional phrases. Fine-grained guidelines that address these cases explicitly produce more consistent training data and better-calibrated evaluation results.
# Illustrate span boundary ambiguity
ambiguous_cases = [
{
"text": ["President", "Biden", "visited", "Paris"],
"narrow_spans": [(1, 2, "PER"), (3, 4, "LOC")], # exclude title
"broad_spans": [(0, 2, "PER"), (3, 4, "LOC")], # include title
},
{
"text": ["IBM", "'s", "revenue", "grew"],
"narrow_spans": [(0, 1, "ORG")], # exclude possessive
"broad_spans": [(0, 2, "ORG")], # include possessive
},
{
"text": ["University", "of", "California", "at", "Berkeley"],
"single_span": [(0, 5, "ORG")], # whole phrase is one ORG
"split_spans": [(0, 3, "ORG"), (4, 5, "LOC")], # split at "Berkeley"
},
]
for case in ambiguous_cases:
tokens = case["text"]
for scheme_name, spans in list(case.items())[1:]:
tags = spans_to_bio(tokens, spans)
print(f"{scheme_name}: {' '.join(tags)}")
print()Span Boundary Ambiguities in Annotation: ============================================================ Sentence: President Biden visited Paris narrow_spans : O B-PER O B-LOC broad_spans : B-PER I-PER O B-LOC Sentence: IBM 's revenue grew narrow_spans : B-ORG O O O broad_spans : B-ORG I-ORG O O Sentence: University of California at Berkeley single_span : B-ORG I-ORG I-ORG I-ORG I-ORG split_spans : B-ORG I-ORG I-ORG O B-LOC
These ambiguity examples are not edge cases to be dismissed. They represent systematic differences that produce measurably different model behavior. A model trained on narrow-span annotations for person names (excluding titles) will have systematically lower recall when applied to text where broad-span conventions are expected. Annotation consistency within a dataset is at least as important as annotation quality at any single example.