BIO Tagging › Converting Spans to BIO Tags
Language AI HandbookAll chapters BIO Tagging › Converting Spans to BIO TagsOne section · Sections in this chapter4 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Annotation tools often store entity information as character or token spans rather than per-token labels. Converting these span annotations to BIO format is a common preprocessing step. Let.s build a converter that handles common edge cases.
The conversion logic is straightforward: initialize all tokens as O, then for each span, mark the first token with B and all subsequent tokens with I.
def spans_to_bio(tokens, spans):
"""
Convert span annotations to BIO tags.
Args:
tokens: List of tokens
spans: List of (start_idx, end_idx, entity_type) tuples.
Indices refer to token positions. end_idx is exclusive.
Returns:
List of BIO tags, one per token.
"""
# Initialize all tokens as Outside
tags = ["O"] * len(tokens)
# Sort by start position to handle overlaps deterministically
sorted_spans = sorted(spans, key=lambda x: x[0])
for start, end, entity_type in sorted_spans:
# Skip invalid spans
if start < 0 or end > len(tokens) or start >= end:
continue
# First token gets B prefix
tags[start] = f"B-{entity_type}"
# Remaining tokens get I prefix
for i in range(start + 1, end):
tags[i] = f"I-{entity_type}"
return tags
# Test the converter
test_tokens = ["John", "Smith", "works", "at", "Google", "Inc", "."]
test_spans = [
(0, 2, "PER"), # John Smith
(4, 6, "ORG"), # Google Inc
]
bio_result = spans_to_bio(test_tokens, test_spans)Span to BIO Conversion: --------------------------------------------- Token BIO Tag --------------------------------------------- John B-PER Smith I-PER works O at O Google B-ORG Inc I-ORG . O Input spans: [0:2] 'John Smith' -> PER [4:6] 'Google Inc' -> ORG
The converter handles the common case cleanly. But real-world data presents edge cases: single-token entities, spans at sentence boundaries, and adjacent same-type entities.
In practice, reliability here requires handling inconsistent inputs. Annotation pipelines receive data from many sources: manual annotators using web-based tools, automatic pre-annotation systems that flag candidate spans, and programmatic extraction from databases or knowledge graphs. Each source may represent spans slightly differently, including off-by-one errors in indices, character offsets rather than token offsets, or overlapping spans when two annotators tagged the same region. The converter must either handle or explicitly reject these inputs, and the choice between handling and rejecting depends on whether downstream consumers can tolerate recovered approximations.
# Edge case tests
edge_cases = [
{
"name": "Single-token entity",
"tokens": ["Paris", "is", "lovely"],
"spans": [(0, 1, "LOC")],
},
{
"name": "Entity at sentence end",
"tokens": ["Visit", "New", "York"],
"spans": [(1, 3, "LOC")],
},
{
"name": "Adjacent same-type entities",
"tokens": ["Obama", "Biden", "met"],
"spans": [(0, 1, "PER"), (1, 2, "PER")],
},
{
"name": "All tokens form one entity",
"tokens": ["Barack", "Obama"],
"spans": [(0, 2, "PER")],
},
]
edge_results = []
for case in edge_cases:
tags = spans_to_bio(case["tokens"], case["spans"])
edge_results.append(
{
"name": case["name"],
"tokens": case["tokens"],
"tags": tags,
}
)Edge Case Handling: ======================================================= Single-token entity: Paris -> B-LOC is -> O lovely -> O Entity at sentence end: Visit -> O New -> B-LOC York -> I-LOC Adjacent same-type entities: Obama -> B-PER Biden -> B-PER met -> O All tokens form one entity: Barack -> B-PER Obama -> I-PER
Single-token entities receive only a B tag since there is no continuation token. Adjacent same-type entities each start with B, correctly distinguishing them. Entity spans at sequence boundaries are handled without special-casing since the index bounds check catches out-of-range spans.
The adjacent same-type case is particularly important to verify. The converter sorts spans by start position and processes them in order, so when "Obama" (span 0-1) and "Biden" (span 1-2) are both PER, the first loop iteration sets tags[0] = "B-PER", and the second sets tags[1] = "B-PER". The result is two separate single-token entities rather than a merged two-token entity, which is exactly correct. This deterministic behavior relies on the span representation distinguishing between (0, 1) and (1, 2), which is why exclusive end indices are the right convention.
BIOES Conversion
For applications requiring BIOES format, we extend the converter to track entity boundaries and assign the appropriate suffix:
def spans_to_bioes(tokens, spans):
"""
Convert span annotations to BIOES tags.
Single-token entities get S tag.
Multi-token entities get B...I...E pattern.
"""
tags = ["O"] * len(tokens)
sorted_spans = sorted(spans, key=lambda x: x[0])
for start, end, entity_type in sorted_spans:
if start < 0 or end > len(tokens) or start >= end:
continue
span_length = end - start
if span_length == 1:
# Single-token entity uses S prefix
tags[start] = f"S-{entity_type}"
else:
# Multi-token entity: B at start, I in middle, E at end
tags[start] = f"B-{entity_type}"
for i in range(start + 1, end - 1):
tags[i] = f"I-{entity_type}"
tags[end - 1] = f"E-{entity_type}"
return tags
# Compare BIO vs BIOES on the same span
comparison_tokens = ["The", "New", "York", "Times", "reported"]
comparison_spans = [(1, 4, "ORG")] # New York Times
bio_output = spans_to_bio(comparison_tokens, comparison_spans)
bioes_output = spans_to_bioes(comparison_tokens, comparison_spans)BIO vs BIOES Comparison: -------------------------------------------------- Token BIO BIOES -------------------------------------------------- The O O New B-ORG B-ORG York I-ORG I-ORG Times I-ORG E-ORG reported O O
The BIOES output makes the entity endpoint explicit: "Times" receives E-ORG rather than I-ORG, marking it as the final token in the span. This is the key difference: with BIO, you can only determine that an entity has ended after seeing a B or O tag at the next position; with BIOES, the E tag tells you immediately.
This "look-ahead" property of BIO is a subtle but real limitation. During sequential decoding, a BIO decoder reading token by token does not know whether the current I tag is the last one in the entity until it reads the next token. This means the decoder cannot output complete entity spans until it sees what comes after. BIOES eliminates this look-ahead dependency: an E tag immediately signals "the entity I've been tracking ends here." For streaming applications or interactive systems that display entity annotations as they process each token, BIOES provides a cleaner interface.