BIO Tagging › Tag Consistency and Validation
Language AI HandbookAll chapters BIO Tagging › Tag Consistency and ValidationOne section · Sections in this chapter6 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Real-world tagging systems produce inconsistent output. A well-designed pipeline includes validation to detect problems and, where possible, repair them automatically.
Validation matters at two distinct pipeline stages. Before training, you validate annotation data to catch inconsistencies introduced during the labeling process. Annotators may disagree on boundaries, tools may produce off-by-one errors, and automatic pre-annotation may generate malformed sequences that human annotators did not catch. Feeding invalid sequences to a sequence model during training can produce subtle bugs: the model might learn spurious patterns from incorrectly tagged tokens, or the loss computation might silently ignore malformed positions.
After inference, you validate model output to ensure it can be parsed correctly downstream. Even a model trained entirely on valid sequences may occasionally predict invalid ones, particularly on out-of-distribution inputs where the model is uncertain. The application reading NER output may crash or produce wrong results if it assumes valid sequences and receives invalid ones.
def validate_bio_sequence(tags):
"""
Validate a BIO tag sequence and report all errors.
Returns list of (position, error_type, description) tuples.
"""
errors = []
prev_tag = "O"
for i, tag in enumerate(tags):
if tag == "O":
prev_tag = tag
continue
if not (tag.startswith("B-") or tag.startswith("I-")):
errors.append((i, "INVALID_TAG", f"Unrecognized tag: {tag}"))
continue
prefix = tag[0]
entity_type = tag[2:] if len(tag) > 2 else ""
if not entity_type:
errors.append(
(i, "MISSING_TYPE", f"Tag missing entity type: {tag}")
)
if prefix == "I":
if prev_tag == "O":
errors.append(
(i, "ORPHAN_I", f"I tag without preceding B: {tag}")
)
elif prev_tag.startswith("B-") or prev_tag.startswith("I-"):
prev_type = prev_tag[2:]
if prev_type != entity_type:
errors.append(
(
i,
"TYPE_MISMATCH",
f"I-{entity_type} follows {prev_tag}",
)
)
prev_tag = tag
return errors
# Test with a problematic sequence
problematic_tags = ["O", "I-PER", "I-PER", "B-LOC", "I-ORG", "O", "B-PER", "O"]
validation_errors = validate_bio_sequence(problematic_tags)Sequence Validation: ------------------------------------------------------- Tags: ['O', 'I-PER', 'I-PER', 'B-LOC', 'I-ORG', 'O', 'B-PER', 'O'] Found 2 error(s): Position 1: [ORPHAN_I] I tag without preceding B: I-PER Position 4: [TYPE_MISMATCH] I-ORG follows B-LOC
Once errors are identified, we can apply heuristic repairs:
def repair_bio_sequence(tags):
"""
Repair common BIO sequence errors.
Strategies applied:
- Convert orphan I tags to B tags
- Resolve type mismatches by starting new entities
Returns (repaired_tags, repair_log).
"""
repaired = tags.copy()
repairs = []
prev_tag = "O"
for i, tag in enumerate(repaired):
if tag == "O":
prev_tag = tag
continue
if tag.startswith("I-"):
entity_type = tag[2:]
if prev_tag == "O":
# Orphan I: convert to B
repaired[i] = f"B-{entity_type}"
repairs.append((i, tag, repaired[i], "orphan I -> B"))
elif prev_tag[0] in "BI" and prev_tag[2:] != entity_type:
# Type mismatch: start new entity
repaired[i] = f"B-{entity_type}"
repairs.append((i, tag, repaired[i], "type mismatch -> new B"))
prev_tag = repaired[i]
return repaired, repairs
repaired_tags, repair_log = repair_bio_sequence(problematic_tags)Sequence Repair: ------------------------------------------------------- Original: ['O', 'I-PER', 'I-PER', 'B-LOC', 'I-ORG', 'O', 'B-PER', 'O'] Repaired: ['O', 'B-PER', 'I-PER', 'B-LOC', 'B-ORG', 'O', 'B-PER', 'O'] Applied 2 repair(s): Position 1: I-PER -> B-PER (orphan I -> B) Position 4: I-ORG -> B-ORG (type mismatch -> new B)
The repair function converts orphan I tags into B tags and creates new entity boundaries at type mismatches. These are the two most common patterns in model output, where the model may predict the correct type but miss a boundary.
The repair visualization highlights positions that changed (shown with orange borders). Position 1 was an orphan I-PER that became B-PER. Position 4 was a type mismatch I-ORG that became B-ORG. Both repairs create clean entity boundaries that the decoder can now process correctly.
It is important to log all repairs rather than applying them silently. When you repair sequences during training data preparation, the log tells you whether your annotation pipeline has systematic problems. If 20% of sequences require repairs, something is wrong with your annotation tool or guidelines. If only 0.1% require repairs, you have an isolated edge case. The repair log is also a diagnostic tool during model development: when a model's output requires many repairs at inference time, it signals the model is confused about boundary detection and may need targeted fine-tuning on boundary examples.