Word Tokenization › Punctuation Handling
Language AI HandbookAll chapters Word Tokenization › Punctuation HandlingOne section · Sections in this chapter4 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Most tokenizers treat punctuation as separate tokens. This makes sense linguistically: a period ends a sentence, a comma separates clauses, and quotation marks delimit speech. These are meaningful units that deserve their own tokens. But separating punctuation from words is not as simple as "remove all punctuation": some punctuation is integral to the word it accompanies.
The challenge with punctuation is that the same character can play very different roles. A period ends a sentence, but it also marks abbreviations, decimal points, and domain names. A hyphen separates word parts in "self-aware" but appears in phone numbers like "555-1234". An apostrophe marks possessives and contractions. There is no single rule that handles all of these correctly, which is why punctuation handling is one of the most complex parts of a tokenizer.
The approach we take is to start with a general rule (punctuation gets separated) and then add exceptions for known special cases. This is the fundamental pattern of rule-based tokenization: broad patterns with targeted exceptions.
import re
def punctuation_tokenize(text):
"""Split on whitespace and separate punctuation."""
# Add spaces around punctuation
text = re.sub(r'([.,!?;:"\'\(\)\[\]])', r" \1 ", text)
# Split and filter empty strings
return [t for t in text.split() if t]
# Test the improved tokenizer
test_text = "Hello, world! How are you?"
punct_tokens = punctuation_tokenize(test_text)Text: 'Hello, world! How are you?' Whitespace tokens: ['Hello,', 'world!', 'How', 'are', 'you?'] Punctuation tokens: ['Hello', ',', 'world', '!', 'How', 'are', 'you', '?']
Now punctuation is separated. But we've introduced new problems. What about abbreviations like "Dr." or "U.S.A."? What about decimal numbers like "3.14"? What about contractions? The regex approach is a good start, but it needs refinement for real-world text.
The regular expression ([.,!?;:"'\(\)\[\]]) matches any of the listed punctuation characters and wraps them in spaces using the substitution \1. This simple pattern handles the most common case, but it doesn't know the difference between "Dr." (an abbreviation) and "end." (a sentence-final period). Adding that knowledge requires a different approach.
Handling Abbreviations
Abbreviations pose a challenge because the period is part of the word, not a sentence-ending punctuation mark. A tokenizer needs to know which periods to separate and which to keep. This is a classic example of a problem that looks easy until you enumerate the cases.
The most common approach is to maintain a list of known abbreviations and protect them from splitting. Think of it as putting a temporary placeholder over the periods in abbreviations before applying the general punctuation rule, then restoring them afterward.
# Common abbreviations that should keep their periods
abbreviations = {
"Dr.",
"Mr.",
"Mrs.",
"Ms.",
"Jr.",
"Sr.",
"vs.",
"etc.",
"i.e.",
"e.g.",
"U.S.",
"U.K.",
"a.m.",
"p.m.",
}
def abbreviation_aware_tokenize(text):
"""Tokenize while preserving common abbreviations."""
# First, protect abbreviations by replacing periods temporarily
protected = text
for abbr in abbreviations:
protected = protected.replace(abbr, abbr.replace(".", "<DOT>"))
# Now do punctuation tokenization
protected = re.sub(r'([.,!?;:"\'\(\)\[\]])', r" \1 ", protected)
tokens = [t for t in protected.split() if t]
# Restore periods in abbreviations
tokens = [t.replace("<DOT>", ".") for t in tokens]
return tokens
# Test with abbreviations
abbr_text = "Dr. Smith met Mr. Jones at 3 p.m. in the U.S."
abbr_tokens = abbreviation_aware_tokenize(abbr_text)Text: 'Dr. Smith met Mr. Jones at 3 p.m. in the U.S.' Naive punctuation: ['Dr', '.', 'Smith', 'met', 'Mr', '.', 'Jones', 'at', '3', 'p', '.', 'm', '.', 'in', 'the', 'U', '.', 'S', '.'] Abbreviation-aware: ['Dr.', 'Smith', 'met', 'Mr.', 'Jones', 'at', '3', 'p.m.', 'in', 'the', 'U.S.']
The abbreviation-aware tokenizer keeps "Dr.", "Mr.", "p.m.", and "U.S." intact while still separating the final period. This pattern of protecting special cases before applying general rules is common in NLP and illustrates a broader principle: when you cannot write a single rule that handles all cases, write rules for the exceptions first and apply the general rule to whatever remains.
Notice that this approach has a coverage problem: our list only contains 14 abbreviations. In practice, a large English abbreviation list might contain thousands of entries. Medical text has hundreds of specialized abbreviations ("mg.", "mL.", "IV.", "p.o."). Legal text has its own set. Financial text has others. Production tokenizers often use abbreviation lists with thousands of entries derived from large corpora.