Word Tokenization › Building a Rule-Based Tokenizer
Language AI HandbookAll chapters Word Tokenization › Building a Rule-Based TokenizerOne section · Sections in this chapter8 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's build a more complete tokenizer that handles common English patterns. We'll combine the techniques we've discussed into a coherent system. The goal is not to build a production-quality tokenizer (you should use NLTK or spaCy for that) but to understand what a rule-based tokenizer does under the hood.
Think of building a rule-based tokenizer as writing a procedure that explicitly encodes your understanding of the language. Every rule you add reflects a decision about how a particular character sequence should be handled. The tokenizer is an executable specification of your tokenization convention.
The architecture we'll use processes text in a fixed sequence of steps. First, we protect exceptions (abbreviations that should not be split). Then we apply transformations (split contractions). Then we apply general rules (separate punctuation). Finally, we restore protected elements and clean up. This ordering matters: if we applied the general punctuation rule first, we would split periods in abbreviations before getting a chance to protect them.
import re
from typing import List
class RuleBasedTokenizer:
"""A rule-based English tokenizer."""
def __init__(self):
# Abbreviations to protect
self.abbreviations = {
"dr.",
"mr.",
"mrs.",
"ms.",
"jr.",
"sr.",
"vs.",
"etc.",
"i.e.",
"e.g.",
"u.s.",
"u.k.",
"a.m.",
"p.m.",
"inc.",
"ltd.",
"co.",
}
# Contractions to split
self.contractions = [
(r"n't\b", " n't"),
(r"'ll\b", " 'll"),
(r"'re\b", " 're"),
(r"'ve\b", " 've"),
(r"'d\b", " 'd"),
(r"'m\b", " 'm"),
(r"'s\b", " 's"),
]
def tokenize(self, text: str) -> List[str]:
"""Tokenize text into words and punctuation."""
# Lowercase for abbreviation matching
text_lower = text.lower()
# Protect abbreviations
protected = text
for abbr in self.abbreviations:
if abbr in text_lower:
# Find and protect (case-insensitive)
pattern = re.compile(re.escape(abbr), re.IGNORECASE)
protected = pattern.sub(
lambda m: m.group().replace(".", "<DOT>"), protected
)
# Split contractions
for pattern, replacement in self.contractions:
protected = re.sub(
pattern, replacement, protected, flags=re.IGNORECASE
)
# Separate punctuation (except protected dots)
protected = re.sub(r'([.,!?;:"\'\(\)\[\]{}])', r" \1 ", protected)
# Restore protected dots
protected = protected.replace("<DOT>", ".")
# Split and clean
tokens = [t.strip() for t in protected.split() if t.strip()]
return tokens
# Create tokenizer instance
tokenizer = RuleBasedTokenizer()Rule-Based Tokenizer initialized with: 17 protected abbreviations 7 contraction patterns
Now let's test our tokenizer on various challenging inputs:
# Test cases
test_cases = [
"Hello, world!",
"Dr. Smith's patients don't like waiting.",
"I can't believe it's not butter!",
"The U.S. economy grew 3.5% in Q4.",
'She said, "Hello!" and left.',
"We'll meet at 3 p.m. tomorrow.",
]
results = [(text, tokenizer.tokenize(text)) for text in test_cases]Tokenization Results: ============================================================ Input: Hello, world! Tokens: ['Hello', ',', 'world', '!'] Count: 4 Input: Dr. Smith's patients don't like waiting. Tokens: ['Dr.', 'Smith', "'", 's', 'patients', 'do', 'n', "'", 't', 'like', 'waiting', '.'] Count: 12 Input: I can't believe it's not butter! Tokens: ['I', 'ca', 'n', "'", 't', 'believe', 'it', "'", 's', 'not', 'butter', '!'] Count: 12 Input: The U.S. economy grew 3.5% in Q4. Tokens: ['The', 'U.S.', 'economy', 'grew', '3', '.', '5%', 'in', 'Q4', '.'] Count: 10 Input: She said, "Hello!" and left. Tokens: ['She', 'said', ',', '"', 'Hello', '!', '"', 'and', 'left', '.'] Count: 10 Input: We'll meet at 3 p.m. tomorrow. Tokens: ['We', "'", 'll', 'meet', 'at', '3', 'p.m.', 'tomorrow', '.'] Count: 9
Our tokenizer applies its rules in layers. The RuleBasedTokenizer class demonstrates the approach: each layer handles a specific case, and the layers are applied in order. The abbreviation protection layer runs first to prevent the punctuation layer from incorrectly splitting abbreviation periods. The contraction layer runs second to split contractions before the general punctuation layer might mishandle the apostrophes.
Limitations of Rule-Based Tokenization
Rule-based tokenizers have inherent limitations that become apparent as you test them on more varied input. Understanding these limitations motivates the shift to library-based tokenizers and, eventually, to learned tokenization methods.
Coverage is the most immediate limitation. You can't anticipate every abbreviation or special case. New abbreviations emerge constantly: "COVID", "blockchain", "fintech", and countless domain-specific terms were not in any abbreviation list twenty years ago. Every time a new abbreviation appears in your text, you must decide whether to add it to your list.
Ambiguity is a deeper problem. "St." could be "Street" or "Saint". "Dr." could be "Doctor" or "Drive". "No." could be "Number" or the word "No". Rules can't disambiguate without context, and adding context-sensitivity to a rule-based system quickly leads to a system that is essentially a statistical model in disguise, just a very complex and brittle one.
Language dependence means that rules written for English don't work for other languages. Each language needs its own rule set, its own abbreviation list, its own contraction handling. Maintaining rule sets for dozens of languages is a substantial engineering effort.
# Edge cases that break our tokenizer
edge_cases = [
"I live on Oak St. near Dr. Brown's office.", # St. = Street
"St. Patrick's Day is in March.", # St. = Saint
"The temp. is 98.6°F.", # Unlisted abbreviation
"Check out the website: https://example.com", # URL
"Email me at user@example.com!", # Email
]
edge_results = [(text, tokenizer.tokenize(text)) for text in edge_cases]Edge Cases: ============================================================ Input: I live on Oak St. near Dr. Brown's office. Tokens: ['I', 'live', 'on', 'Oak', 'St', '.', 'near', 'Dr.', 'Brown', "'", 's', 'office', '.'] Input: St. Patrick's Day is in March. Tokens: ['St', '.', 'Patrick', "'", 's', 'Day', 'is', 'in', 'March', '.'] Input: The temp. is 98.6°F. Tokens: ['The', 'temp', '.', 'is', '98', '.', '6°F', '.'] Input: Check out the website: https://example.com Tokens: ['Check', 'out', 'the', 'website', ':', 'https', ':', '//example', '.', 'com'] Input: Email me at user@example.com! Tokens: ['Email', 'me', 'at', 'user@example', '.', 'com', '!']
The tokenizer handles some cases well but struggles with URLs and email addresses. Abbreviations missing from its list also cause problems. Notice that "St." is handled by the abbreviation list, but this means "St. Patrick's Day" incorrectly keeps "St." as an abbreviation when it refers to a title (Saint). Production tokenizers address these with broader rules or statistical methods, or they simply accept that some edge cases will be handled imperfectly and focus on getting the common cases right.