Sentence Segmentation › Rule-Based Sentence Segmentation
Language AI HandbookAll chapters Sentence Segmentation › Rule-Based Sentence SegmentationOne section · Sections in this chapter3 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before machine learning approaches entered NLP, practitioners built rule-based systems using hand-crafted patterns. These systems use abbreviation lists, regular expressions, and heuristics to identify boundaries. Understanding their design and limitations is valuable for historical context and because rule-based ideas persist inside modern systems as preprocessing layers and fallback strategies.
Rule-based segmenters were the dominant approach throughout the 1980s and 1990s. MUCMark and GATE used explicit abbreviation lists and decision trees encoding linguistic knowledge, as did early versions of the Stanford NLP toolkit. The appeal was interpretability: you could look at the rules and understand exactly why a given boundary was predicted. The drawback was brittleness. Every time a new domain introduced new abbreviations or punctuation conventions, a human expert had to update the rules by hand.
The fundamental architecture of a rule-based system is straightforward. You scan the text looking for potential boundary tokens (periods, question marks, exclamation points), then apply a series of filters to decide whether each candidate is a true boundary. The filters typically check: Is the word before the period a known abbreviation? Does the word after the period start with a capital letter? Is the word after the period in a list of words that commonly start sentences? Is the preceding token a single uppercase letter (suggesting an initialism like "U.")? Each filter can veto or confirm the boundary.
NLP practitioners built rule-based systems using hand-crafted patterns. These systems use abbreviation lists, regular expressions, and heuristics to identify boundaries.
A Simple Rule-Based Approach
Let's build a basic sentence segmenter step by step:
class SimpleSegmenter:
"""A rule-based sentence segmenter."""
def __init__(self):
# Common abbreviations that don't end sentences
self.abbreviations = {
"mr",
"mrs",
"ms",
"dr",
"prof",
"sr",
"jr",
"vs",
"etc",
"viz",
"al",
"eg",
"ie",
"cf",
"inc",
"ltd",
"corp",
"co",
"jan",
"feb",
"mar",
"apr",
"jun",
"jul",
"aug",
"sep",
"oct",
"nov",
"dec",
"st",
"rd",
"th",
"ave",
"blvd",
"approx",
"dept",
"est",
"min",
"max",
"govt",
"natl",
"intl",
}
# Titles that precede names
self.titles = {
"mr",
"mrs",
"ms",
"dr",
"prof",
"rev",
"gen",
"col",
"lt",
"sgt",
}
def is_abbreviation(self, token):
"""Check if token is a known abbreviation."""
# Remove trailing period for comparison
word = token.rstrip(".").lower()
return word in self.abbreviations
def is_likely_sentence_end(self, before, punct, after):
"""Determine if punctuation likely ends a sentence."""
# Question marks and exclamation points usually end sentences
if punct in "?!":
# Unless inside quotes followed by lowercase
if after and after[0].islower():
return False
return True
# For periods, apply heuristics
if punct == ".":
# Check if preceding word is abbreviation
words_before = before.split()
if words_before:
last_word = words_before[-1]
if self.is_abbreviation(last_word + "."):
# Abbreviation followed by lowercase = not sentence end
if after and after.strip() and after.strip()[0].islower():
return False
# Abbreviation followed by capital might still be sentence end
# Use additional heuristics...
# Period followed by capital letter suggests sentence boundary
if after and after.strip():
first_char = after.strip()[0]
if first_char.isupper():
return True
return FalseThe SimpleSegmenter class maintains two key data structures: a set of common abbreviations (like "dr", "mr", "inc") that shouldn't trigger sentence splits, and a set of titles that typically precede names. The is_likely_sentence_end method applies heuristics to determine if punctuation marks a true boundary.
Now let's add the main segmentation logic:
def segment(self, text):
"""Split text into sentences."""
sentences = []
current = []
# Pattern to find potential sentence boundaries
# Matches period, question mark, or exclamation followed by space and capital
boundary_pattern = re.compile(r"([.!?])\s+")
# Split on potential boundaries
parts = boundary_pattern.split(text)
i = 0
while i < len(parts):
current.append(parts[i])
if i + 1 < len(parts) and parts[i + 1] in ".!?":
punct = parts[i + 1]
before = "".join(current)
after = parts[i + 2] if i + 2 < len(parts) else ""
if self.is_likely_sentence_end(before, punct, after):
current.append(punct)
sentences.append("".join(current).strip())
current = []
i += 2
else:
current.append(punct)
i += 2
else:
i += 1
# Add remaining text
if current:
remaining = "".join(current).strip()
if remaining:
sentences.append(remaining)
return sentences
# Add method to class
SimpleSegmenter.segment = segmentThe segment method uses a regex pattern to find potential boundaries (punctuation followed by whitespace), then applies our heuristics to decide which boundaries are real. Now let's test the segmenter on challenging inputs:
segmenter = SimpleSegmenter()
test_texts = [
"Hello world. How are you?",
"Dr. Smith went to Washington. He met the president.",
"I paid $3.50 for coffee. It was expensive.",
"She works at U.S. Steel Corp. The company is huge.",
]
results = [(text, segmenter.segment(text)) for text in test_texts]Simple Segmenter Results: ====================================================================== Input: "Hello world. How are you?" Sentences found: 2 1. "Hello world." 2. "How are you?" Input: "Dr. Smith went to Washington. He met the president." Sentences found: 3 1. "Dr." 2. "Smith went to Washington." 3. "He met the president." Input: "I paid $3.50 for coffee. It was expensive." Sentences found: 2 1. "I paid $3.50 for coffee." 2. "It was expensive." Input: "She works at U.S. Steel Corp. The company is huge." Sentences found: 3 1. "She works at U.S." 2. "Steel Corp." 3. "The company is huge."
The results reveal the segmenter's limitations. While it correctly handles "Hello world." and recognizes "Dr." as an abbreviation, it may struggle with compound abbreviations like "U.S. Steel Corp." where multiple abbreviations appear in sequence. The heuristic of "capital letter after period suggests new sentence" works in simple cases but fails when abbreviations precede proper nouns.
Limitations of Rule-Based Approaches
Hand-crafted rules face several fundamental problems, and understanding these problems is what motivates the statistical approach that follows.
Incomplete coverage: No abbreviation list is complete. New abbreviations emerge constantly, and domain-specific texts use specialized terms. A list built for news articles will miss the abbreviations common in medical records. A list built for English scientific papers will miss the conventions of German, French, or Japanese academic writing. The maintainer is always playing catch-up.
Language dependence: Rules designed for English fail for other languages. German capitalizes all nouns, which completely breaks the "capital letter after period equals new sentence" heuristic. In German, "Das Dr. Müller Projekt ist fertig." does not contain a sentence boundary after "Müller", but a capital-letter rule would mark it as a boundary. Spanish begins questions and exclamations with inverted marks (¿ and ¡), which an English rule system would not handle. East Asian languages like Chinese and Japanese use the full-width period (。) instead of the ASCII period, and they have no spaces between words, making space-based heuristics useless.
Context blindness: Static rules can't capture the context-dependent nature of abbreviations. "St." might mean "Saint" (as in "St. Patrick's Day") or "Street" (as in "42nd St."). Both uses are followed by capitalized words, so the capital-letter heuristic cannot distinguish them. Whether "St." ends a sentence depends on what grammatical role the following word plays, which requires syntactic understanding far beyond pattern matching.
Maintenance burden: As edge cases accumulate, rule systems become complex and fragile. Adding one rule to fix one case can break another. A rule added to handle "U.S." might interfere with detecting true sentence boundaries after single-letter initials. The system becomes a tangled web of special cases where every fix introduces new potential failures. Historically, rule-based systems for sentence segmentation grew to thousands of lines of code, most of it special-case handling, and still achieved lower accuracy than statistical systems trained on a few thousand sentences of annotated text.