Sentence Segmentation › Building a Production Segmenter
Language AI HandbookAll chapters Sentence Segmentation › Building a Production SegmenterOne section · Sections in this chapter8 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
For production use, you'll want a segmenter that balances accuracy with speed and handles the full range of text patterns encountered in real applications. It should handle more than the clean examples in academic benchmarks. The approach is to combine multiple lightweight strategies rather than relying on a single algorithm to handle every case perfectly.
The core idea is layered protection. Before running any statistical segmentation, identify patterns in the text that contain periods but should never be split: URLs, email addresses, decimal numbers, and known patterns like version strings ("Python 3.11.2") or date formats ("03/15/2024"). Replace these with placeholder tokens that contain no periods. Run the Punkt tokenizer on the protected text. Then replace the placeholders with the original strings. This preprocessing-and-restoration pattern handles the categories where Punkt is weakest, while letting Punkt's statistical machinery handle the harder cases involving abbreviations and proper nouns.
Here's a practical implementation:
class ProductionSegmenter:
"""
A production-ready sentence segmenter combining multiple approaches.
"""
def __init__(self, use_spacy=False):
self.use_spacy = use_spacy
# Precompile regex patterns
self.url_pattern = re.compile(
r"https?://\S+|www\.\S+|\S+\.(com|org|net|edu|gov)\b"
)
self.email_pattern = re.compile(r"\S+@\S+\.\S+")
self.number_pattern = re.compile(r"\d+\.\d+")
# Placeholder for protected content
self.placeholder_map = {}
def _protect_special(self, text):
"""Replace URLs, emails, and numbers with placeholders."""
self.placeholder_map = {}
counter = 0
for pattern in [
self.url_pattern,
self.email_pattern,
self.number_pattern,
]:
for match in pattern.finditer(text):
placeholder = f"__PROTECTED_{counter}__"
self.placeholder_map[placeholder] = match.group()
text = text.replace(match.group(), placeholder, 1)
counter += 1
return text
def _restore_special(self, sentences):
"""Restore protected content in sentences."""
restored = []
for sent in sentences:
for placeholder, original in self.placeholder_map.items():
sent = sent.replace(placeholder, original)
restored.append(sent)
return restored
def segment(self, text):
"""Segment text into sentences."""
# Protect special content
protected_text = self._protect_special(text)
# Use NLTK Punkt for segmentation
sentences = sent_tokenize(protected_text)
# Restore protected content
sentences = self._restore_special(sentences)
# Post-process: merge incorrectly split sentences
sentences = self._merge_fragments(sentences)
return sentences
def _merge_fragments(self, sentences):
"""Merge sentence fragments that were incorrectly split."""
if len(sentences) <= 1:
return sentences
merged = [sentences[0]]
for sent in sentences[1:]:
# If sentence starts with lowercase, merge with previous
if sent and sent[0].islower():
merged[-1] = merged[-1] + " " + sent
else:
merged.append(sent)
return mergedThe ProductionSegmenter combines multiple strategies. It first replaces URLs and email addresses with placeholders, doing the same for decimal numbers to prevent false splits. It then applies NLTK's Punkt tokenizer for the core segmentation and finally merges any sentence fragments that start with lowercase letters.
Let's test this approach on challenging inputs:
prod_segmenter = ProductionSegmenter()
production_tests = [
"Visit https://example.com/page.html for details. Click the link.",
"Contact john.doe@company.com for help. Response time is 24hrs.",
"The price is $19.99 per month. That's a 50% discount.",
"Dr. Jane Smith, Ph.D., leads the team. She has 20 years of experience.",
]
prod_results = [
(text, prod_segmenter.segment(text)) for text in production_tests
]Production Segmenter Results: ====================================================================== Input: "Visit https://example.com/page.html for details. Click the link." 1. "Visit https://example.com/page.html for details." 2. "Click the link." Input: "Contact john.doe@company.com for help. Response time is 24hrs." 1. "Contact john.doe@company.com for help." 2. "Response time is 24hrs." Input: "The price is $19.99 per month. That's a 50% discount." 1. "The price is $19.99 per month." 2. "That's a 50% discount." Input: "Dr. Jane Smith, Ph.D., leads the team. She has 20 years of experience." 1. "Dr. Jane Smith, Ph.D., leads the team." 2. "She has 20 years of experience."
The production segmenter correctly handles all test cases. The URL with its multiple periods (https://example.com/page.html) is preserved intact, the email address isn't split, and the decimal price $19.99 doesn't create a false boundary. This layered approach, combining preprocessing, core segmentation, and postprocessing, handles real-world text patterns well.
Notice that the _merge_fragments postprocessing step applies a simple heuristic: if a predicted sentence starts with a lowercase letter, merge it with the previous sentence. This catches cases where Punkt incorrectly splits a sentence mid-clause. The heuristic is crude and will occasionally merge sentences that should remain separate (for example, in transcribed speech where a new sentence begins with a lowercase word), but it is a useful safety net that improves overall accuracy on standard text. In practice, you would tune or disable this step based on your specific text domain.
In practice, the production segmenter shown here handles the vast majority of English text correctly. When you encounter a domain with many failures, the most effective improvement strategy is usually to fine-tune the Punkt model on a sample of in-domain text. Collect a few hundred to a few thousand sentences of representative text, train a new PunktSentenceTokenizer, and compare performance against the default model. The training process takes seconds and typically produces significant accuracy gains for domain-specific abbreviations.