Sentence Segmentation › Sentence Segmentation

Splitting text into sentences sounds trivial. Just look for periods, right? But consider this: "Dr. Smith earned $3.5M in 2023. He works at U.S. Steel Corp." That single paragraph contains four periods, but only one marks a true sentence boundary. The others appear in abbreviations or numbers; the company name also contains one. Sentence segmentation, also called sentence boundary detection, is the task of identifying where one sentence ends and another begins.

Why does this matter for NLP? Sentences are fundamental units of meaning. Machine translation systems translate sentence by sentence. Summarization algorithms need to extract complete sentences. Sentiment analysis often operates at the sentence level. Get the boundaries wrong, and downstream tasks inherit corrupted input.

Think of sentence segmentation as the foundation on which almost every NLP pipeline rests. If you misidentify sentence boundaries, you hand your downstream components fractured, incoherent fragments. A machine translation model that receives "Dr. Smith went to Washington." split at the period after "Dr." ends up translating "Dr." as one unit and "Smith went to Washington." as another. The resulting output is nonsense, and no amount of sophisticated translation modeling can compensate for broken input. The same logic applies to summarizers, question-answering systems, and speech synthesizers. A seemingly small segmentation error propagates into every stage that follows.

The challenge is structural: punctuation in written language is overloaded. The period serves at least six distinct functions, only one of which signals a sentence boundary. A naive system that treats every period as a sentence terminator will shatter text into fragments at every abbreviation, decimal number, and URL. Conversely, a system that is too conservative will merge distinct sentences into a single run-on unit, losing the boundaries that give text its logical structure.

This chapter explores why periods lie, how rule-based systems attempt to disambiguate them, and how the Punkt algorithm uses unsupervised learning to detect sentence boundaries without hand-crafted rules. You'll implement segmenters from scratch, compare approaches across different text domains, and learn to evaluate their performance quantitatively. By the end, you will understand how to split sentences, why the problem is harder than it looks, and what the best modern tools do to solve it.