Sentence Segmentation › Handling Edge Cases

Real-world text contains numerous edge cases that challenge even sophisticated segmenters. Most NLP tutorials present clean, well-formed text where the right answer is obvious. Production systems face something very different: user-generated content with informal grammar, technical documents with non-standard punctuation, legal transcripts with interrupted speech, and web-scraped text littered with encoding artifacts. Understanding the common failure modes helps you anticipate where your segmenter will struggle and design appropriate postprocessing strategies.

The four categories of edge cases that follow represent the most commonly encountered problems. Each one exploits a different weakness in the assumptions that most segmenters make. Quotation marks embed one sentence's punctuation inside another sentence's scope. Numbered lists use periods in structural roles rather than sentence-ending roles. Ellipses represent intentionally unresolved sentence boundaries. Multiple punctuation marks stack signals that segmenters must interpret coherently. A production-quality segmenter needs explicit strategies for all four.

Sentences can contain quoted speech or parenthetical remarks that include their own sentence-ending punctuation. This creates a structural nesting problem: the outer sentence has not ended, but the inner quoted material has its own punctuation that a simple scanner would interpret as a sentence boundary. The challenge is especially acute in American typographic convention, where periods and question marks go inside closing quotation marks regardless of whether they end the outer sentence. British convention places them outside, but American convention places them inside, so the same surface string has different structural interpretations depending on the typographic tradition.

Numbered or bulleted lists present unique challenges because they use periods in a structural role that is entirely different from sentence-ending. The period after "1." in "1. Prepare the data." marks a list item, not a sentence boundary in the traditional sense. Whether this should be treated as a sentence boundary depends on the downstream task. For text-to-speech, each list item should probably be read as a separate prosodic unit. For machine translation, the items might need to be translated individually. For summarization, each item might be a candidate for inclusion. The "right" answer depends on application semantics, not on linguistic convention.

Ellipses (...) can appear mid-sentence or at sentence boundaries and represent a different kind of ambiguity from abbreviations. An abbreviation period has a determinate answer: either the word is an abbreviation or it is not, and the period either ends the sentence or does not. An ellipsis is intentionally indeterminate: it signals a pause, a trailing thought, or an omission, and whether it constitutes a sentence boundary depends on the writer's intent. In transcribed speech or informal writing, ellipses are especially common and especially ambiguous. A segmenter that always splits on ellipses will over-segment; one that never splits on them will merge sentences that should be separate.

Some sentences end with multiple punctuation marks: