Sentence Segmentation › Performance Comparison

Let's compare different segmentation approaches on a diverse test set to see how far each approach has come relative to the naive baseline.

The following table compares F1 scores across different text categories:

The naive approach of splitting on every period achieves only 30% F1 on abbreviation-heavy text, a catastrophic failure. Rule-based approaches improve substantially but still struggle with complex patterns, particularly URLs and emails where the pattern space is too large to enumerate by hand. Punkt's unsupervised learning achieves over 90% on most categories, and the production segmenter's preprocessing pushes accuracy even higher for URLs and emails, reaching 95% F1.

Notice that the production segmenter does not achieve perfect scores even on categories where one might expect it. Quotation handling (0.92) reflects the ambiguity in how punctuation inside quotes interacts with sentence boundaries. This is the ceiling imposed by the fundamental ambiguity of the task, not a limitation of the algorithm. No algorithm that works purely on character patterns can perfectly resolve these cases without understanding the discourse structure of the surrounding text.

The performance gap between simple text (0.98) and abbreviation-heavy text (0.94) even for the production segmenter reflects a fundamental asymmetry: simple sentence-ending periods are easy to detect, but abbreviation periods require learned knowledge about which words are abbreviations. The remaining 6% error rate in abbreviation-heavy text represents cases where either the abbreviation is not in the training vocabulary, or where an abbreviation ends a sentence and the system cannot determine which interpretation is correct. These are the inherently hard cases, and they motivate the shift toward neural approaches that can use richer contextual representations.