Text Normalization › Limitations and Challenges

Text normalization is powerful, but it is not perfect. Every normalization decision involves a tradeoff between two kinds of error: missing valid matches (false negatives from under-normalization) and incorrectly merging distinct entities (false positives from over-normalization). There is no single setting that achieves zero error on both dimensions simultaneously. Understanding these tradeoffs is what separates a practitioner who applies normalization thoughtfully from one who applies it indiscriminately.

The most significant limitation is irreversible information loss. When you apply case folding, accent stripping, or compatibility normalization, you permanently destroy information about the original text. Proper nouns lose their capitalization signal that distinguishes "Apple" (the company) from "apple" (the fruit). Accented characters lose their diacritics, making "resume" and "résumé" indistinguishable even when the context demands the distinction. Ligatures lose their typographic identity, which may matter in a system analyzing historical printing practices. Once normalized, you cannot recover the original form without storing it separately. This argues for a dual-storage approach: preserve the original text for display and downstream tasks that need fidelity, and maintain a separately normalized form for search and comparison.

Language specificity is a second major limitation. There is no normalization strategy that works correctly for all languages simultaneously. Turkish requires locale-aware case handling that differs from English. Arabic and Hebrew have their own combining marks and contextual forms. Chinese and Japanese, along with Korean, have traditional and simplified variant characters that raise questions about whether compatibility normalization is appropriate. Thai and other Southeast Asian scripts have complex character ordering rules. The Unicode normalization algorithm handles the technical representation questions, but it cannot answer the semantic question of whether two strings should be considered equivalent in a given application. A search engine for a specific language should be calibrated for that language's specific normalization needs.

A related challenge is the context-dependence of the right normalization. The same text might need different normalization in different parts of the same system. A document storage system should use conservative normalization (NFC, whitespace cleanup, nothing else) to preserve the original text. The search index for that same system should use aggressive normalization (NFKC, case folding, accent stripping) to maximize recall. A named entity recognizer operating on that text should probably use moderate normalization: NFC and whitespace cleanup, but not case folding or accent stripping, because capitalization and accent patterns are important features for recognizing names. Applying a single normalization strategy to all use cases is almost always wrong for at least one of them.

Security is a less obvious but important concern. Unicode contains thousands of characters that look nearly identical to common ASCII characters but have different code points. The Cyrillic letter "а" (U+0430) looks indistinguishable from the Latin letter "a" (U+0061) in most fonts. An attacker can register a domain name or username that contains Cyrillic characters but appears to a human as if it were made of ASCII characters. This is called a homograph attack. NFKC normalization does not protect against this attack because Cyrillic and Latin characters are canonical, not compatibility equivalents. Defending against homograph attacks requires more aggressive normalization, such as converting all characters to their ASCII equivalents where possible, or enforcing script consistency (all characters in a username must belong to the same script).

Finally, the Unicode standard itself continues to evolve. New characters and scripts are added with each major version, and occasionally normalization behaviors are adjusted. Code that handles all current Unicode correctly may not handle future Unicode correctly without updates. This is a reason to rely on well-maintained library implementations of Unicode algorithms rather than reimplementing them from scratch.