Text Normalization › Summary

Text normalization converts text into consistent, comparable forms. We covered:

• Unicode normalization forms: NFC composes, NFD decomposes, NFKC and NFKD add compatibility mappings
• Case folding: Use casefold() for case-insensitive comparison, not lower()
• Diacritic handling: NFD decomposition plus filtering removes accents cleanly and without a lookup table
• Whitespace normalization: Unicode has many whitespace characters beyond space and tab, and zero-width space is invisible but breaks matching
• Ligature expansion: NFKC expands most typographic ligatures, preserving linguistic digraphs like æ and œ
• Full-width conversion: NFKC converts full-width ASCII to standard ASCII automatically
• Pipeline order: Normalization steps interact, and the order affects results for complex Unicode sequences

Key takeaways:

• NFC is the default choice for general text storage
• NFKC with casefold is best for search and comparison
• Always normalize before comparing strings for equality
• Normalization order matters: plan your pipeline carefully
• Test with representative data: edge cases will surprise you
• Preserve originals: keep unnormalized text when possible, and store normalized forms separately
• Match aggressiveness to use case: storage and search may need different normalization levels from ML features

In the next chapter, we will explore tokenization, the process of breaking text into meaningful units for further processing. The quality of your normalization directly affects tokenization quality: a tokenizer that encounters un-normalized text with ligatures, zero-width spaces, or inconsistent Unicode forms will produce noisy, inconsistent tokens. The work you do in normalization pays dividends at every subsequent stage of the NLP pipeline.