Text Normalization › Key Functions and Parameters

When working with text normalization in Python, these are the essential functions and their most important parameters:

• unicodedata.normalize(form, text): Applies Unicode normalization to a string. The form parameter specifies the normalization form: 'NFC' (canonical composition, default for storage), 'NFD' (canonical decomposition, useful for accent stripping), 'NFKC' (compatibility composition, aggressive, for search), or 'NFKD' (compatibility decomposition).

• unicodedata.category(char): Returns a two-letter category code for a Unicode character. Common categories include 'Mn' (Mark, Nonspacing, for combining diacritics), 'Cc' (control characters), and 'Zs' (space separator). Useful for filtering specific character types during normalization.

• str.casefold(): Returns a casefolded copy of the string for case-insensitive comparison. More aggressive than lower(), handles special cases like German "ß" to "ss". Preferred over lower() for Unicode-aware case-insensitive matching.

• str.lower() vs str.casefold(): Use lower() for display (standard Unicode lowercasing) and casefold() for comparison (full Unicode case folding that handles language-specific mappings).

• re.sub(pattern, replacement, text): Essential for whitespace normalization. Common patterns include r'[\u00A0\u2000-\u200A\u202F\u205F\u3000]' for various Unicode spaces, r'[\u200B-\u200D\uFEFF]' for zero-width characters, and r' +' for multiple consecutive spaces.