Text Normalization › Unicode Normalization Forms

The Unicode standard defines four normalization forms to address representation ambiguity. Each form serves different purposes, and choosing the right one requires understanding what each form does and what it preserves.

Think of the four normalization forms as two orthogonal dimensions: compose versus decompose, and canonical versus compatibility. The compose/decompose dimension controls whether the form builds combined characters (precomposed) or breaks them apart (decomposed). The canonical/compatibility dimension controls whether the form respects all semantic distinctions (canonical) or aggressively maps visually similar but semantically distinct characters onto a common form (compatibility). Crossing these two dimensions gives you NFC (canonical compose), NFD (canonical decompose), NFKC (compatibility compose), and NFKD (compatibility decompose).

The canonical forms (NFC and NFD) make only those transformations that the Unicode standard considers representations of the same abstract character. They do not change meaning or lose semantic information. If you normalize "café" to NFC or NFD and then display the result, it looks exactly the same and means exactly the same thing. You have just changed the internal byte representation, not the character's identity.

The compatibility forms (NFKC and NFKD) go further. They apply compatibility mappings, which group characters that the Unicode standard considers semantically equivalent but visually distinct. The Latin ligature "fi" and the two-character sequence "fi" are compatibility equivalents: they look nearly the same in most fonts and mean the same thing, but they have different code points. NFKC maps the ligature to the two-character form. This is useful for search and comparison, but it destroys information about the original text's typography.

NFC (Normalization Form Canonical Composition) converts text to its shortest representation by combining base characters with their accents into single precomposed characters where possible. Starting from any valid Unicode string, NFC first applies canonical decomposition (separating all characters into their base plus combining marks), then applies canonical composition (re-merging sequences that have precomposed equivalents).

NFC is the most widely used normalization form in practice because it produces compact representations that match what users expect when they type accented characters on a keyboard. When you press a single key for "é" on a French keyboard, you produce the precomposed form. When a macOS application pastes text, it typically uses NFC. When web browsers submit form data, they use NFC. Normalizing your entire corpus to NFC eliminates the most common source of representation mismatch without any loss of information.

The key insight here is what NFC does under the hood. It does not simply look for decomposed sequences and merge them; it applies the full Unicode canonical composition algorithm, which handles edge cases like sequences with multiple combining marks in specific orders. After NFC, the five-code-point sequence "c", "a", "f", "e", U+0301 becomes the four-code-point sequence "c", "a", "f", U+00E9. The combining acute accent has been absorbed into the precomposed "é". The string now equals the original precomposed form.

NFC is the most commonly used normalization form. It produces the most compact representation and matches what most users expect when they type accented characters.

NFD (Normalization Form Canonical Decomposition) does the opposite of NFC: it breaks precomposed characters into their base character plus combining marks. Given "é" (U+00E9), NFD produces the sequence "e" (U+0065) followed by combining acute accent (U+0301).

Why would you want the decomposed form? NFD is particularly useful when you need to inspect or manipulate the diacritical components of characters independently. The most common use case is accent stripping: by decomposing to NFD, you can easily identify and remove combining marks (Unicode category "Mn" for Mark, Nonspacing) without any complex character-by-character lookup table. The combining marks are literally separated from their base characters, so filtering them out is a one-line operation.

Notice that NFD increases the string length from 4 to 5 code points. The single U+00E9 code point has been split into U+0065 (base letter "e") and U+0301 (combining acute accent). The visual appearance is unchanged, but the internal structure has been made explicit. NFD is useful when you need to manipulate accents separately from base characters, such as removing diacritics or analyzing character components.

The "K" forms apply compatibility decomposition in addition to canonical normalization. This maps characters that are semantically equivalent but visually distinct, going beyond the conservative approach of the canonical forms.

Think of NFKC as a "normalize for search" operation. When a user types "financial" with the ligature character (perhaps copied from a PDF), they probably want to find documents containing "financial" with separate letters. NFKC makes this work automatically. Similarly, a user searching for "km" should find "㎞" (the combined kilometer symbol), and a search for "IV" should find "Ⅳ" (the Roman numeral four). Compatibility normalization handles all these cases.

NFKC is aggressive. It converts the "fi" ligature to separate "f" and "i" characters, expands the circled digit to just "1", converts the Roman numeral to the plain letter sequence "IV", and transforms full-width characters to their ASCII equivalents. The "㎞" symbol becomes the two-character sequence "km". This is useful for search and comparison but destroys formatting and typographic information. If you are storing text for display, NFKC is too aggressive; use NFC instead. If you are building a search index or comparing strings for equality, NFKC is usually the right choice.

The table below shows how each normalization form transforms different input characters. NFC and NFD are canonical forms that preserve character identity, while NFKC and NFKD apply compatibility mappings that may change the character representation.

The canonical forms (NFC, NFD) preserve ligatures and special characters, changing only the internal representation. The compatibility forms (NFKC, NFKD) aggressively normalize to base characters, expanding ligatures and converting full-width to half-width. For rows where the input already has a precomposed canonical form (like café before the second row), all four forms produce identical output.

The right form depends on your use case, and using the wrong one can cause subtle problems. Applying NFKC to text you intend to display will change ligatures in ways the original author may not have intended. Applying only NFC to text you intend to search may miss legitimate matches between ligature and expanded forms.

The following table provides a quick reference for common use cases: