Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Text Normalization › Worked Example: Step-by-Step Normalization Trace

Michael BrenndoerferUpdated March 19, 20253 min read

Language AI HandbookAll chapters

Part 1 of 67Text as Data

Chapter 2Text Normalization › Worked Example: Step-by-Step Normalization Trace

Reading section 9 of 15 · Worked Example: Step-by-Step Normalization Trace · 3 minSection 9 of 15 · 3 min

Section 9 of 15 · 3 min read

Text Normalization › Worked Example: Step-by-Step Normalization TraceOne section · Sections in this chapter9 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To solidify these concepts, let us trace a single string through a complete normalization pipeline step by step. We will use the French phrase "Ｒésumé ﬁnancier" (a financial resume written with a mix of full-width characters, an accent, a ligature, and a double space) and follow each transformation in detail.

The input contains four distinct normalization problems layered into a single string. Working through them in order shows why each step is necessary and what it contributes.

In[34]:

Code

# Worked example: trace normalization steps on a messy French string
import re
import unicodedata

# Input: full-width R, accented e (precomposed), double space, fi ligature
input_text = "Ｒésumé  ﬁnancier"

# Step 1: Apply NFKC to handle full-width conversion, ligature expansion,
#         and canonical composition in one pass
step1 = unicodedata.normalize("NFKC", input_text)

# Step 2: Normalize whitespace (collapse multiple spaces)
step2 = re.sub(r" +", " ", step1).strip()

# Step 3: Case fold for case-insensitive comparison
step3 = step2.casefold()

# Step 4: Remove diacritics via NFD + Mn filter + NFC
step4_nfd = unicodedata.normalize("NFD", step3)
step4_filtered = "".join(
    c for c in step4_nfd if unicodedata.category(c) != "Mn"
)
step4 = unicodedata.normalize("NFC", step4_filtered)

Out[35]:

Console

Step-by-step normalization trace:
============================================================
Input:   'Ｒésumé  ﬁnancier'
         Code points: 16

Step 1 (NFKC):        'Résumé  financier'
  Full-width 'Ｒ' -> 'R', ligature 'ﬁ' -> 'fi'
  Code points: 17

Step 2 (whitespace):  'Résumé financier'
  Double space collapsed to single space
  Code points: 16

Step 3 (casefold):    'résumé financier'
  'R' -> 'r', 'R' -> 'r'
  Code points: 16

Step 4 (no accents):  'resume financier'
  'é' -> 'e' (acute accent stripped)
  Code points: 16

Let us trace exactly what each step accomplishes:

After Step 1 (NFKC), the full-width "Ｒ" (U+FF32) becomes "R" (U+0052) via the compatibility mapping, and the ligature "ﬁ" (U+FB01) expands to "fi" (U+0066, U+0069). The accented "é" stays as "é" because NFKC applies canonical composition, and the precomposed form is already canonical. The double space is preserved because NFKC does not touch whitespace.

After Step 2 (whitespace normalization), the double space between "Résumé" and "financier" is collapsed to a single space. This step must come after NFKC because NFKC may introduce or eliminate whitespace variants.

After Step 3 (case folding), "Résumé" becomes "résumé". The capital "R" and the capital accented "R" (already converted from full-width in Step 1) both fold to lowercase. This prepares the string for case-insensitive comparison.

After Step 4 (diacritic removal), "résumé" becomes "resume". The NFD decomposition separates the acute accents from the base letters "e", the Mn-category filter removes the combining accents, and NFC recomposes the result. The final string "resume financier" will now match any combination of the original variations regardless of full-width characters, ligatures, case, spacing, or accents.

This four-step pipeline is the core of accent-insensitive, case-insensitive, and ligature-transparent string matching. You can use all four steps together for maximum flexibility, or stop at any intermediate step depending on how much information loss is acceptable for your application.