Text Normalization › Building a Normalization Pipeline
Language AI HandbookAll chapters Text Normalization › Building a Normalization PipelineOne section · Sections in this chapter10 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Real-world text normalization combines multiple techniques. The order of operations matters: normalizing Unicode form before case folding ensures that Unicode-composed characters are in canonical form before the case algorithm runs. Stripping accents via NFD before casefolding ensures that combining marks are visible to the filter. Collapsing whitespace after Unicode normalization catches any spaces introduced or normalized by the Unicode step.
import re
import unicodedata
class TextNormalizer:
"""A configurable text normalization pipeline."""
def __init__(
self,
unicode_form="NFC",
lowercase=False,
casefold=False,
strip_accents=False,
normalize_whitespace=True,
strip_control_chars=True,
):
"""
Initialize the normalizer with configuration options.
Parameters:
- unicode_form: 'NFC', 'NFD', 'NFKC', 'NFKD', or None
- lowercase: Apply str.lower()
- casefold: Apply str.casefold() (overrides lowercase)
- strip_accents: Remove diacritical marks
- normalize_whitespace: Collapse and standardize whitespace
- strip_control_chars: Remove control characters
"""
self.unicode_form = unicode_form
self.lowercase = lowercase
self.casefold = casefold
self.strip_accents = strip_accents
self.normalize_whitespace = normalize_whitespace
self.strip_control_chars = strip_control_chars
def __call__(self, text):
"""Apply the normalization pipeline to text."""
# Step 1: Unicode normalization (first pass)
if self.unicode_form:
text = unicodedata.normalize(self.unicode_form, text)
# Step 2: Strip accents (requires NFD decomposition)
if self.strip_accents:
text = unicodedata.normalize("NFD", text)
text = "".join(c for c in text if unicodedata.category(c) != "Mn")
# Recompose after stripping
if self.unicode_form in ("NFC", "NFKC"):
text = unicodedata.normalize("NFC", text)
# Step 3: Case normalization
if self.casefold:
text = text.casefold()
elif self.lowercase:
text = text.lower()
# Step 4: Control character removal
if self.strip_control_chars:
# Remove C0, C1 controls except whitespace
text = "".join(
c
for c in text
if unicodedata.category(c) != "Cc" or c in "\t\n\r"
)
# Step 5: Whitespace normalization
if self.normalize_whitespace:
# Convert various spaces to regular space
text = re.sub(r"[\u00A0\u2000-\u200A\u202F\u205F\u3000]", " ", text)
# Remove zero-width characters
text = re.sub(r"[\u200B-\u200D\uFEFF]", "", text)
# Collapse multiple spaces
text = re.sub(r" +", " ", text)
# Normalize line endings and collapse multiple newlines
text = re.sub(r"\r\n|\r", "\n", text)
text = re.sub(r"\n{3,}", "\n\n", text)
text = text.strip()
return text
# Create different normalizers for different use cases
search_normalizer = TextNormalizer(
unicode_form="NFKC",
casefold=True,
strip_accents=True,
normalize_whitespace=True,
)
storage_normalizer = TextNormalizer(
unicode_form="NFC", normalize_whitespace=True
)Normalization Pipeline Comparison: ============================================================ Original: ' Ｈéllo\xa0\xa0Wörld！ finance ' Search normalizer (aggressive): Result: 'hello world! finance' Storage normalizer (conservative): Result: 'Ｈéllo Wörld！ finance'
The two normalizers produce materially different outputs from the same input. The search normalizer aggressively transforms the text for maximum matching flexibility: it converts full-width characters to ASCII, strips accents, folds case, expands the ligature "fi" to "fi", and collapses all whitespace variants. The storage normalizer preserves the original character forms while only standardizing whitespace, maintaining the text's visual fidelity for display purposes. This illustrates a core design principle: normalization is not one-size-fits-all. You should choose the level of aggressiveness that matches your application's requirements, and you should often maintain two separate normalized forms, one for storage and display, another for search and comparison.
Pipeline Order Matters
The order of normalization steps can affect results in subtle ways. While the most common operations are commutative for simple ASCII text, they can interact unexpectedly with complex Unicode sequences.
# Demonstrate order dependency
text = "CAFÉ"
# Order 1: Lowercase then strip accents
order1 = text.lower()
order1 = "".join(
c
for c in unicodedata.normalize("NFD", order1)
if unicodedata.category(c) != "Mn"
)
# Order 2: Strip accents then lowercase
order2 = "".join(
c
for c in unicodedata.normalize("NFD", text)
if unicodedata.category(c) != "Mn"
)
order2 = order2.lower()Order of Operations: Original: 'CAFÉ' Lowercase -> Strip accents: 'cafe' Strip accents -> Lowercase: 'cafe' Same result? True
In this case, the order does not matter for simple accented letters. But with more complex transformations involving case-sensitive patterns or locale-specific rules, order can be significant. Consider a string that contains both "ß" and an accented capital: casefolding first produces "ss" from "ß", then accent stripping removes the accent; going in the other order may interact differently depending on the normalization form applied. Always test your pipeline with representative data from all the languages and scripts you expect to process.