Text Normalization › Text Normalization and Unicode in NLP

In the previous chapter, we saw how a single character like "é" can be represented in multiple ways: as a single precomposed code point (U+00E9) or as a base letter plus a combining accent (U+0065 + U+0301). Both look identical on screen, but Python considers them different strings. This seemingly minor issue can break string matching, corrupt search results, and introduce subtle bugs into your NLP pipelines that are extraordinarily difficult to diagnose.

Text normalization is the process of transforming text into a consistent, canonical form. It goes beyond encoding to address a deeper question: when should two different byte sequences be considered the "same" text? The answer is not always obvious. For a search engine, "café" and "cafe" should probably match. For a literary database preserving original orthography, they should not. For a deduplication system, "SOCIÉTÉ GÉNÉRALE" and "Societe Generale" should be recognized as the same company name. These decisions are not technical trivia but fundamental design choices that shape how your system understands language.

Think of text normalization as the typographer's equivalent of a style guide. A style guide tells writers how to handle capitalization and hyphenation, alongside abbreviation, so that readers encounter consistent, readable prose. Normalization tells your software how to handle Unicode forms and case, plus whitespace and special characters, so that two strings that mean the same thing compare as equal. Without a shared style guide, the same piece of information can appear in dozens of incompatible forms, and your pipeline breaks down quietly on whichever variant it hasn't seen before.

The stakes are higher than they appear. Modern text comes from an enormous variety of sources: web pages scraped from servers in Tokyo, PDFs converted from scanned documents, user input typed on mobile keyboards, log files generated by services running in Frankfurt, chat messages copied and pasted from different operating systems. Each source has its own conventions for representing text, and none of them agreed in advance to use the same Unicode normalization form, the same approach to whitespace, or the same policy on ligatures and full-width characters. When you merge these sources into a training corpus or feed them into a retrieval pipeline, the inconsistencies pile up invisibly.

This chapter covers the four Unicode normalization forms (NFC, NFD, NFKC, NFKD), case folding versus simple lowercasing, accent and diacritic handling, whitespace normalization, ligature expansion, full-width to half-width conversion, and how to assemble these techniques into a reliable normalization pipeline. Each technique targets a specific class of variation. Mastering them gives you the confidence to process text from any source without silent failures.

Throughout the chapter, we will work through concrete Python examples using the standard library's unicodedata module, which gives you direct access to Unicode character properties and normalization algorithms. By the end, you will have both the conceptual framework and the practical code to handle the trickiest normalization challenges in real production systems.