Text Normalization › Whitespace Normalization
Language AI HandbookAll chapters Text Normalization › Whitespace NormalizationOne section · Sections in this chapter6 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Whitespace seems simple, but Unicode defines many whitespace characters beyond the familiar space and tab. This variety of whitespace characters is not accidental: different writing systems and typographic traditions need different kinds of space. Mathematical typesetting needs thin spaces between numbers and units. East Asian typography needs ideographic spaces that align with CJK characters. Web typography needs non-breaking spaces to prevent line breaks at specific points. But from a text processing perspective, most of these variants are unwanted noise that should be normalized to a standard space character.
The practical impact of whitespace diversity is more significant than it might appear. When text is copied from a PDF, the PDF reader often substitutes non-breaking spaces (U+00A0) for regular spaces to preserve line breaks from the original document. When text is extracted from HTML, zero-width spaces (U+200B) used for word-wrapping hints in Asian text may be included. When text comes from a web form, the user may have copy-pasted content with various Unicode space characters. Without explicit normalization, these invisible characters cause string matching failures and corrupt tokenization.
# Unicode whitespace characters
whitespace_chars = [
("\u0020", "Space"),
("\u00a0", "No-Break Space"),
("\u2002", "En Space"),
("\u2003", "Em Space"),
("\u2009", "Thin Space"),
("\u200b", "Zero Width Space"),
("\u3000", "Ideographic Space"),
("\t", "Tab"),
("\n", "Newline"),
("\r", "Carriage Return"),
]
# Check which are detected by str.isspace()
space_check = [(char, name, char.isspace()) for char, name in whitespace_chars]Unicode Whitespace Characters:
-------------------------------------------------------
Char Code Name isspace()
-------------------------------------------------------
U+0020 Space True
\xa0 U+00A0 No-Break Space True
\u2002 U+2002 En Space True
\u2003 U+2003 Em Space True
\u2009 U+2009 Thin Space True
\u200b U+200B Zero Width Space False
\u3000 U+3000 Ideographic Space True
\t U+0009 Tab True
\n U+000A Newline True
\r U+000D Carriage Return TrueNotice that the zero-width space (U+200B) is not considered whitespace by Python's isspace(). These invisible characters can cause subtle bugs: a string that appears to contain only a word may contain a zero-width space that prevents it from matching. Splitting on whitespace will not remove it. String equality checks will fail. Vocabulary lookups will miss it. The zero-width space is particularly common in text extracted from web pages and mobile applications where it is used as a word boundary hint.
The following table shows the UTF-8 byte sizes and isspace() behavior for common Unicode whitespace characters:
| Character | Code Point | UTF-8 Bytes | isspace() |
|---|---|---|---|
| Space | U+0020 | 1 | ✓ |
| Tab | U+0009 | 1 | ✓ |
| Newline | U+000A | 1 | ✓ |
| Carriage Return | U+000D | 1 | ✓ |
| No-Break Space | U+00A0 | 2 | ✓ |
| En Space | U+2002 | 3 | ✓ |
| Em Space | U+2003 | 3 | ✓ |
| Thin Space | U+2009 | 3 | ✓ |
| Zero Width Space | U+200B | 3 | ✗ |
| Ideographic Space | U+3000 | 3 | ✓ |
The byte size variation has practical implications. A document using ideographic spaces (common in CJK text) will be larger than one using standard ASCII spaces. Zero-width characters, despite being invisible, still consume 3 bytes each in UTF-8, and they can accumulate when copying text from web pages or PDFs. Note that zero-width space is not detected by isspace(), which can cause subtle matching bugs in any code that relies on this method to identify word boundaries.
Normalizing Whitespace
An effective whitespace normalizer must handle several distinct problems: it needs to map the many variants of whitespace to a standard form, remove invisible characters that are not recognized as whitespace, normalize line endings across operating systems, and optionally collapse sequences of whitespace into single characters.
An effective whitespace normalizer should:
- Convert all whitespace variants to standard spaces
- Collapse multiple spaces into one
- Strip leading and trailing whitespace
- Optionally handle zero-width characters
import re
def normalize_whitespace(text, collapse=True, strip=True):
"""Normalize various whitespace characters to standard spaces."""
# Unicode whitespace pattern (broader than \s)
# Includes all Unicode Zs (space separator) category
whitespace_pattern = r"[\u0020\u00A0\u1680\u2000-\u200A\u202F\u205F\u3000]"
# Replace all whitespace variants with standard space
text = re.sub(whitespace_pattern, " ", text)
# Handle zero-width characters
text = re.sub(r"[\u200B\u200C\u200D\uFEFF]", "", text)
# Normalize line endings
text = re.sub(r"\r\n|\r", "\n", text)
if collapse:
# Collapse multiple spaces to single space
text = re.sub(r" +", " ", text)
# Collapse multiple newlines to double newline (paragraph break)
text = re.sub(r"\n{3,}", "\n\n", text)
if strip:
text = text.strip()
return text
# Test with messy whitespace
messy_text = "Hello\u00a0\u00a0World\u200b!\u3000\u3000Test"
cleaned = normalize_whitespace(messy_text)Whitespace Normalization: Original: 'Hello\xa0\xa0World\u200b!\u3000\u3000Test' Length: 20 Cleaned: 'Hello World! Test' Length: 17
The normalizer reduced the string by converting the various Unicode spaces (no-break space, ideographic space) to standard spaces, removing the zero-width space entirely, and collapsing consecutive spaces into single spaces. This produces consistent, predictable whitespace that will not cause matching failures. The Unicode character ranges in the regex patterns cover the full set of space separators defined in the Unicode General Category "Zs", plus the zero-width and directional formatting characters that frequently appear in scraped web text.