Text Normalization › Case Folding vs. Lowercasing
Language AI HandbookAll chapters Text Normalization › Case Folding vs. LowercasingOne section · Sections in this chapter4 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Case-insensitive comparison seems simple: just convert both strings to lowercase. But Unicode makes this surprisingly complex, and getting it wrong produces subtle bugs that are hard to track down in a multilingual system.
The intuition behind case normalization is sound: users searching for "Python" should find documents containing "PYTHON" and "python". The implementation, however, must handle dozens of edge cases where simple lowercasing fails. These cases are not obscure curiosities. They affect real languages used by hundreds of millions of people.
The key insight is that case conversion is not a simple character-by-character lookup. In some languages, the lowercase and uppercase forms of a letter are not in a one-to-one relationship. In others, the same character has different lowercase forms depending on its position in a word. And in at least one major language, the historical lowercase of a letter literally does not exist as a single code point. Python's standard lower() method handles English correctly but stumbles on these cases. The casefold() method, implementing Unicode Case Folding, handles them properly.
The Problem with Simple Lowercasing
# German sharp s (ß) uppercases to SS in standard Python
german_word = "straße" # street
lowered = german_word.lower()
uppered = german_word.upper()
round_trip = uppered.lower()Case conversion with German ß: Original: 'straße' (length 6) .lower(): 'straße' (length 6) .upper(): 'STRASSE' (length 7) Round-trip: 'strasse' (length 7) Original == round-trip? False
The German "ß" (eszett or sharp s) uppercases to "SS" (two characters), and lowercasing "SS" gives "ss", not "ß". Round-tripping through case conversion changes the string. This is not a bug; it reflects German orthographic rules where "ß" traditionally had no uppercase form. While Unicode 5.1 (2008) added the capital ẞ (U+1E9E), Python's upper() still converts to "SS" for compatibility with the traditional standard. The consequence is that "straße".upper().lower() != "straße", which means simple case normalization fails to unify these variants.
This has real consequences. If you lowercase all text before storing in a search index, then a user searching for "Straße" will find documents containing "straße", "Straße", and "STRASSE" (because all lowercase to "straße" or "strasse" depending on the input form), but the results may be inconsistent depending on which lowercased forms happen to match.
Case Folding
Case folding is a Unicode operation designed for case-insensitive comparison. Unlike simple lowercasing, case folding handles language-specific mappings and ensures that equivalent strings compare equal regardless of their original case. Unicode Case Folding is defined in Unicode Standard Annex #21 and the CaseFolding.txt data file. Python's str.casefold() method implements the full Unicode case folding algorithm, which is a superset of lowercasing.
Python's str.casefold() method implements Unicode case folding. The key difference from lower() is that casefold() applies the full set of Unicode case folding mappings, including the mapping of "ß" to "ss". This means that "straße", "Straße", "STRASSE", "strasse", and "STRAßE" all casefold to the same string, enabling correct case-insensitive comparison.
# Compare lower() vs casefold()
words = ["Straße", "STRASSE", "straße", "strasse"]
lower_results = [w.lower() for w in words]
casefold_results = [w.casefold() for w in words]Comparing lower() vs casefold(): -------------------------------------------------- Word lower() casefold() -------------------------------------------------- Straße straße strasse STRASSE strasse strasse straße straße strasse strasse strasse strasse Case-insensitive matches: Using lower(): 2 distinct values Using casefold(): 1 distinct values
With casefold(), all four variations of "street" in German normalize to the same string, enabling correct case-insensitive comparison. With lower(), you get two distinct values because "STRASSE" lowercases to "strasse" but "Straße" lowercases to "straße". A deduplication system using lower() would incorrectly treat these as different companies or different documents.
The following table shows how many distinct strings remain after applying lower() versus casefold() to groups of equivalent words. The ideal is 1 (all variants unified to a single canonical form):
| Word Group | Variants | lower() distinct | casefold() distinct |
|---|---|---|---|
| German "street" | Straße, STRASSE, straße, strasse, STRAßE | 2 | 1 |
| German "size" | Größe, GRÖSSE, größe, groesse, GROESSE | 3 | 2 |
| Greek sigma | σ, ς, Σ | 2 | 1 |
| Mixed case | Hello, HELLO, hello, HeLLo | 1 | 1 |
| Turkish I | Istanbul, ISTANBUL, istanbul, İstanbul | 2 | 2 |
For German words with "ß", casefold() correctly unifies all variants to a single canonical form, while lower() leaves two distinct values. Greek sigma variants (σ, ς, Σ) are particularly interesting: casefold() maps them all to the same form, recognizing that they represent the same letter in different positions. Standard English words show identical behavior for both methods, confirming that casefold() is a strict superset of lower() functionality. Note that Turkish requires locale-aware handling for correct dotted/dotless I normalization, which neither method provides automatically.
Language-Specific Case Rules
Some case conversions depend on language context. The most prominent example is Turkish, which has four forms of the letter "I" rather than two:
# Turkish dotted and dotless i
turkish_examples = [
("I", "English uppercase I"),
("i", "English lowercase i"),
("İ", "Turkish uppercase dotted I (U+0130)"),
("ı", "Turkish lowercase dotless i (U+0131)"),
]
# Standard Python case conversion (not locale-aware)
case_results = [
(char, desc, char.lower(), char.upper(), char.casefold())
for char, desc in turkish_examples
]Turkish I variants and case conversion: --------------------------------------------------------------------------- Char Description lower upper casefold --------------------------------------------------------------------------- 'I' English uppercase I 'i' 'I' 'i' 'i' English lowercase i 'i' 'I' 'i' 'İ' Turkish uppercase dotted I (U+0130) 'i̇' 'İ' 'i̇' 'ı' Turkish lowercase dotless i (U+0131) 'ı' 'I' 'ı'
In Turkish, the correct case pairs are: uppercase dotted "İ" (U+0130) pairs with lowercase dotted "i" (U+0069), and uppercase dotless "I" (U+0049) pairs with lowercase dotless "ı" (U+0131). Python's default case operations follow English rules, where "I" lowercases to "i" and "i" uppercases to "I". This means that in Turkish text, "İstanbul".lower() produces "istanbul" instead of the correct "istanbul" (by coincidence the same here), but more problematically, "ISTANBUL".lower() produces "istanbul" when it should produce "ıstanbul" with a dotless i. For locale-aware case conversion in Turkish, you need specialized libraries or explicit character mapping tables. This is a known limitation of Python's built-in string methods and affects any NLP pipeline processing Turkish text.