Text Normalization › Accent and Diacritic Handling
Language AI HandbookAll chapters Text Normalization › Accent and Diacritic HandlingOne section · Sections in this chapter5 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Many NLP applications benefit from accent-insensitive matching. A user searching for "resume" should probably find "résumé". A name matching system should recognize "Muller" as a plausible match for "Müller". A search engine indexing academic papers should find "naive Bayes" when the user searches for "naïve Bayes". These are all fundamentally the same problem: bridging the gap between text that has been stripped of diacritical marks and text that preserves them.
Think of diacritic removal as a controlled lossy compression. You are deliberately discarding information, the accent marks, in exchange for broader matching. The tradeoff is acceptable for search and comparison, where users often omit accents when typing quickly, but it is destructive for language understanding tasks where the accents carry meaning. French "où" (where) and "ou" (or) are different words. Spanish "año" (year) and "ano" (anatomy) are different words. German "schön" (beautiful) and "schon" (already) are different words. The same operation that improves search recall in one context actively hurts accuracy in another.
Removing Diacritics
The standard approach uses NFD normalization followed by filtering. The algorithm has an elegant three-step structure: decompose the text so that every accent becomes a separate combining mark, filter out all combining marks, then recompose the remaining characters. This works because NFD guarantees that every diacritical mark is represented as an explicit code point with Unicode category "Mn" (Mark, Nonspacing), making filtering straightforward.
import unicodedata
def remove_diacritics(text):
"""Remove diacritical marks from text."""
# Decompose into base characters and combining marks
decomposed = unicodedata.normalize("NFD", text)
# Filter out combining marks (category 'Mn' = Mark, Nonspacing)
filtered = "".join(c for c in decomposed if unicodedata.category(c) != "Mn")
# Recompose any remaining sequences
return unicodedata.normalize("NFC", filtered)
# Test with various accented text
test_texts = [
"résumé",
"naïve",
"Ñoño",
"Zürich",
"Ångström",
]
stripped = [(text, remove_diacritics(text)) for text in test_texts]Diacritic Removal: ----------------------------------- Original Stripped ----------------------------------- résumé resume naïve naive Ñoño Nono Zürich Zurich Ångström Angstrom
This technique decomposes accented characters into base letters plus combining marks, removes the marks, and recomposes. The result is plain ASCII-compatible text. The final recompose step (applying NFC) is a housekeeping measure: in most cases there is nothing left to compose after removing the combining marks, but it ensures the output is in canonical form regardless.
The key insight behind this technique is that Unicode category "Mn" (Mark, Nonspacing) is designed specifically for combining diacritical marks. By filtering on this category, you remove exactly the right set of characters without needing to maintain a lookup table of accented characters. Any new accented character added to Unicode in a future version will automatically be handled correctly, as long as it uses combining marks (which is the standard practice).
Preserving Semantic Distinctions
Be careful: removing diacritics can change meaning in some languages, sometimes dramatically.
# Diacritics that change meaning
semantic_examples = [
("père", "father (French)"),
("pêre", "would be meaningless"),
("año", "year (Spanish)"),
("ano", "anus (Spanish)"),
("für", "for (German)"),
("fur", "different word"),
]When diacritics matter: -------------------------------------------------- père -> pere (father (French)) pêre -> pere (would be meaningless) año -> ano (year (Spanish)) ano -> ano (anus (Spanish)) für -> fur (for (German)) fur -> fur (different word)
For search applications, you might want to match both forms. For translation or language understanding, preserving diacritics is essential. A practical approach is to build a search index that contains both the original and the diacritic-stripped form of each document, so that queries with or without accents both retrieve relevant results. This is sometimes called "accent folding" in search engine terminology.
The collision rate when stripping diacritics varies dramatically by language:
| Language | Collision Rate | Collisions / Total Words | Example Collisions |
|---|---|---|---|
| Spanish | 50% | 10 / 20 | año/ano (year/anus), sí/si (yes/if) |
| French | 47% | 7 / 15 | où/ou (where/or), côte/cote (coast/quote) |
| German | 50% | 6 / 12 | schön/schon (beautiful/already), drücken/drucken (press/print) |
| Portuguese | 50% | 6 / 12 | pôde/pode (could/can), pôr/por (put/by) |
| English | 50% | 5 / 10 | résumé/resume, café/cafe |
Spanish and French use diacritics extensively to distinguish word meanings. German umlauts often differentiate semantically different words. English treats diacritics largely as optional styling for loanwords, but stripping them still causes collisions with the non-diacritical forms. These numbers show why blanket diacritic removal can be problematic for multilingual NLP applications. The right approach depends on your task: for cross-lingual information retrieval, diacritic folding improves recall but hurts precision. For named entity recognition, it may cause the model to lose important orthographic features.