Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Text Normalization › Practical Example: Deduplication

Michael BrenndoerferUpdated March 19, 20255 min read

Language AI HandbookAll chapters

Part 1 of 67Text as Data

Chapter 2Text Normalization › Practical Example: Deduplication

Reading section 11 of 15 · Practical Example: Deduplication · 5 minSection 11 of 15 · 5 min

Section 11 of 15 · 5 min read

Text Normalization › Practical Example: DeduplicationOne section · Sections in this chapter11 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let us apply normalization to a real task: finding duplicate entries in a dataset. Company name deduplication is a classic problem in data cleaning that illustrates why normalization must be consistent. A company's name may appear in dozens of forms across databases and documents, as well as data entry systems.

In[40]:

Code

# Simulated dataset with near-duplicate entries
company_names = [
    "Société Générale",
    "SOCIÉTÉ GÉNÉRALE",
    "Societe Generale",
    "Société  Générale",  # Extra space
    "Ｓｏｃｉｅｔｅ　Ｇｅｎｅｒａｌｅ",  # Full-width
    "Apple Inc.",
    "Apple Inc",
    "APPLE INC.",
    "apple inc",
    "Müller GmbH",
    "Mueller GmbH",
    "MÜLLER GMBH",
    "Muller GmbH",
]

# Create a normalizer for deduplication
dedup_normalizer = TextNormalizer(
    unicode_form="NFKC",
    casefold=True,
    strip_accents=True,
    normalize_whitespace=True,
)

# Group by normalized form
from collections import defaultdict

groups = defaultdict(list)
for name in company_names:
    normalized = dedup_normalizer(name)
    groups[normalized].append(name)

Out[41]:

Console

Duplicate Detection Results:
============================================================

Normalized form: 'societe generale'
  Matches (5):
    - Société Générale
    - SOCIÉTÉ GÉNÉRALE
    - Societe Generale
    - Société  Générale
    - Ｓｏｃｉｅｔｅ　Ｇｅｎｅｒａｌｅ

Normalized form: 'apple inc.'
  Matches (2):
    - Apple Inc.
    - APPLE INC.

Normalized form: 'apple inc'
  Matches (2):
    - Apple Inc
    - apple inc

Normalized form: 'muller gmbh'
  Matches (3):
    - Müller GmbH
    - MÜLLER GMBH
    - Muller GmbH

The normalizer correctly groups variations of "Société Générale" and "Apple Inc." together. It also groups "Müller" with "Mueller" since stripping accents converts "ü" to "u". Notice that "Mueller GmbH" and "Muller GmbH" end up in the same group: "Mueller" with stripped accents becomes "mueller" (the umlaut "ü" would strip to "u"), and "Muller" after stripping is already "muller". This is correct behavior for deduplication: these names are likely referring to the same company in different transliteration conventions.

Out[42]:

Visualization

Bar chart showing decreasing numbers of unique strings as cumulative normalization steps are applied, from 13 raw entries to 5 fully normalized forms. — Deduplication effectiveness with progressive normalization. Each bar shows how many unique strings remain after applying cumulative normalization steps. Starting with 13 raw entries, aggressive normalization reduces this to 5 unique canonical forms, identifying most near-duplicates (remaining differences are due to punctuation and ü/ue spelling variants).

The visualization shows how each normalization step progressively reduces the number of unique strings. Raw text shows 13 distinct entries, but after full normalization, only 5 unique strings remain. Each step contributes to duplicate detection: NFKC handles full-width characters, whitespace normalization catches extra spaces, case folding unifies capitalization variants, and accent stripping converts "ü" to "u". The remaining 5 strings differ due to punctuation ("Apple Inc." vs "Apple Inc") and German spelling conventions ("Mueller" as the ue-spelling vs "Müller" which becomes "muller" after accent stripping).

The table below traces the transformation of "Société Générale" through a complete normalization pipeline:

Progressive normalization pipeline transforming "Société Générale" to a canonical form.
Stage	Output	Effect
Raw Input	"Société Générale"	Various encodings and representations
Unicode (NFKC)	"Société Générale"	Canonical form, ligatures expanded
Whitespace	"Société Générale"	Spaces collapsed, zero-width removed
Case Fold	"société générale"	Case-insensitive comparison ready
Strip Accents	"societe generale"	Accent-insensitive matching ready

Each stage addresses a specific type of variation. The final output is a canonical form suitable for search and comparison. All representations of this company name will normalize to the same string, enabling reliable deduplication and retrieval regardless of how the name was entered or what system produced it.