Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Text Normalization › Full-Width to Half-Width Conversion

Michael BrenndoerferUpdated March 19, 20253 min read

Language AI HandbookAll chapters

Part 1 of 67Text as Data

Chapter 2Text Normalization › Full-Width to Half-Width Conversion

Reading section 8 of 15 · Full-Width to Half-Width Conversion · 3 minSection 8 of 15 · 3 min

Section 8 of 15 · 3 min read

Text Normalization › Full-Width to Half-Width ConversionOne section · Sections in this chapter8 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

East Asian text often uses full-width versions of ASCII characters. These take up the same width as CJK characters, creating visual alignment in mixed text. A product name like "ＩＢＭ" uses full-width Latin letters so that the three characters align with three CJK characters in adjacent rows. In typographic contexts, this is a deliberate design choice. In NLP contexts, it is a source of variation that prevents matching.

The full-width ASCII characters occupy the Unicode block from U+FF01 to U+FF60, directly mirroring the printable ASCII characters from U+0021 to U+0060. The mapping is a simple arithmetic offset: subtract 0xFEE0 from any full-width character to get its half-width equivalent. This regularity makes conversion straightforward, and it means you can implement the conversion without any lookup table.

In[30]:

Code

# Full-width ASCII characters
fullwidth_examples = [
    ("Ａ", "A", "Full-width A"),
    ("ａ", "a", "Full-width a"),
    ("０", "0", "Full-width 0"),
    ("！", "!", "Full-width exclamation"),
    ("　", " ", "Ideographic space"),
]


# Full-width to half-width conversion
def fullwidth_to_halfwidth(text):
    """Convert full-width ASCII to half-width."""
    result = []
    for char in text:
        code = ord(char)
        # Full-width ASCII range: U+FF01 to U+FF5E maps to U+0021 to U+007E
        if 0xFF01 <= code <= 0xFF5E:
            result.append(chr(code - 0xFF01 + 0x21))
        # Ideographic space to regular space
        elif code == 0x3000:
            result.append(" ")
        else:
            result.append(char)
    return "".join(result)


# Test conversion
fullwidth_text = "Ｈｅｌｌｏ　Ｗｏｒｌｄ！　１２３"
halfwidth_text = fullwidth_to_halfwidth(fullwidth_text)

Out[31]:

Console

Full-Width to Half-Width Conversion:
  Full-width: 'Ｈｅｌｌｏ　Ｗｏｒｌｄ！　１２３'
  Half-width: 'Hello World! 123'

Character-by-character:
  'Ｈ' (U+FF28) -> 'H' (U+0048)
  'ｅ' (U+FF45) -> 'e' (U+0065)
  'ｌ' (U+FF4C) -> 'l' (U+006C)
  'ｌ' (U+FF4C) -> 'l' (U+006C)
  'ｏ' (U+FF4F) -> 'o' (U+006F)

Each full-width character maps to its ASCII equivalent by subtracting a fixed offset (0xFEE0) from the code point. The ideographic space (U+3000) is a special case that maps to the regular space (U+0020). This conversion is essential when processing East Asian text that mixes CJK characters with Latin letters and digits. A company name like "ＡＢＣ株式会社" should match "ABC株式会社" in any reasonable search system.

NFKC normalization also handles full-width to half-width conversion automatically:

In[32]:

Code

# NFKC also converts full-width characters
nfkc_result = unicodedata.normalize("NFKC", fullwidth_text)
manual_match = nfkc_result == halfwidth_text

Out[33]:

Console

NFKC vs manual conversion:
  NFKC result:   'Hello World! 123'
  Manual result: 'Hello World! 123'
  Match: True

The NFKC normalization produces identical results to the manual conversion function, confirming that NFKC handles full-width to half-width mapping as part of its compatibility normalization. This means you can use NFKC for broad compatibility normalization without implementing character-specific conversion logic, simplifying your normalization pipeline considerably. The manual implementation is shown here to make the underlying mapping explicit, but in practice NFKC is the preferred approach because it handles all compatibility mappings in a single pass.