Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Character Encoding › Unicode: One Code to Rule Them All

Michael BrenndoerferUpdated March 18, 20257 min read

Language AI HandbookAll chapters

Part 1 of 67Text as Data

Chapter 1Character Encoding › Unicode: One Code to Rule Them All

Reading section 3 of 13 · Unicode: One Code to Rule Them All · 7 minSection 3 of 13 · 7 min

Section 3 of 13 · 7 min read

Character Encoding › Unicode: One Code to Rule Them AllOne section · Sections in this chapter3 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

By the late 1980s, the encoding chaos had become untenable. Software companies were spending enormous effort handling multiple encodings, and data exchange remained problematic. The Unicode Consortium was incorporated in 1991, building on work that began in the late 1980s, with an ambitious goal: create a single character set that could represent every writing system ever used by humanity.

The ambition behind Unicode is easy to underestimate. The standard doesn't just cover the languages spoken by billions of people today. It includes Linear B, the ancient Mycenaean Greek writing system that went extinct around 1100 BCE. It includes Cuneiform, used in ancient Mesopotamia. It includes the Phaistos Disc signs, a Bronze Age script that still has not been fully deciphered. The Consortium works with scholars and governments, along with technology companies around the world, to ensure that every writing system, living or dead, finds a place in the standard. In this sense, Unicode is as much a cultural preservation project as a technical specification.

Unicode simplifies multilingual NLP. Before Unicode, building a multilingual text processing pipeline meant maintaining separate code paths for each encoding. After Unicode, you work in a single unified space. A model trained on Unicode text can, in principle, operate on text from any language without special accommodation for its encoding.

Unicode

Unicode is a universal character encoding standard that assigns numbers (called code points) to characters across writing systems and symbol sets, including emoji. Unicode 17.0 defines 159,801 characters covering 172 scripts.

Unicode assigns each character a unique code point, written as U+ followed by a hexadecimal number. For example:

U+0041 is 'A'
U+03B1 is 'α' (Greek alpha)
U+4E2D is '中' (Chinese character for "middle")
U+1F600 is '😀' (grinning face emoji)

In[10]:

Code

# Exploring Unicode code points
characters = [
    ("A", "Latin capital A"),
    ("é", "Latin small e with acute"),
    ("α", "Greek small alpha"),
    ("中", "CJK character for middle"),
    ("😀", "Grinning face emoji"),
    ("𝕳", "Mathematical double-struck H"),
]

# Get code point information
code_points = [(char, name, ord(char)) for char, name in characters]

Out[11]:

Console

Unicode Code Points:
------------------------------------------------------------
  A     U+0041  (     65)  Latin capital A
  é     U+00E9  (    233)  Latin small e with acute
  α     U+03B1  (    945)  Greek small alpha
  中     U+4E2D  ( 20,013)  CJK character for middle
  😀     U+1F600  (128,512)  Grinning face emoji
  𝕳     U+1D573  (120,179)  Mathematical double-struck H

The code points span a vast range. Basic Latin characters like 'A' occupy low values (under 128), while the emoji sits at over 128,000, far beyond what a single byte could represent.

Unicode PlanesLink Copied

Unicode organizes its vast character space into 17 planes, each containing 65,536 code points ( $2^{16}$ ). The first plane is by far the most important:

Plane 0 (Basic Multilingual Plane, BMP): U+0000 to U+FFFF. Contains characters for many modern scripts, common symbols, and punctuation.
Plane 1 (Supplementary Multilingual Plane): U+10000 to U+1FFFF. Historic scripts, musical notation, mathematical symbols, and emoji.
Plane 2 (Supplementary Ideographic Plane): U+20000 to U+2FFFF. Additional, mostly rare or historic CJK characters.
Plane 3 (Tertiary Ideographic Plane): U+30000 to U+3FFFF. Further CJK unified ideographs.
Planes 4-13: Reserved for future use.
Planes 14-16: Special purpose and private use.

In[12]:

Code

# Determine which plane a character belongs to
def get_plane(char):
    cp = ord(char)
    plane = cp >> 16  # Divide by 65536
    return plane


test_chars = ["A", "é", "中", "😀", "𝕳", "🎵"]
planes = [(char, get_plane(char), ord(char)) for char in test_chars]

Out[13]:

Console

Character Planes:
  'A' (U+00041) → Plane 0: Basic Multilingual Plane (BMP)
  'é' (U+000E9) → Plane 0: Basic Multilingual Plane (BMP)
  '中' (U+04E2D) → Plane 0: Basic Multilingual Plane (BMP)
  '😀' (U+1F600) → Plane 1: Supplementary Multilingual Plane (SMP)
  '𝕳' (U+1D573) → Plane 1: Supplementary Multilingual Plane (SMP)
  '🎵' (U+1F3B5) → Plane 1: Supplementary Multilingual Plane (SMP)

For most NLP work, you'll primarily encounter characters in the BMP. However, emoji (increasingly common in social media text) and certain mathematical symbols live in Plane 1, so your code must handle characters beyond the BMP correctly. In practice, this means being careful with operations that assume one 16-bit code unit per code point. Modern Python 3's str type handles supplementary characters transparently, but narrow Unicode builds used by some older Python versions represented them as surrogate pairs.

Out[14]:

Visualization

Horizontal bar chart of Unicode 17.0 character counts by plane, led by Plane 2, the BMP, Plane 1, and Plane 3. — Distribution of the 159,801 characters encoded in Unicode 17.0 across the 17 planes. Plane 2 contains the largest assigned repertoire because of its many CJK ideographs, while the BMP contains most characters encountered in everyday text. Plane 3 also contains assigned CJK ideographs; planes 4-13 remain unassigned, and planes 15-16 are reserved for private use rather than standardized characters.

Code Points vs. CharactersLink Copied

Unicode code points don't always correspond one-to-one with what humans perceive as "characters." Some visual characters can be represented multiple ways:

In[15]:

Code

# The letter 'é' can be represented two ways
# Method 1: Single precomposed character
e_acute_composed = "\u00e9"  # U+00E9: Latin small letter e with acute

# Method 2: Base character + combining mark
e_acute_decomposed = "e\u0301"  # U+0065 (e) + U+0301 (combining acute accent)

# They look identical but are different byte sequences
are_equal = e_acute_composed == e_acute_decomposed

Out[16]:

Console

Two ways to write 'é':
  Composed:   'é' = U+00E9 (1 code point)
  Decomposed: 'é' = U+0065 + U+0301 (2 code points)

Look identical? Yes
Are equal in Python? False

Length comparison:
  len(composed) = 1
  len(decomposed) = 2

Despite looking identical to human eyes, Python considers these two strings different. The composed form has length 1, while the decomposed form has length 2. This has serious implications for text processing. String comparison, length calculation, and search operations can give unexpected results. We'll address this with Unicode normalization in the next chapter.

The key insight here is that Unicode separates the concept of a character (a unit of human meaning) from the concept of a code point (a number in the Unicode table). A single visible character can require multiple code points to represent, especially when combining marks are involved. Thai vowels, Hebrew niqqud (vowel points), and Arabic diacritics all use combining characters. When you count the "length" of a string in terms of code points, you might get a number larger than the number of visible characters a reader would perceive. This distinction becomes critical when you implement operations like truncating a string to a maximum display width.