Character Encoding › UTF-8: The Encoding That Won
Language AI HandbookAll chapters Character Encoding › UTF-8: The Encoding That WonOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Unicode defines a character repertoire and assigns code points. Unicode encoding forms specify how those values are represented as code units and, when serialized, bytes. The three principal Unicode Transformation Formats (UTFs) are:
- UTF-32: Uses exactly 4 bytes per code point. Simple but often wasteful.
- UTF-16: Uses 2 or 4 bytes per Unicode scalar value. Common in Windows and Java APIs.
- UTF-8: Uses 1 to 4 bytes per Unicode scalar value. Dominant on the web.
UTF-8, invented by Ken Thompson and Rob Pike in 1992, has become the de facto standard for text on the internet. As of 2024, over 98% of websites use UTF-8. The story of its invention is worth knowing: Thompson and Pike designed the encoding scheme on a placemat at a New Jersey diner in a single evening in 1992, and had a working implementation running the next day. The elegance of what they sketched out that night, an encoding that gracefully handles backward compatibility, self-synchronization, and variable width in a single unified design, shows how much a compact design can achieve under constraints.
The choice between UTF-8, UTF-16, and UTF-32 affects more than storage efficiency. It determines how your programs handle random access into strings, how files are interpreted by different tools, and how data moves across system boundaries. UTF-8's dominance means that choosing it is almost always the right default for new projects. Understanding why it won helps you make informed decisions in the rare cases where a different encoding might be preferable.
UTF-8 (Unicode Transformation Format, 8-bit) is a variable-width encoding that represents Unicode scalar values using one to four bytes. It is backward-compatible with ASCII, meaning any valid ASCII text is also valid UTF-8.
How UTF-8 Works
UTF-8's genius lies in its variable-width design. Think of it as a self-describing byte stream: when positioned at a valid leading byte, you can determine how many bytes the current code point occupies. ASCII code points use 1 byte, while higher code points use more. This design achieves two things simultaneously: it makes ASCII text indistinguishable from UTF-8 (every valid ASCII file is also valid UTF-8), and it handles the full Unicode range without wasting space on ASCII-heavy text.
The encoding rules are summarized in this table, where x positions hold the actual bits of the code point value:
| Code Point Range | Bytes | Bit Pattern | Data Bits |
|---|---|---|---|
| U+0000 to U+007F | 1 | 0xxxxxxx | 7 |
| U+0080 to U+07FF | 2 | 110xxxxx 10xxxxxx | 11 |
| U+0800 to U+FFFF, excluding U+D800 to U+DFFF | 3 | 1110xxxx 10xxxxxx 10xxxxxx | 16 |
| U+10000 to U+10FFFF | 4 | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx | 21 |
The "x" positions in the bit patterns hold the actual code point value. For example, a 2-byte character has 5 data bits in the first byte and 6 in the second, giving total data bits, which can represent values up to (U+07FF).
The templates show where data bits go, but not every apparent bit combination is valid. UTF-8 forbids overlong encodings, excludes the UTF-16 surrogate range U+D800 to U+DFFF, and stops at U+10FFFF. Consequently, valid two-byte lead bytes run from 0xC2 to 0xDF (not 0xC0 to 0xDF), and some three- and four-byte lead bytes impose tighter bounds on the next byte.
To encode a code point into UTF-8, you select the appropriate row from the table based on the code point's magnitude, then fill in the x positions with the binary representation of the code point, padding with leading zeros if necessary. To decode, you identify the pattern of leading bits in the first byte, determine how many total bytes to read, strip the header bits, and concatenate the data bits to recover the original code point. The self-describing nature of the encoding is what makes this possible without any external metadata.
The leading bits tell you how many bytes the character uses:
- If the first bit is 0, it's a 1-byte character (ASCII)
- If the first bits are 110, it's a 2-byte character
- If the first bits are 1110, it's a 3-byte character
- If the first bits are 11110, it's a 4-byte character
- Continuation bytes always start with 10
Let's see this encoding in action:
def show_utf8_encoding(char):
"""Display how a character is encoded in UTF-8."""
code_point = ord(char)
utf8_bytes = char.encode("utf-8")
# Format bytes as binary
binary = " ".join(f"{b:08b}" for b in utf8_bytes)
hex_repr = " ".join(f"{b:02x}" for b in utf8_bytes)
return {
"char": char,
"code_point": code_point,
"num_bytes": len(utf8_bytes),
"binary": binary,
"hex": hex_repr,
}
# Test with characters from different ranges
test_chars = ["A", "é", "中", "😀"]
encodings = [show_utf8_encoding(c) for c in test_chars]UTF-8 Encoding Examples: ====================================================================== Character: 'A' Code point: U+0041 (65) UTF-8 bytes: 1 Binary: 01000001 Hex: 41 Character: 'é' Code point: U+00E9 (233) UTF-8 bytes: 2 Binary: 11000011 10101001 Hex: c3 a9 Character: '中' Code point: U+4E2D (20013) UTF-8 bytes: 3 Binary: 11100100 10111000 10101101 Hex: e4 b8 ad Character: '😀' Code point: U+1F600 (128512) UTF-8 bytes: 4 Binary: 11110000 10011111 10011000 10000000 Hex: f0 9f 98 80
Look at the binary patterns. 'A' (code point 65) fits in 7 bits and uses a single byte starting with 0. The French 'é' needs 2 bytes, starting with 110. The Chinese character '中' needs 3 bytes, starting with 1110. And the emoji needs all 4 bytes, starting with 11110.
Worked Example: Encoding '中' Step by Step
Let's trace through the encoding of the Chinese character '中' (U+4E2D) manually to make the algorithm concrete. This walkthrough applies to Unicode scalar values in the BMP range U+0800 to U+FFFF, excluding the surrogate range U+D800 to U+DFFF.
Step 1: Find the code point in binary.
The code point for '中' is U+4E2D, which is 19,757 in decimal. Converting to binary:
Step 2: Select the encoding tier.
The value 19,757 falls in the range U+0800 to U+FFFF, so we need the 3-byte encoding with 16 available data bits and the bit pattern 1110xxxx 10xxxxxx 10xxxxxx.
Step 3: Distribute the code point bits across the x positions.
The 3-byte template provides 4 + 6 + 6 = 16 data bit positions. We have 16 bits from step 1, so the fit is exact. We fill in the x positions from left to right:
Template: 1110xxxx 10xxxxxx 10xxxxxx
Code point: 0100 111000 101101
Result: 11100100 10111000 10101101
Step 4: Convert to hexadecimal.
Reading the three bytes:
11100100=0xE410111000=0xB810101101=0xAD
So '中' encodes to the byte sequence E4 B8 AD in UTF-8. You can verify this with "中".encode("utf-8").hex() in Python, which produces 'e4b8ad'. Notice that each of the two continuation bytes begins with 10, confirming they are continuation bytes and not the start of a new character. This is the self-synchronizing property in action.
Why this matters for NLP: Byte-level BPE tokenizers, including the tokenizer introduced with GPT-2, begin from a reversible representation of UTF-8 bytes. Understanding that '中' produces the three-byte sequence E4 B8 AD helps explain why such a tokenizer might represent it as one merged token or split it across several tokens, depending on the merges learned from its training corpus.
Why UTF-8 Won
UTF-8's dominance follows from several practical advantages:
- ASCII compatibility: Any ASCII text is valid UTF-8 without modification. This made adoption painless for the English-speaking computing world that had decades of ASCII data.
- Self-synchronizing: You can jump into the middle of a UTF-8 stream and find character boundaries. Continuation bytes (starting with 10) are distinct from start bytes, so you can always resynchronize.
- No byte-order issues: Unlike UTF-16 and UTF-32, UTF-8 has no endianness problems. The same bytes mean the same thing on any system.
- Efficiency for ASCII-heavy text: English text and code, along with markup and many data formats, are predominantly ASCII. UTF-8 represents these with no overhead.
The self-synchronizing property deserves a deeper look because it has practical implications for text processing. If a UTF-16 stream is corrupted by a dropped byte, every subsequent character boundary shifts by one, producing cascading garbage. In UTF-8, a dropped byte in the middle of a multi-byte sequence may corrupt one character, but the decoder can resynchronize at the next start byte (which cannot begin with 10). This resilience benefits streaming text processing, network protocols, and error recovery.
The absence of byte-order ambiguity also made UTF-8 easier to deploy. In the late 1990s and early 2000s, one competing representation was UCS-2, a fixed-width predecessor to UTF-16 used by early Windows APIs. UCS-2 had endianness issues: software needed a byte-order mark or an external convention to know the byte order. UTF-8 simply worked the same way everywhere, which was a decisive advantage for the open internet.
The story of UTF-8 winning the web is also a story of Unix and open-source software. UTF-8 became the standard locale choice on Linux systems, and the growth of Linux-based web servers carried it along. By the time HTML5 became a W3C Recommendation in 2014, web standards strongly favored UTF-8 and the web had already largely converged on it through organic adoption.
# Compare encoding sizes for different types of text
texts = {
"English": "Hello, World!",
"French": "Héllo, Wörld!",
"Chinese": "你好，世界！",
"Mixed": "Hello 世界! 😀",
"Code": "def hello(): return 42",
}
sizes = {}
for name, text in texts.items():
utf8_size = len(text.encode("utf-8"))
utf16_size = len(text.encode("utf-16-le")) # Without BOM
utf32_size = len(text.encode("utf-32-le")) # Without BOM
sizes[name] = (len(text), utf8_size, utf16_size, utf32_size)Encoding Size Comparison (bytes): ----------------------------------------------------------------- Text Type Chars UTF-8 UTF-16 UTF-32 ----------------------------------------------------------------- English 13 13 26 52 French 13 15 26 52 Chinese 6 18 12 24 Mixed 11 18 24 44 Code 22 22 44 88
For the English and code samples, UTF-8 matches the code point count exactly because every character shown is ASCII and therefore uses 1 byte. Each character in this Chinese sample is in the BMP and uses 3 bytes in UTF-8, versus 2 bytes in UTF-16. CJK ideographs in supplementary planes use 4 bytes in both UTF-8 and UTF-16. UTF-32 consistently uses 4 bytes per code point, resulting in significant overhead for ASCII-heavy text.
For this Chinese sample, UTF-8 remains reasonably compact, although UTF-16 is smaller. UTF-32 maintains a fixed width per code point, at the cost of 4x overhead for ASCII. No UTF encoding has a fixed width per user-perceived character because one grapheme can contain multiple code points.