Character Encoding › Character Encoding

Before computers can process language, they must solve a fundamental problem: how do you represent human writing as numbers? When text is stored or transmitted, every letter and symbol, including emoji, is encoded as a sequence of bytes. Character encoding is the system that maps between human-readable text and these numerical representations. Understanding encoding is essential for NLP practitioners because encoding errors can corrupt data silently, turning meaningful text into unintelligible garbage.

Think of character encoding as the agreed-upon translation dictionary that both the sender and receiver of text must share. When your Python script reads a file, it needs to know which dictionary to use to convert those raw bytes into characters. If both parties use the same dictionary, communication succeeds. If they use different dictionaries, or if the dictionary isn't specified at all, the result is scrambled nonsense. The frustrating part is that this failure is often invisible: the code runs without errors, the text just happens to be wrong.

The stakes are higher than they might appear. A dataset of one million news articles is worthless if 30% of the non-English articles were decoded with the wrong encoding and now contain garbled characters. A language model trained on such corrupted data will learn to associate those garbled patterns with real words, producing strange artifacts in generation. Encoding errors introduced at the data pipeline stage propagate silently all the way into model weights, and by the time the problem is discovered, retraining from scratch may be the only remedy.

This chapter traces the evolution from ASCII's humble 7-bit origins through Unicode's ambitious goal of representing every writing system, and finally to UTF-8, the encoding that now dominates the web. You'll learn why encoding matters, how to detect and fix encoding problems, and how to handle text correctly in Python. The concepts here form the bedrock for every chapter that follows: tokenization and normalization both build on them, as does the byte-pair encoding used inside modern LLM tokenizers.