Character Encoding › Summary
Michael BrenndoerferUpdated March 18, 20251 min read
Language AI HandbookAll chapters Part 1 of 67Text as Data
Character Encoding › SummaryOne section · Sections in this chapter12 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Character encoding bridges the gap between human writing and computer storage. We traced the evolution from ASCII's 128 characters through the fragmented world of regional encodings to Unicode's universal character set and UTF-8's elegant variable-width encoding.
Key takeaways:
- ASCII uses 7 bits for 128 characters, covering only English
- Unicode 17.0 encodes 159,801 characters across 172 scripts
- UTF-8 encodes Unicode scalar values using 1-4 bytes, with ASCII compatibility and no endianness issues
- Mojibake results from decoding bytes with the wrong encoding
- Always specify encoding when reading or writing text files in Python
- Use UTF-8 as your default encoding for new projects
- Encoding detection is heuristic and imperfect; verify when possible
In the next chapter, we'll build on this foundation to explore text normalization, addressing the challenge of multiple Unicode representations for the same visual character.