Character Encoding › Practical Encoding in Python
Language AI HandbookAll chapters Character Encoding › Practical Encoding in PythonOne section · Sections in this chapter8 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Python 3 made a fundamental change from Python 2: strings are Unicode by default. The str type holds Unicode code points, while bytes holds raw byte sequences. Converting between them requires explicit encoding and decoding.
This distinction matters more than it might appear to newcomers. In Python 2, the str type was a byte string, and mixing byte strings with unicode strings produced confusing implicit coercions that led to hard-to-debug errors. Python 3's strict separation between str (text) and bytes (binary data) forces you to be explicit about encoding boundaries. The encoding happens at the point where text enters or leaves your program: when reading from files, when receiving HTTP responses, or when communicating over a network socket. Inside your program, everything is Unicode, and you never need to worry about which encoding is in use.
Think of the Python 3 model as having a strict customs border between the "text world" and the "bytes world." Every crossing requires an explicit declaration: you must say which encoding to use when converting from bytes to text (decoding) and from text to bytes (encoding). The cost of this explicitness is a few extra keystrokes. The benefit is that encoding bugs surface immediately as exceptions rather than silently corrupting your data.
Encoding and Decoding
# String (Unicode) to bytes (encoding)
text = "Hello, 世界!"
utf8_bytes = text.encode("utf-8")
utf16_bytes = text.encode("utf-16")
# Bytes to string (decoding)
decoded = utf8_bytes.decode("utf-8")
# Type checking
text_type = type(text)
bytes_type = type(utf8_bytes)Python String/Bytes Types: text = 'Hello, 世界!' type(text) = <class 'str'> utf8_bytes = b'Hello, \xe4\xb8\x96\xe7\x95\x8c!' type(utf8_bytes) = <class 'bytes'> decoded = 'Hello, 世界!' text == decoded: True
Error Handling
What happens when encoding or decoding fails? Python offers several error handling strategies:
# Text that can't be encoded in ASCII
text = "Hello, 世界!"
# Different error handling strategies
strategies = {
"strict": None, # Raises exception
"ignore": text.encode("ascii", errors="ignore"),
"replace": text.encode("ascii", errors="replace"),
"xmlcharrefreplace": text.encode("ascii", errors="xmlcharrefreplace"),
"backslashreplace": text.encode("ascii", errors="backslashreplace"),
}
# Try strict mode
try:
strategies["strict"] = text.encode("ascii", errors="strict")
except UnicodeEncodeError as e:
strategies["strict"] = f"Error: {e}"Encoding 'Hello, 世界!' to ASCII with different error handlers: ------------------------------------------------------------ strict : Error: 'ascii' codec can't encode characters in position 7-8: ordinal not in range(128) ignore : b'Hello, !' replace : b'Hello, ??!' xmlcharrefreplace : b'Hello, 世界!' backslashreplace : b'Hello, \\u4e16\\u754c!'
Each strategy handles unencodable characters differently. The strict mode raises an exception, forcing you to handle the problem explicitly. The ignore mode silently drops characters, which can corrupt your data. The replace mode substitutes question marks, making problems visible. The xmlcharrefreplace and backslashreplace modes preserve recognizable code point information in an ASCII representation, which is useful for diagnostics but is not transparent byte-for-byte round-tripping.
For NLP work, errors='replace' or errors='ignore' can be useful when processing noisy data, but be aware that you're losing information. The surrogateescape error handler is particularly useful for round-tripping binary data that might contain encoding errors. It encodes undecodable bytes as Unicode surrogate characters (in the range U+DC80 to U+DCFF) that survive the round-trip and can be converted back to the original bytes. This is useful when you need to process files that are mostly text but contain some binary garbage, and you want to preserve all bytes faithfully without crashing.
In practice, choose your error handler based on the consequences of data loss. When building a training dataset, the default strict behavior is often the right choice: fail loudly on bad encoding and handle the exception at the ingestion boundary. When a production path must continue, replace makes damage visible in the output. Use ignore only when silently dropping characters is an explicit, acceptable tradeoff.
Reading and Writing Files
Always specify encoding when opening text files:
import os
import tempfile
# Create a temporary file to demonstrate
temp_dir = tempfile.mkdtemp()
file_path = os.path.join(temp_dir, "test.txt")
# Write with explicit encoding
text = "Héllo, 世界! 🌍"
with open(file_path, "w", encoding="utf-8") as f:
f.write(text)
# Read with explicit encoding
with open(file_path, "r", encoding="utf-8") as f:
read_text = f.read()
# What happens without encoding specification?
# Python normally uses a locale-dependent default, or UTF-8 in UTF-8 Mode.
import locale
default_encoding = locale.getpreferredencoding(False)
# Clean up
os.remove(file_path)
os.rmdir(temp_dir)File I/O with Encoding: Written: 'Héllo, 世界! 🌍' Read: 'Héllo, 世界! 🌍' Match: True System default encoding: UTF-8
The file round-trips correctly because we specified encoding='utf-8' explicitly. Without this, Python normally uses a locale-dependent default (or UTF-8 when Python's UTF-8 Mode is enabled), so behavior can differ across environments.
Processing Text Data
When building NLP pipelines, handle encoding at the boundaries:
def safe_read_text(file_path, fallback_encoding=None, min_confidence=0.7):
"""
Read text as BOM-marked UTF-8 or strict UTF-8, then use a sufficiently
confident detector result. An explicit fallback is optional.
"""
raw = Path(file_path).read_bytes()
if raw.startswith(b"\xef\xbb\xbf"):
return raw.decode("utf-8-sig"), "utf-8-sig"
try:
return raw.decode("utf-8"), "utf-8"
except UnicodeDecodeError:
pass
detected = chardet.detect(raw)
encoding = detected.get("encoding")
confidence = detected.get("confidence") or 0.0
if encoding is not None and confidence >= min_confidence:
return raw.decode(encoding), encoding
if fallback_encoding is not None:
return raw.decode(fallback_encoding), fallback_encoding
raise UnicodeError(
f"Encoding uncertain: {encoding!r} at {confidence:.0%} confidence"
)
# Example: process a batch of files
def normalize_encoding(text):
"""Ensure text is properly normalized Unicode."""
import unicodedata
# Normalize to NFC (composed form)
return unicodedata.normalize("NFC", text)The safe_read_text function handles an optional UTF-8 BOM, then tries strict UTF-8, and accepts a heuristic result only above a confidence threshold. A caller with reliable domain knowledge can supply a legacy fallback explicitly. Blindly trying Latin-1 before other fallbacks would be a bug because Latin-1 can decode every possible byte sequence, making all later branches unreachable. The normalize_encoding function ensures consistent Unicode representation after reading, which we'll explore in the next chapter.