Character Encoding › Mojibake: When Encoding Goes Wrong

Michael BrenndoerferUpdated March 18, 20254 min read
Language AI HandbookAll chapters
Part 1 of 67Text as Data
Chapter 1Character Encoding › Mojibake: When Encoding Goes Wrong
Reading section 7 of 13 · Mojibake: When Encoding Goes Wrong · 4 minSection 7 of 13 · 4 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 1 of 513

Section 7 of 13 · 4 min read

Character Encoding › Mojibake: When Encoding Goes WrongOne section · Sections in this chapter7 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Mojibake

Mojibake (from Japanese 文字化け, "character transformation") refers to garbled text that results from decoding bytes using the wrong character encoding. The term describes the visual appearance of incorrectly decoded text.

Mojibake is the bane of text processing. It occurs when bytes encoded in one system are decoded using a different, incompatible encoding. The result is nonsensical characters that often follow recognizable patterns. The term itself comes from Japanese computing culture, where the phenomenon was particularly common: Japanese text encoded in Shift-JIS or EUC-JP displayed as garbage on systems expecting ASCII or a different Japanese encoding.

The insidious quality of mojibake is that it is often silent. When you decode Latin-1 bytes as UTF-8, Python frequently throws a UnicodeDecodeError because not all Latin-1 byte sequences are valid UTF-8. But when you decode UTF-8 bytes as Latin-1, there is no error at all. Latin-1 accepts all possible byte values (it is an 8-bit encoding that maps every byte to a character), so the decode operation succeeds but produces wrong characters. Your pipeline runs to completion, logs no errors, and delivers a dataset full of garbage. This is the silent failure mode that makes encoding bugs so damaging in NLP pipelines.

In practice, the most common form of mojibake on the modern web is UTF-8 text that was served or stored without declaring its encoding, and then read by older software that assumed Latin-1 or Windows-1252. The accented characters that appear frequently in European languages have multi-byte UTF-8 representations that, when misread as Windows-1252, produce characteristic two- or three-character garbage sequences. Learning to recognize these patterns is a practical skill for any NLP engineer working with web-scraped data.

In[32]:
Code
# Common mojibake patterns
original = "Héllo Wörld"

# Encode as UTF-8, decode as Latin-1 (common web error)
utf8_as_latin1 = original.encode("utf-8").decode("latin-1")

# Encode as Latin-1, decode as UTF-8 (causes errors or replacement)
try:
    latin1_as_utf8 = original.encode("latin-1").decode("utf-8")
except UnicodeDecodeError as e:
    latin1_as_utf8 = f"Error: {e}"
Out[33]:
Console
Mojibake Examples:
--------------------------------------------------
Original:                    'Héllo Wörld'
UTF-8 decoded as Latin-1:    'Héllo Wörld'
Latin-1 decoded as UTF-8:    Error: 'utf-8' codec can't decode byte 0xe9 in position 1: invalid continuation byte

The first case is particularly insidious. UTF-8's multi-byte sequences are valid Latin-1 byte sequences, so no error occurs. Instead, you get garbage like "Héllo Wörld" where each accented character becomes two or three strange characters.

Recognizing Mojibake Patterns

Different encoding mismatches produce characteristic garbage:

In[34]:
Code
# Common mojibake signatures
test_chars = ["é", "ñ", "ü", "中", "—", "“", "’"]

mojibake_examples = []
for char in test_chars:
    try:
        # UTF-8 interpreted as Windows-1252, a common web mismatch
        utf8_bytes = char.encode("utf-8")
        as_cp1252 = utf8_bytes.decode("cp1252")
        mojibake_examples.append((char, as_cp1252, len(utf8_bytes)))
    except UnicodeDecodeError:
        mojibake_examples.append((char, "Error", 0))
Out[35]:
Console
UTF-8 Mojibake Patterns (decoded as Windows-1252):
---------------------------------------------
Original   Mojibake        UTF-8 bytes
---------------------------------------------
'é'        'é'           2
'ñ'        'ñ'           2
'ü'        'ü'           2
'中'        '中'           3
'—'        '—'           3
'“'        '“'           3
'’'        '’'           3

When you see patterns like "é" for "é" or "—" for an em dash, you're probably looking at UTF-8 text that was incorrectly decoded as Latin-1 or Windows-1252. The character 'é' (U+00E9) encodes to C3 A9 in UTF-8; both Latin-1 and Windows-1252 map those bytes to 'Ã' and '©', producing the characteristic "é" signature. The em dash (U+2014) encodes as E2 80 94; Windows-1252 displays those bytes as "—". Once you learn a handful of these patterns, spotting encoding problems in a dataset becomes much easier.

Fixing Mojibake

Sometimes you can reverse mojibake by re-encoding with the wrong encoding and decoding with the right one:

In[36]:
Code
# Attempt to fix mojibake
mojibake_text = "Héllo Wörld"

# Reverse the damage: encode as Latin-1, decode as UTF-8
try:
    fixed = mojibake_text.encode("latin-1").decode("utf-8")
except (UnicodeDecodeError, UnicodeEncodeError):
    fixed = "Could not fix"
Out[37]:
Console
Mojibake Repair:
  Corrupted: 'Héllo Wörld'
  Fixed:     'Héllo Wörld'

This works because the original bytes can be reconstructed exactly from this particular mis-decoding. Mojibake becomes destructive when a conversion drops data, inserts replacement characters, or passes through a codec with undefined byte mappings; repeated conversions can compound the damage.