Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Character Encoding › Byte Order Marks and Endianness

Michael BrenndoerferUpdated March 18, 20253 min read

Language AI HandbookAll chapters

Part 1 of 67Text as Data

Chapter 1Character Encoding › Byte Order Marks and Endianness

Reading section 5 of 13 · Byte Order Marks and Endianness · 3 minSection 5 of 13 · 3 min

Section 5 of 13 · 3 min read

Character Encoding › Byte Order Marks and EndiannessOne section · Sections in this chapter5 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When using multi-byte encodings like UTF-16 or UTF-32, a question arises: which byte comes first? Consider the code point U+FEFF. In UTF-16, this could be stored as either:

FE FF (big-endian, most significant byte first)
FF FE (little-endian, least significant byte first)

To understand why this matters, consider the analogy of reading a two-digit number. The number 42 written as "4, then 2" is straightforward. But if someone hands you two separate slips of paper with digits, you need to know which slip is the tens place and which is the ones place. Multi-byte encodings face the same problem: a two-byte value like 00 41 could mean code point 65 (big-endian, the letter 'A') or code point 16640 (little-endian, a completely different character). The byte order determines the correct interpretation.

Byte Order Mark (BOM)

A Byte Order Mark is a special Unicode character (U+FEFF) placed at the beginning of a text file to indicate the byte order (endianness) of the encoding. In UTF-8, it serves only as an encoding signature since UTF-8 has no endianness issues.

When U+FEFF appears at the start of a byte stream, it can serve as a BOM. Its byte sequence lets readers distinguish UTF-16LE from UTF-16BE (and likewise the UTF-32 byte orders). The historical use of U+FEFF as a zero-width no-break space inside text is deprecated in favor of U+2060 WORD JOINER.

In[24]:

Code

# Different encodings and their BOMs
text = "Hello"

utf8_bom = text.encode("utf-8-sig")
utf16_le = text.encode("utf-16-le")
utf16_be = text.encode("utf-16-be")
utf16_with_bom = text.encode("utf-16")  # Includes BOM

Out[25]:

Console

Encoding 'Hello' with different byte orders:

UTF-8 with BOM: ef bb bf 48 65 6c 6c 6f
  BOM bytes: ef bb bf (UTF-8 signature)

UTF-16-LE (no BOM): 48 00 65 00 6c 00 6c 00 6f 00
UTF-16-BE (no BOM): 00 48 00 65 00 6c 00 6c 00 6f
UTF-16 (with BOM):  ff fe 48 00 65 00 6c 00 6c 00 6f 00
  BOM bytes on this system: ff fe

UTF-8 technically doesn't need a BOM since it has no byte-order ambiguity. However, Microsoft tools often add a UTF-8 BOM (EF BB BF) to indicate the file is UTF-8 rather than some other encoding. This can cause problems with Unix tools that don't expect it.

In[26]:

Code

# The UTF-8 BOM can cause subtle bugs
utf8_with_bom = b"\xef\xbb\xbfHello"
utf8_no_bom = b"Hello"

# Decoding both
decoded_with_bom = utf8_with_bom.decode("utf-8")
decoded_no_bom = utf8_no_bom.decode("utf-8")

# Using utf-8-sig to handle BOM automatically
decoded_sig = utf8_with_bom.decode("utf-8-sig")

Out[27]:

Console

BOM handling in Python:
  With BOM (utf-8):     'Hello' (length 6)
  Without BOM:          'Hello' (length 5)
  With BOM (utf-8-sig): 'Hello' (length 5)

The BOM appears as an invisible character at the start!
First char with BOM: U+FEFF (Zero Width No-Break Space)

When reading a file that you otherwise expect to be UTF-8, using utf-8-sig instead of utf-8 accepts an optional BOM and removes it from the decoded string.