Byte Pair Encoding › The BPE Decoding Algorithm
Language AI HandbookAll chapters Byte Pair Encoding › The BPE Decoding AlgorithmOne section · Sections in this chapter5 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Decoding is trivial: concatenate all tokens, removing the </w> marker (or using the boundary marker to infer where spaces belong). There is no ambiguity in reconstruction because BPE encoding is a deterministic, lossless transformation. Every encoding step can be reversed by joining the characters that were separated, so the original text is always perfectly recoverable.
This losslessness is a property BPE inherits from its data compression origins. In compression, you need to be able to reconstruct the original input exactly. In tokenization, this means that no information is lost during the encode-decode cycle, which is important for tasks like text generation where the model's output tokens must be converted back to readable text.
def decode_tokens(tokens):
"""Join BPE tokens back into the original string."""
return "".join(tokens)
# Round-trip test
test_string = "renew"
encoded = encode_word(test_string, merge_rules)
decoded = decode_tokens(encoded)Original : renew Encoded : ['r', 'e', 'ne', 'w'] Decoded : renew Lossless : True
In production tokenizers, word boundary information is encoded using a prefix convention rather than an end-of-word marker. GPT-2 uses a special Unicode character (, U+0120) prepended to any token that begins a word. This lets the decoder reconstruct spaces: wherever a appears, insert a space before that token. SentencePiece uses the underscore prefix . These conventions are equivalent in function; they differ only in how boundary information is attached to tokens.
The choice of prefix-based rather than suffix-based boundary marking matters for one practical reason: it affects how token IDs cluster in the embedding space. When the word "cat" appears at the beginning of a sentence (preceded by a space) and in the middle of a sentence, it gets the same token if you use prefix marking. With suffix marking, the "cat" in both positions gets the same token. The difference matters at sentence boundaries and after punctuation, where spacing conventions interact with the tokenization.