Cross-Entropy Loss › Bits Per Character

While cross-entropy measures bits per token, bits per character (BPC) provides a tokenization-agnostic metric for comparing model efficiency. Since different tokenizers produce different sequence lengths for the same text, BPC normalizes by the number of characters in the original text. This normalization is needed for fair comparison because a model using a more aggressive tokenizer (with larger tokens) will naturally achieve lower per-token perplexity than an equivalent model using character-level tokenization, even if both models capture the same underlying linguistic patterns.

Given a sequence with $C$ characters and cross-entropy loss $H$ (in bits), bits per character is:

where:

• $\text{BPC}$: bits per character, measuring compression efficiency independent of tokenization
• $H$: the cross-entropy loss in bits (not nats)
• $T$: the number of tokens in the sequence
• $C$: the number of characters in the original text

For character-level language models where $T = C$, BPC equals the cross-entropy. For subword tokenizers where $T < C$ (multiple characters per token), we must account for the compression ratio. The multiplication by $T$ converts the per-token loss to total bits for the sequence, while division by $C$ spreads those bits across the original characters.

Current state-of-the-art language models achieve approximately 0.7-0.9 BPC on standard English text corpora, approaching but not reaching the estimated entropy of English (typically estimated between 0.6-1.0 BPC depending on the domain and estimation method). This gap suggests there remains room for improvement in language modeling, though diminishing returns may make further advances increasingly difficult.

When comparing models using different tokenization strategies, BPC provides fairer comparison than per-token metrics:

• Character-level models: Directly report BPC as the loss, since each token corresponds to one character.
• Subword models: Must divide total bits by character count to enable fair comparison with character-level models.
• Byte-level models: Similar to character-level but operating on UTF-8 bytes, where one character may consist of multiple bytes.

This normalization is important when evaluating Byte Pair Encoding or SentencePiece tokenizers against character-level baselines, as the same model architecture can appear to have wildly different per-token perplexities solely due to tokenization differences. Without BPC normalization, you might conclude that a subword model is dramatically better than a character model, when in fact the improvement comes primarily from the tokenizer grouping characters into larger units rather than from better linguistic understanding.