Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Perplexity › Bits Per Character and Bits Per Word

Michael BrenndoerferMarch 26, 20254 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 10Perplexity › Bits Per Character and Bits Per Word

Reading section 6 of 17 · Bits Per Character and Bits Per Word · 4 minSection 6 of 17 · 4 min

Section 6 of 17 · 4 min read

Perplexity › Bits Per Character and Bits Per WordOne section · Sections in this chapter6 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Perplexity can be expressed in different units depending on what granularity you're predicting at. Since perplexity and cross-entropy are related by $\text{PP} = 2^{\text{cross-entropy}}$ , we can also report results directly as cross-entropy in bits, which is sometimes more natural for comparing across different vocabularies and granularities.

Bits per word (BPW): This is the cross-entropy when predicting words. Since $\text{PP} = 2^{\text{BPW}}$ , we have $\text{BPW} = \log_2(\text{PP})$ . A perplexity of 100 corresponds to $\log_2(100) \approx 6.64$ bits per word.

Bits per character (BPC): When predicting characters instead of words, we use bits per character. Character-level models typically achieve 1-2 BPC on English text (corresponding to perplexity of 2-4 per character). BPC is useful when comparing models with different vocabularies or when dealing with morphologically rich languages where the appropriate vocabulary size is unclear.

The relationship between word-level and character-level metrics depends on average word length. If the average word has $L$ characters, a rough approximation is:

\text{BPC} \approx \frac{\text{BPW}}{L + 1}

where:

$\text{BPC}$ : bits per character (cross-entropy at character level)
$\text{BPW}$ : bits per word (cross-entropy at word level)
$L$ : average word length in characters
The $+1$ accounts for the space between words, which is also a character to predict

This approximation assumes the model's uncertainty is distributed roughly uniformly across characters within words. In practice, character-level models often achieve better compression than this formula suggests because they can exploit subword regularities, such as common prefixes and suffixes along with letter combinations that occur across many words.

The choice between word-level perplexity and character-level BPC matters when comparing models across languages. English has an average word length of about 5 characters, so a word-level perplexity of 100 (6.64 BPW) corresponds to roughly 6.64/6 $\approx$ 1.1 BPC. For languages with longer average words, BPC provides a more comparable metric. For languages with very short words or ideographic writing systems, the conversions break down differently.

In[13]:

Code

def perplexity_to_bits(pp):
    """Convert perplexity to bits (cross-entropy)."""
    return np.log2(pp)


def bits_to_perplexity(bits):
    """Convert bits (cross-entropy) to perplexity."""
    return 2**bits


# Example conversions
perplexities = [10, 50, 100, 500, 1000]

Out[14]:

Console

Perplexity to Bits Conversion:
----------------------------------------
  Perplexity   Bits per word
----------------------------------------
          10            3.32
          50            5.64
         100            6.64
         500            8.97
        1000            9.97

Out[15]:

Visualization

Line plot showing the exponential relationship between bits per word and perplexity on a log scale. — Relationship between bits per word and perplexity on a log scale. The exponential relationship means that each additional bit corresponds to doubling the perplexity. Reducing perplexity from 1000 to 500 requires the same bit improvement as reducing from 100 to 50, yet the latter is typically far harder to achieve in practice.

The logarithmic scale shows an important pattern: improvements in perplexity become harder as models get better. Reducing perplexity from 1000 to 500 saves the same number of bits as reducing from 100 to 50, but the latter is typically much harder to achieve. This is because the low-hanging fruit of obvious statistical patterns has already been captured; further improvements require capturing subtler, longer-range dependencies in language.