Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Perplexity › Natural Log vs Log Base 2

Michael BrenndoerferMarch 26, 20253 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 10Perplexity › Natural Log vs Log Base 2

Reading section 7 of 17 · Natural Log vs Log Base 2 · 3 minSection 7 of 17 · 3 min

Section 7 of 17 · 3 min read

Perplexity › Natural Log vs Log Base 2One section · Sections in this chapter7 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A practical note on conventions: perplexity can be computed using either $\log_2$ (base-2 logarithm) or $\ln$ (natural logarithm), and this choice is more common than you might expect from reading the literature.

When using $\log_2$ , cross-entropy is measured in bits and perplexity equals $2^{\text{cross-entropy}}$ . When using $\ln$ , cross-entropy is measured in nats (natural units), and the corresponding "perplexity" equals $e^{\text{cross-entropy}}$ .

The numerical values are different:

H_{\text{bits}}(P, Q) = \frac{H_{\text{nats}}(P, Q)}{\ln 2} \approx 1.4427 \times H_{\text{nats}}(P, Q)

where:

$H_{\text{bits}}$ : cross-entropy measured in bits (using $\log_2$ )
$H_{\text{nats}}$ : cross-entropy measured in nats (using $\ln$ )
$\ln 2 \approx 0.693$

Most academic papers and machine learning libraries report loss values using natural logarithm because $\ln$ has nicer mathematical properties (its derivative is simply $1/x$ ). This means the "loss" that gradient descent minimizes is typically cross-entropy in nats, not bits.

To convert between the two:

\text{PP}_{\text{base-2}} = e^{H_{\text{nats}}} = 2^{H_{\text{bits}}}

The perplexity value itself is the same regardless of which log base you use for the intermediate cross-entropy calculation, because $2^{H_{\text{bits}}} = 2^{H_{\text{nats}}/\ln 2} = e^{H_{\text{nats}}}$ . What changes is only how you express the intermediate cross-entropy value. Most practitioners report perplexity as a single number without worrying about this distinction, since the final perplexity value is unambiguous.

In[16]:

Code

def perplexity_from_nats(test_tokens, model_nat_log_prob):
    """
    Calculate perplexity using natural log (nats).
    Common in deep learning frameworks like PyTorch.

    Args:
        test_tokens: List of tokens from test corpus
        model_nat_log_prob: Function returning ln P(token | context)
    """
    log_prob_sum = 0.0
    count = 0

    for i, token in enumerate(test_tokens):
        log_p = model_nat_log_prob(token, i, test_tokens)
        if log_p == float("-inf"):
            return float("inf")
        log_prob_sum += log_p
        count += 1

    # Cross-entropy in nats
    cross_ent_nats = -log_prob_sum / count
    # Convert to perplexity: e^cross_entropy_nats = 2^cross_entropy_bits
    return np.exp(cross_ent_nats)


def unigram_nat_log_prob(token, position, tokens):
    """Return ln P(token) for unigram model with smoothing."""
    count = word_counts.get(token, 0)
    vocab_size = len(word_counts) + 1
    prob = (count + 1) / (total_words + vocab_size)
    return np.log(prob)  # Natural log

Out[17]:

Console

Perplexity computation with different log bases:
--------------------------------------------------
Using log2 (bits):    PP = 7.3068
Using ln (nats):      PP = 7.3068
Difference: 0.000000

Both methods give identical perplexity values.
The intermediate cross-entropy differs in units (bits vs nats),
but the final perplexity is the same.

This confirms that the choice of log base is a matter of convention: the final perplexity value is invariant. In practice, when you see a model's "loss" of 3.5 in a deep learning framework, that's typically cross-entropy in nats. To convert to perplexity, you compute $e^{3.5} \approx 33$ , meaning the model's effective branching factor is about 33 choices per word.