Cross-Entropy Loss › Code Implementation
Language AI HandbookAll chapters Cross-Entropy Loss › Code ImplementationOne section · Sections in this chapter10 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement cross-entropy calculations from scratch and using standard libraries. We'll demonstrate the relationship between manual calculation, PyTorch's implementation, and the conversion to perplexity.
import numpy as np
import torch
# Set seed for reproducibility
np.random.seed(42)
torch.manual_seed(42)We'll start with a basic implementation using NumPy to understand the mechanics, then show the optimized PyTorch version used in actual training loops.
def cross_entropy_manual(y_true_idx, y_pred_probs, eps=1e-12):
"""
Manual cross-entropy calculation.
Args:
y_true_idx: Integer index of true class
y_pred_probs: Array of predicted probabilities (must sum to 1)
eps: Small constant to avoid log(0)
Returns:
Cross-entropy in nats (natural log)
"""
# Clip probabilities to avoid log(0)
y_pred_clipped = np.clip(y_pred_probs, eps, 1.0 - eps)
# Get probability assigned to true class
prob_true = y_pred_clipped[y_true_idx]
# Return negative log probability
return -np.log(prob_true)
def perplexity_from_entropy(cross_entropy, base="e"):
"""Convert cross-entropy to perplexity."""
if base == "2":
return 2**cross_entropy
else:
return np.exp(cross_entropy)Now let's test with our worked example from earlier:
Vocabulary: ['the', 'cat', 'sat', 'mat']
True token: 'sat' (index 2)
Model probabilities: {'the': np.float64(0.1), 'cat': np.float64(0.05), 'sat': np.float64(0.7), 'mat': np.float64(0.15)}
Cross-entropy: 0.3567 nats (0.5146 bits)
Perplexity: 1.4286
Uniform distribution cross-entropy: 1.3863 nats
Uniform perplexity: 4.0000 (should equal vocab size 4)The manual calculation confirms our mathematical understanding. Now let's implement sequence-level evaluation using PyTorch, which handles batched operations efficiently:
def evaluate_sequence(model_probs, true_indices):
"""
Calculate cross-entropy and perplexity for a sequence.
Args:
model_probs: Tensor of shape (seq_len, vocab_size) with probability distributions
true_indices: Tensor of shape (seq_len,) with true token indices
Returns:
Dictionary with cross_entropy (nats), bits_per_token, perplexity, and bpc
"""
seq_len = len(true_indices)
# Extract probabilities of true tokens
true_probs = model_probs[torch.arange(seq_len), true_indices]
# Cross-entropy (negative log likelihood averaged over sequence)
ce = -torch.log(true_probs).mean()
# Convert to bits
ce_bits = ce / torch.log(torch.tensor(2.0))
# Perplexity
perp = torch.exp(ce)
return {
"cross_entropy_nats": ce.item(),
"cross_entropy_bits": ce_bits.item(),
"perplexity": perp.item(),
}Let's simulate a sequence evaluation with our example sentence "the cat sat mat":
Sequence evaluation results: Cross-entropy: 0.7925 nats Cross-entropy: 1.1434 bits/token Perplexity: 2.2090 Bits per character: 0.3049 (assuming 15 chars)
PyTorch provides optimized implementations that handle numerical stability through log-softmax and avoid explicit probability calculations when working with logits directly:
import torch.nn.functional as F
# PyTorch's cross_entropy function operates on logits, not probabilities
logits = torch.log(
probs_tensor + 1e-12
) # Convert back to log space for demonstration
# F.cross_entropy expects (N, C) for logits and (N,) for targets
# It combines log_softmax + nll_loss efficiently
pytorch_ce = F.cross_entropy(logits, true_indices)PyTorch cross-entropy: 0.7925 Matches manual calculation: True
The close agreement between PyTorch's optimized implementation and our manual calculation confirms that both approaches correctly compute cross-entropy. The tiny numerical differences (if any) stem from floating-point precision variations in how probabilities are handled. This validation gives us confidence that our manual implementation correctly implements the mathematical definition, while PyTorch's version offers superior numerical stability and computational efficiency for production use.
Visualizing Loss Curves
Let's simulate training curves to demonstrate typical cross-entropy dynamics:
import numpy as np
def simulate_training_curve(
steps=1000, initial_loss=8.0, final_loss=2.5, noise=0.05
):
"""
Simulate realistic training and validation loss curves.
"""
# Training loss: smooth exponential decay with noise
t = np.linspace(0, 1, steps)
train_loss = final_loss + (initial_loss - final_loss) * np.exp(-5 * t)
train_loss += np.random.normal(0, noise, steps) * (
1 - 0.8 * t
) # Decreasing noise
# Validation loss: similar but with gap and potential overfitting at end
val_loss = (
train_loss + 0.3 + 0.2 * np.maximum(0, t - 0.7) ** 2
) # Gap increases slightly at end
val_loss += np.random.normal(0, noise * 1.5, steps)
return train_loss, val_loss
train_loss, val_loss = simulate_training_curve()
steps = np.arange(len(train_loss))
The plot demonstrates how cross-entropy decreases during training while maintaining the exponential relationship with perplexity shown on the secondary axis. Notice how validation loss typically runs 10-30% higher than training loss due to the model not having seen that specific data during optimization.
Comparing Different Model Sizes
Let's examine how cross-entropy scales with model capacity, which reflects the power laws discussed in Power Laws in Deep Learning:
Model Size vs Cross-Entropy (simulated): -------------------------------------------------- 0.1B params: CE = 6.185 nats, Perplexity = 485.3 0.3B params: CE = 5.623 nats, Perplexity = 276.7 0.8B params: CE = 5.301 nats, Perplexity = 200.6 1.3B params: CE = 5.089 nats, Perplexity = 162.3 2.7B params: CE = 4.814 nats, Perplexity = 123.3 6.7B params: CE = 4.493 nats, Perplexity = 89.4 13.0B params: CE = 4.272 nats, Perplexity = 71.7 175.0B params: CE = 3.506 nats, Perplexity = 33.3
This demonstrates the predictable relationship between model scale and cross-entropy: with the simulated exponent above, doubling model parameters yields a constant reduction in loss of about 5% in the relevant regime.