Cross-Entropy Loss › Binary vs Categorical Cross-Entropy

Language modeling uses categorical cross-entropy because we select among many discrete classes (vocabulary tokens). However, understanding binary cross-entropy illuminates special cases and alternative formulations that appear in auxiliary training objectives and discriminative fine-tuning tasks.

For multi-class classification with $K$ classes, where the model outputs a probability distribution $\hat{y}$ and the true class is represented as a one-hot vector $y$:

$$\mathcal{L} = -\sum_{k=1}^{K} y_k \log(\hat{y}_k)$$

where:

• $\mathcal{L}$: the cross-entropy loss for a single prediction
• $K$: the total number of classes (vocabulary size)
• $y_k$: the true probability of class $k$ (1 for the correct class, 0 otherwise in one-hot encoding)
• $\hat{y}_k$: the model's predicted probability for class $k$

Since $y$ is one-hot (all zeros except for the true class where $y_{\text{true}} = 1$), the sum collapses to a single term:

$$\begin{aligned} \mathcal{L} &= -\sum_{k=1}^{K} y_k \log(\hat{y}_k) \\ &= -\log(\hat{y}_{\text{true}}) \end{aligned}$$

where:

• $\hat{y}_{\text{true}}$: the model's predicted probability for the true class (the single class where $y_k = 1$)

In language modeling, $K$ equals the vocabulary size (often 32,000 to 100,000 tokens). The Softmax Function ensures $\hat{y}$ sums to 1 across all vocabulary items. This formulation handles the extreme multi-class nature of language modeling, where we must discriminate among tens of thousands of possible next tokens at each step.

For binary classification or independent multi-label problems:

$$\mathcal{L} = -\left[y \log(\hat{y}) + (1-y) \log(1-\hat{y})\right]$$

where:

• $\mathcal{L}$: the binary cross-entropy loss
• $y$: the true binary label (0 or 1)
• $\hat{y}$: the model's predicted probability that the label is 1
• $1-y$: the probability that the label is 0
• $1-\hat{y}$: the model's predicted probability that the label is 0

Language models rarely use binary cross-entropy directly for next-token prediction, but it appears in specialized objectives like Replaced Token Detection, where the model predicts whether each token is original or replaced. In this case, the task reduces to binary classification for each position independently. The discriminator component of ELECTRA uses this formulation to learn by distinguishing real tokens from synthetically corrupted ones. This provides an alternative to traditional masked language modeling.

Binary cross-entropy also appears in contrastive learning objectives and ranking tasks. When a model must choose between two candidate continuations, the relative log-probabilities determine which is preferred, and binary cross-entropy over these preferences can serve as a training signal. This pattern appears in reinforcement learning from human feedback, where the reward model learns to predict human preferences between pairs of model outputs.

A practical modification to cross-entropy prevents overconfidence. Instead of using hard targets (0 or 1), label smoothing distributes a small amount of probability mass $\epsilon$ across all classes:

$$\mathcal{L} = -\sum_{k=1}^{K} \left[(1-\epsilon) y_k + \frac{\epsilon}{K}\right] \log(\hat{y}_k)$$

where:

• $\epsilon$: the smoothing parameter (typically 0.01 to 0.1), representing the amount of probability mass to distribute uniformly
• $(1-\epsilon) y_k$: the adjusted target probability for the true class (reduced from 1)
• $\frac{\epsilon}{K}$: the uniform smoothing term added to all classes
• $\hat{y}_k$: the model's predicted probability for class $k$

This prevents the model from becoming overconfident and improves generalization by so gradients continue to flow even when the model predicts the correct class with high probability. Without smoothing, once a model achieves near-perfect prediction for a training example, gradients vanish and learning stops for that example. With smoothing, the model continues to learn, adjusting its internal representations to better model the underlying distribution rather than merely memorizing training labels. Typical values for $\epsilon$ range from 0.01 to 0.1, with higher values giving more regularization but potentially slowing convergence.

The effect of label smoothing can be understood from the decomposition perspective. Smoothing replaces the hard empirical distribution (all mass on one token) with a mixture: $(1-\epsilon)$ of the mass stays on the true token, and $\epsilon$ is spread uniformly. This makes the target distribution softer, requiring the model to assign non-negligible probability to all vocabulary items rather than collapsing its distribution to a single peak. In practice, label smoothing consistently improves downstream task performance and was used in the original Transformer paper, where it improved translation BLEU scores by 0.1 points despite slightly increasing training perplexity.