Cross-Entropy Loss › Implementation Parameters

When implementing cross-entropy loss in practice, several parameters control numerical stability and behavior. Each choice reflects a tradeoff between computational efficiency, numerical safety, and training quality.

The eps parameter represents a small constant added to probabilities to prevent numerical errors from log(0). Typical values range from $10^{-12}$ to $10^{-7}$, as shown in our manual implementation where we clip probabilities before taking the logarithm. Without this protection, a model that accidentally assigns zero probability to the true token, either due to numerical underflow or an unfortunate initialization, would produce infinite loss and destroy the training process through exploding gradients. In practice, when operating on logits rather than probabilities (as PyTorch does internally), the log-sum-exp trick makes explicit clipping unnecessary, but the epsilon remains important in custom implementations.

The reduction parameter specifies how to aggregate losses across the batch. The options are 'mean' (average across all elements), 'sum' (total loss), and 'none' (per-element loss without reduction). Our sequence evaluation uses mean reduction to normalize by sequence length. This keeps longer sequences do not dominate the loss disproportionately. When training with gradient accumulation over multiple micro-batches before a weight update, the choice between mean and sum reduction affects whether you need to scale the loss by the number of accumulation steps. Mean reduction is typically safer because it produces loss values that are comparable across different batch sizes. This makes hyperparameter transfer between experiments more reliable.

The from_logits convention (called weight_softmax in some frameworks) indicates whether inputs are raw logits or probabilities. Operating directly on logits, as PyTorch's F.cross_entropy does internally, improves numerical stability by avoiding explicit exponentiation and division in a separate softmax step. When operating on logits, the function applies log-softmax internally in a numerically stable way using the log-sum-exp trick described above.

The label_smoothing parameter takes a float between 0 and 1 that distributes a small amount of probability mass to all classes, preventing overconfidence and improving generalization. Typical values range from 0.01 to 0.1. Higher values provide stronger regularization but may slow convergence and prevent the model from achieving peak accuracy on the training set. The optimal value depends on the task: generation tasks often use smaller values (0.01-0.05) while classification fine-tuning tasks may benefit from larger values (0.05-0.1). Label smoothing is particularly beneficial when the training data contains labeling noise, as it prevents the model from memorizing noisy labels with extreme confidence.

The ignore_index parameter (common in PyTorch) specifies a token index whose loss should be excluded from the computation. In language modeling, this is typically the padding token used to fill sequences to a uniform length within a batch. Without ignoring padding tokens, the model would receive spurious gradient signals pushing it to assign high probability to whatever padding token follows the actual end of each sequence. Setting ignore_index correctly is needed for correct training with variable-length sequences.