Linear Classifiers › Multiclass Classification: Softmax

Binary classification (two classes) covers many problems, but NLP is full of multiclass problems: classifying documents into dozens of topics, assigning one of hundreds of part-of-speech tags, or selecting the next word from a vocabulary of 50,000 tokens. Logistic regression extends to multiple classes through the softmax function.

For $K$ classes, we define $K$ weight vectors $\mathbf{w}_1, \mathbf{w}_2, \ldots, \mathbf{w}_K$ and $K$ biases $b_1, b_2, \ldots, b_K$. The model computes one score per class. For class $k$ and input $\mathbf{x}$:

$$z_k = \mathbf{w}_k \cdot \mathbf{x} + b_k \quad \text{for } k = 1, 2, \ldots, K$$

where:

• $\mathbf{w}_k \in \mathbb{R}^d$: the weight vector for class $k$, one per class
• $b_k$: the bias for class $k$
• $z_k$: the logit (raw score) for class $k$

The resulting score vector $\mathbf{z} = [z_1, z_2, \ldots, z_K]$ is called the vector of logits. To convert logits into a probability distribution, we apply the softmax function. For class $k$:

$$P(y = k \mid \mathbf{x}) = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}$$

where:

• $e^{z_k}$: the exponential of the $k$-th class logit, which ensures all values are positive
• $\sum_{j=1}^{K} e^{z_j}$: the sum of exponentials across all $K$ classes, the normalizing constant
• $P(y = k \mid \mathbf{x}) \in (0, 1)$: the resulting probability for class $k$

The softmax output is a valid probability distribution: all values are positive (because exponentials are always positive) and they sum to exactly 1 (by construction). The class with the highest logit always receives the highest probability. The exponential function amplifies differences: if class $A$ has a logit 2 units above class $B$, then $A$ receives $e^2 \approx 7.4$ times the unnormalized probability mass of $B$.

The softmax function is ubiquitous in modern deep learning. Every language model that predicts the next token applies softmax over a vocabulary of tens of thousands of entries to produce a distribution over what comes next. Every classification network ends with a softmax layer. Understanding how softmax works at the scale of logistic regression gives you the intuition to understand what is happening at vastly larger scale in transformer models.

Computing $e^{z_k}$ directly causes overflow when logits are large (for example, when $z_k = 1000$, $e^{1000}$ exceeds the floating-point limit). The standard fix is to subtract the maximum logit before exponentiating. This does not change the result because, for any constant $c$:

$$\frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}} = \frac{e^{z_k - c}}{\sum_{j=1}^{K} e^{z_j - c}}$$

where:

• $c = \max_j z_j$: the constant we subtract (chosen as the maximum so that the largest exponential is $e^0 = 1$)
• All other values $e^{z_j - c} \leq 1$: avoiding overflow

Setting $c = \max_j z_j$ ensures the numerically largest exponential equals 1 and all others are at most 1. This technique appears in every modern framework, including NumPy, PyTorch, and TensorFlow. If you ever implement softmax from scratch and forget this step, your code will silently produce NaN values on inputs with large logits.

The first document, mentioning "game," "server," and "team," gets split probability between sports and technology because it matches both class templates. The second document, mentioning "election" and "vote," is confidently classified as politics. This ambiguity in the first document reflects semantic ambiguity: "game server crashed during finals" could plausibly come from either a gaming article or a sports one. The model's uncertainty is well-calibrated.

When we have $K$ classes, the individual weight vectors $\mathbf{w}_1, \ldots, \mathbf{w}_K$ can be stacked as rows of a weight matrix $\mathbf{W} \in \mathbb{R}^{K \times d}$. The logit vector becomes:

$$\mathbf{z} = \mathbf{W} \mathbf{x} + \mathbf{b}$$

This is a matrix-vector product: each row of $\mathbf{W}$ computes a dot product with $\mathbf{x}$, producing one score per class. This operation is the first linear layer of any neural network. In a deep network, you stack many such transformations, but each individual transformation has exactly this form. Recognizing this pattern across different scales of complexity is key to understanding neural network architecture.