Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Linear Classifiers › Training with Gradient Descent

Michael BrenndoerferApril 28, 20258 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 61Linear Classifiers › Training with Gradient Descent

Reading section 8 of 16 · Training with Gradient Descent · 8 minSection 8 of 16 · 8 min

Section 8 of 16 · 8 min read

Linear Classifiers › Training with Gradient DescentOne section · Sections in this chapter8 / 16 ↓

Track your reading progress

Sign in to mark sections as read and track your learning journey

Sign in →

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

So far, we have assumed the weights $\mathbf{w}$ and bias $b$ are given. In practice, we learn them from labeled training data by minimizing a loss function. The loss function quantifies how wrong the model's predictions are, and gradient descent iteratively adjusts the parameters to reduce that wrongness.

The Cross-Entropy LossLink Copied

For binary classification, the standard loss function for logistic regression is the binary cross-entropy. Given a training example $(\mathbf{x}, y)$ where $y \in \{0, 1\}$ and the model predicts $\hat{p} = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$ , the loss is:

\mathcal{L}(\mathbf{w}, b) = -y \log \hat{p} - (1 - y) \log(1 - \hat{p})

where:

$y \in \{0, 1\}$ : the true label
$\hat{p} = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$ : the model's predicted probability for the positive class
$-\log \hat{p}$ : the active term when $y = 1$ ; this approaches 0 as $\hat{p} \to 1$ and approaches $+\infty$ as $\hat{p} \to 0$
$-\log(1 - \hat{p})$ : the active term when $y = 0$ ; this approaches 0 as $\hat{p} \to 0$ and approaches $+\infty$ as $\hat{p} \to 1$

The cross-entropy loss has an important property: it penalizes confident wrong predictions much more severely than uncertain ones. If the model assigns probability 0.01 to the true class, the loss is $-\log(0.01) \approx 4.6$ . If it assigns 0.9, the loss is only $-\log(0.9) \approx 0.1$ .

For multiclass problems with $K$ classes, the cross-entropy generalizes. The true label $y$ is represented as a one-hot vector $\mathbf{y} = [y_1, y_2, \ldots, y_K]$ where $y_k = 1$ if $k$ is the true class and $y_k = 0$ otherwise. The multiclass cross-entropy is:

\mathcal{L}(\mathbf{W}, \mathbf{b}) = -\sum_{k=1}^{K} y_k \log P(y = k \mid \mathbf{x})

where:

$y_k \in \{0, 1\}$ : the one-hot indicator for class $k$
$P(y = k \mid \mathbf{x})$ : the softmax probability for class $k$
Since exactly one $y_k = 1$ , this simplifies to $-\log P(y = \text{true class} \mid \mathbf{x})$ : only the true class probability is penalized

Why Cross-Entropy and Not Mean Squared Error?Link Copied

You might wonder why we use cross-entropy loss instead of mean squared error (MSE). After all, MSE is simpler and widely used in regression. The issue is that MSE applied to probability outputs produces gradients that vanish when the model is very wrong. Consider a training example where the true label is $y = 1$ but the model predicts $\hat{p} = 0.001$ (extremely wrong). The MSE gradient is proportional to $(\hat{p} - y) \cdot \hat{p}(1-\hat{p})$ , which is tiny because $\hat{p}(1-\hat{p})$ is tiny near 0. The model would learn almost nothing from this catastrophically wrong prediction. Cross-entropy avoids this: its gradient is $\hat{p} - y = 0.001 - 1 = -0.999$ , which is large, producing a strong learning signal even when the model is very wrong.

This is one of the most important practical lessons in designing learning systems: the choice of loss function matters for what it measures and for the quality of the learning signal it provides. Cross-entropy is the natural loss for models that produce probability estimates, both because of its probabilistic derivation (negative log-likelihood) and because of its favorable gradient properties.

Computing GradientsLink Copied

To minimize the loss, we need the gradient of $\mathcal{L}$ with respect to the weights $\mathbf{w}$ and bias $b$ . A key result of differentiating through the sigmoid and logarithm is that the gradient has a remarkably simple form. For a single training example $(\mathbf{x}, y)$ :

\frac{\partial \mathcal{L}}{\partial \mathbf{w}} = (\hat{p} - y) \mathbf{x}

\frac{\partial \mathcal{L}}{\partial b} = \hat{p} - y

where:

$\hat{p} - y$ : the prediction error, the difference between the model's predicted probability and the true label
$\mathbf{x}$ : the input features, which scale the gradient for each individual weight dimension

The gradient formula has an elegant structure: it is exactly the prediction error times the input. If the model predicts $\hat{p} = 0.9$ but the true label is $y = 0$ (error = 0.9), the gradient is large and pushes the weights to reduce the score for this input. If the prediction is correct and confident ( $\hat{p} = 0.95$ , $y = 1$ , error = 0.05), the gradient is small and the update is tiny. We only learn substantially from mistakes.

Deriving this gradient is a good exercise. Starting from $\mathcal{L} = -y \log \sigma(z) - (1-y)\log(1 - \sigma(z))$ with $z = \mathbf{w} \cdot \mathbf{x} + b$ , and using the chain rule together with $\sigma'(z) = \sigma(z)(1-\sigma(z))$ , the derivatives of the logarithms cancel beautifully with the derivative of the sigmoid, yielding exactly $\hat{p} - y$ . This cancellation is not accidental: it is the reason that cross-entropy is the "right" loss for sigmoid outputs, a pairing that produces clean gradients and fast learning.

Stochastic Gradient DescentLink Copied

Gradient descent updates parameters iteratively:

\begin{aligned} \mathbf{w} &\leftarrow \mathbf{w} - \eta \frac{\partial \mathcal{L}}{\partial \mathbf{w}} \\ b &\leftarrow b - \eta \frac{\partial \mathcal{L}}{\partial b} \end{aligned}

where:

$\eta > 0$ : the learning rate, controlling step size
$\frac{\partial \mathcal{L}}{\partial \mathbf{w}}$ : the gradient pointing uphill; subtracting it moves downhill toward lower loss

For large datasets, computing gradients over the full training set is expensive. Stochastic gradient descent (SGD) uses a single random example or a small mini-batch per update, which is much faster and introduces useful noise that can help escape shallow local minima. The Stochastic Gradient Descent chapter covers SGD in depth.

In[13]:

Code

def binary_cross_entropy(y_true, y_pred_proba):
    """Compute binary cross-entropy loss for a batch."""
    eps = 1e-15  # Prevent log(0)
    p = np.clip(y_pred_proba, eps, 1 - eps)
    return -np.mean(y_true * np.log(p) + (1 - y_true) * np.log(1 - p))


def train_logistic_regression(
    X, y, learning_rate=0.1, n_epochs=100, random_state=42
):
    """Train logistic regression with full-batch gradient descent."""
    n_samples, n_features = X.shape

    w = np.zeros(n_features)
    b = 0.0
    losses = []

    for epoch in range(n_epochs):
        # Forward pass
        z = X @ w + b
        p_hat = sigmoid(z)

        # Loss
        loss = binary_cross_entropy(y, p_hat)
        losses.append(loss)

        # Gradients (averaged over the batch)
        error = p_hat - y
        grad_w = (X.T @ error) / n_samples
        grad_b = error.mean()

        # Parameter update
        w -= learning_rate * grad_w
        b -= learning_rate * grad_b

    return w, b, losses

In[14]:

Code

# Binary classification: sports vs. politics (first 6 documents)
X_binary = X[:6]
y_binary = np.array([0, 0, 0, 1, 1, 1])  # 0=sports, 1=politics

w_trained, b_trained, losses = train_logistic_regression(
    X_binary, y_binary, learning_rate=0.5, n_epochs=200
)

Out[15]:

Console

Training progress:
   Epoch      Loss    Accuracy
       1    0.6931       1.000
      11    0.5130       1.000
      51    0.2208       1.000
     101    0.1204       1.000
     200    0.0612       1.000

Final training accuracy: 1.000

The loss decreases steadily and accuracy reaches a high level as the model learns which TF-IDF features distinguish sports from politics articles.

Out[16]:

Visualization

Line plot of cross-entropy loss over training epochs, showing monotone decrease. — Binary cross-entropy loss over 200 training epochs for the sports vs. politics classifier. The loss decreases rapidly in the first 50 epochs as the model discovers the most discriminative features, then continues to decrease more slowly as it fine-tunes the boundary. The smooth, monotone decrease is characteristic of gradient descent on a convex loss function.

The Convexity AdvantageLink Copied

One of the most important theoretical properties of logistic regression is that the cross-entropy loss is a convex function of the parameters $\mathbf{w}$ and $b$ . Convexity means that the loss surface has no local minima: any local minimum is guaranteed to be the global minimum. Gradient descent on a convex function will always find the optimal solution, provided the learning rate is small enough and training runs long enough.

This guarantee does not hold for neural networks with hidden layers, which have non-convex loss surfaces with many local minima. Practitioners working with deep networks must worry about initialization, learning rate schedules, and optimization methods suited to these surfaces. With logistic regression, the optimization is straightforward. This makes logistic regression a reliable baseline and a sanity check: if your neural network performs worse than logistic regression on the same features, something has gone wrong with the training procedure.

The convexity also means that the solution is unique (assuming the data is not linearly separable with a finite margin). There is a single global optimum that all gradient descent trajectories converge to, regardless of initialization. This uniqueness is valuable for reproducibility and for understanding what the model has learned.

Track your reading progress

Sign in to mark sections as read and track your learning journey

Sign in →

The Cross-Entropy LossLink Copied

Why Cross-Entropy and Not Mean Squared Error?Link Copied

Computing GradientsLink Copied

Stochastic Gradient DescentLink Copied

The Convexity AdvantageLink Copied

Stay up to date