Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Linear Classifiers › Logistic Regression: From Score to Probability

Michael BrenndoerferApril 28, 20255 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 61Linear Classifiers › Logistic Regression: From Score to Probability

Reading section 5 of 16 · Logistic Regression: From Score to Probability · 5 minSection 5 of 16 · 5 min

Section 5 of 16 · 5 min read

Linear Classifiers › Logistic Regression: From Score to ProbabilityOne section · Sections in this chapter5 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The raw score $z = \mathbf{w} \cdot \mathbf{x} + b$ can be any real number, from negative infinity to positive infinity. This makes it hard to interpret as a probability, since probabilities must lie between 0 and 1. Logistic regression solves this by passing the score through the sigmoid function.

To convert $z$ into a probability, we need a function that maps $\mathbb{R}$ to $(0, 1)$ . The sigmoid function does exactly this:

\sigma(z) = \frac{1}{1 + e^{-z}}

where:

$z$ : the raw linear score $\mathbf{w} \cdot \mathbf{x} + b$ , any real number
$e^{-z}$ : the exponential of $-z$ ; this quantity is large when $z$ is negative and small when $z$ is positive
$1 + e^{-z}$ : the denominator, which normalizes the output
$\sigma(z) \in (0, 1)$ : the resulting probability, always strictly between 0 and 1

The sigmoid function has three key properties:

When $z \to +\infty$ : $e^{-z} \to 0$ , so $\sigma(z) \to 1$ (confident positive prediction)
When $z \to -\infty$ : $e^{-z} \to \infty$ , so $\sigma(z) \to 0$ (confident negative prediction)
When $z = 0$ : $\sigma(0) = \frac{1}{1+1} = 0.5$ (maximum uncertainty, on the decision boundary)

The function is smooth and differentiable everywhere, a requirement for gradient descent. Its derivative also has a convenient form:

\sigma'(z) = \sigma(z)(1 - \sigma(z))

where:

$\sigma(z)$ : the sigmoid output at $z$
$1 - \sigma(z)$ : one minus the sigmoid, since $\sigma(z) + (1 - \sigma(z)) = 1$

This derivative is easy to compute once you have the forward pass value, which makes backpropagation through the sigmoid efficient. You compute $\sigma(z)$ on the forward pass and store it; on the backward pass, you compute $\sigma(z)(1 - \sigma(z))$ without any additional expensive operations.

Logistic Regression

Logistic regression applies the sigmoid function to the linear score to produce a probability estimate. It predicts $P(y=1 \mid \mathbf{x}) = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$ , where $\sigma$ is the sigmoid function. Despite the name "regression," it is used for classification.

In[6]:

Code

def sigmoid(z):
    """Map a real-valued score to the interval (0, 1)."""
    return 1.0 / (1.0 + np.exp(-z))


def logistic_predict_proba(x, w, b):
    """Return the probability of the positive class."""
    z = np.dot(w, x) + b
    return sigmoid(z)

Out[7]:

Console

Review 1 (enthusiastic):
  Linear score: 3.60
  P(positive):  0.973

Review 2 (disappointed):
  Linear score: -3.80
  P(positive):  0.022

Review 3 (mixed):
  Linear score: -0.20
  P(positive):  0.450

Review 1 gets a high probability of being positive. Review 2 gets a low probability. Review 3, near the decision boundary, gets a probability close to 0.5. These probabilities are more informative than a hard yes/no decision because they communicate the model's confidence. A downstream system can decide how to act based on these probabilities: it might automatically file a very high-confidence prediction but route uncertain ones to a human reviewer.

The Sigmoid CurveLink Copied

Visualizing the sigmoid function helps build intuition about how it compresses scores into probabilities.

Out[8]:

Visualization

Line plot of the sigmoid function from z=-6 to z=6, showing the S-shaped curve. — The sigmoid function maps any real-valued score to the range (0, 1). The function equals 0.5 when z = 0, which corresponds to the decision boundary. It approaches 1 for large positive scores and 0 for large negative scores. The steepest region near z = 0 is where the function is most sensitive to input changes and where the gradient during training is largest.

Notice how the sigmoid "saturates" in the tails: for $z > 4$ or $z < -4$ , the function barely changes. This saturation has consequences for training. When the classifier is very confident but wrong, the gradient of the sigmoid is tiny, meaning learning happens slowly. This is the vanishing gradient problem in its simplest form, and it is one reason why alternative activations like ReLU (covered in the Activation Functions chapter) were developed for hidden layers of deep networks. For output layers producing probability estimates, however, sigmoid and softmax remain standard.

The Log-Odds ConnectionLink Copied

The sigmoid function has a natural probabilistic interpretation through log-odds. If $p = P(y=1 \mid \mathbf{x})$ , then the odds in favor of the positive class are $p / (1-p)$ , and the log-odds (also called the logit) are $\log(p / (1-p))$ . Inverting the sigmoid relationship shows that:

\log \frac{p}{1 - p} = z = \mathbf{w} \cdot \mathbf{x} + b

This means logistic regression is modeling the log-odds of the positive class as a linear function of the input features. A unit increase in feature $i$ multiplies the odds of the positive class by $e^{w_i}$ . This multiplicative relationship in odds space corresponds to an additive relationship in log-odds space, which is what makes the math tractable. The log-odds framing also connects logistic regression to Naive Bayes classifiers: under a specific set of distributional assumptions (that features are conditionally independent given the class), Naive Bayes produces a classifier with the same logistic regression form, with weights determined by feature likelihood ratios.