Logistic Regression › Formula

The core formula behind logistic regression involves transforming a linear combination of features into a probability using the logistic function. Let's break this down step by step.

First, we start with a linear combination of features, just like in linear regression:

$$\eta = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p$$

where:

• $\eta$ is the linear predictor (a scalar value for a single observation that can take any real value from $-\infty$ to $+\infty$)
• $\beta_0$ is the intercept (the baseline log-odds when all features equal zero)
• $\beta_1, \beta_2, \ldots, \beta_p$ are the regression coefficients for each feature (quantify the effect of each feature on the log-odds)
• $x_1, x_2, \ldots, x_p$ are the feature values (predictor variables for a single observation)
• $p$ is the number of features (total count of predictor variables in the model)

This linear predictor can take any real value from $-\infty$ to $+\infty$.

The main idea of logistic regression is the logistic function (also called the sigmoid function), which maps the linear predictor into a probability:

$$p = \frac{1}{1 + e^{-\eta}} = \frac{e^{\eta}}{1 + e^{\eta}}$$

where:

• $p$ is the predicted probability (bounded between 0 and 1, representing the probability that $Y = 1$)
• $e$ is Euler's number (approximately 2.718, the base of the natural logarithm)
• $\eta$ is the linear predictor from above (the linear combination $\beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p$)

Let's understand why this transformation works:

When $\eta$ is very large (positive):

• $e^{-\eta}$ becomes very small (close to 0)
• $1 + e^{-\eta}$ becomes close to 1
• $p = \frac{1}{1 + e^{-\eta}} \approx \frac{1}{1 + 0} = 1$

When $\eta$ is very small (negative):

• $e^{-\eta}$ becomes very large
• $1 + e^{-\eta}$ becomes very large
• $p = \frac{1}{1 + e^{-\eta}} \approx \frac{1}{\text{very large number}} \approx 0$

When $\eta = 0$:

• $e^{-\eta} = e^0 = 1$
• $p = \frac{1}{1 + 1} = \frac{1}{2} = 0.5$

Let's visualize the logistic function to better understand its behavior:

This visualization shows the characteristic S-shaped curve of the logistic function. Notice how:

At η = 0: The probability is exactly 0.5, and the curve is steepest here

As η becomes very negative: The probability approaches 0 asymptotically

As η becomes very positive: The probability approaches 1 asymptotically

The curve is smooth: No sharp corners or discontinuities, making optimization tractable

The curve is symmetric: Around the point (0, 0.5)

The logit function is the inverse of the logistic function and represents the log-odds:

$$\text{logit}(p) = \log\left(\frac{p}{1-p}\right) = \eta$$

where:

• $\text{logit}(p)$ is the logit function (transforms probabilities to log-odds)
• $p$ is the probability (bounded between 0 and 1)
• $\frac{p}{1-p}$ is the odds (the ratio of the probability of success to the probability of failure)
• $\log(\cdot)$ is the natural logarithm (base $e$)
• $\eta$ is the linear predictor (the right-hand side: $\beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p$)

This equation reads as: "The log-odds of the probability equals the linear predictor."

Let's visualize this relationship to understand why logistic regression models the log-odds as a linear function:

This visualization shows why logistic regression models the log-odds rather than the probability directly. The logit transformation converts probabilities (which are bounded between 0 and 1) into log-odds (which can be any real number from $-\infty$ to $+\infty$). This unbounded range makes it possible to use a linear model:

At p = 0.5: The log-odds are 0, representing equal odds for both outcomes

As p approaches 0: The log-odds approach $-\infty$, representing very low odds

As p approaches 1: The log-odds approach $+\infty$, representing very high odds

The relationship is smooth and monotonic: Higher probabilities correspond to higher log-odds

Putting it all together, the complete logistic regression model is:

$$\log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p$$

where:

• $p = P(Y=1|X)$ is the probability that the response variable equals 1 given predictors $X$ (the parameter we're modeling)
• $\frac{p}{1-p}$ is the odds of success (the ratio of the probability of success to the probability of failure)
• $\log\left(\frac{p}{1-p}\right)$ is the log-odds or logit (the natural logarithm of the odds)
• $\beta_0$ is the intercept (the log-odds when all features equal zero)
• $\beta_1, \beta_2, \ldots, \beta_p$ are the regression coefficients (represent the change in log-odds for a one-unit increase in each feature)
• $x_1, x_2, \ldots, x_p$ are the feature values (predictor variables for a single observation)
• $p$ is the number of features (total count of predictor variables in the model)

This is read as: "The log-odds of the probability equals the linear combination of features."

In matrix form, logistic regression can be written as:

$$\log\left(\frac{\mathbf{p}}{1-\mathbf{p}}\right) = \mathbf{X}\boldsymbol{\beta}$$

where:

• $\mathbf{p}$ is the vector of predicted probabilities (an $n \times 1$ vector where each element is the predicted probability for one observation)
• $\mathbf{X}$ is the design matrix of features (an $n \times (p+1)$ matrix including a column of 1s for the intercept)
• $\boldsymbol{\beta}$ is the vector of coefficients (a $(p+1) \times 1$ vector containing the intercept and all feature coefficients)
• $n$ is the number of observations (rows in the dataset)
• $p$ is the number of features (excluding the intercept)
• The logit function $\log\left(\frac{\mathbf{p}}{1-\mathbf{p}}\right)$ is applied element-wise to $\mathbf{p}$

Bounded Output: The logistic function ensures probabilities remain between 0 and 1

Smooth and Continuous: The function is differentiable everywhere, making optimization tractable

Symmetric: The function is symmetric around 0.5 when $\eta = 0$ (i.e., $p(0) = 0.5$)

Monotonic: As the linear predictor increases, the probability increases monotonically

Asymptotic: The function approaches 0 and 1 asymptotically without reaching them

Convex Log-likelihood: The negative log-likelihood is convex, so a unique global maximum