Logistic Regression › Example

Let's work through a detailed step-by-step example of logistic regression using a simple dataset. Suppose we want to predict whether a student will pass an exam based on their study hours and previous test scores.

Given Data:

• Study hours: [2, 3, 4, 5, 6, 7, 8]
• Previous scores: [60, 65, 70, 75, 80, 85, 90]
• Pass (1) or Fail (0): [0, 0, 1, 1, 1, 1, 1]

Step 1: Set up the design matrix

First, we organize our data into matrix form. Each row represents one student, and each column is a feature (including a column of 1s for the intercept):

where:

• $\mathbf{X}$ is the design matrix (a $7 \times 3$ matrix with 7 students and 3 columns)
• The first column contains all 1s (for the intercept term $\beta_0$)
• The second column contains study hours (feature $x_1$)
• The third column contains previous scores (feature $x_2$)

The response vector (whether each student passed) is:

where:

• $\mathbf{y}$ is the response vector (a $7 \times 1$ vector of binary outcomes)
• $y_i = 0$ indicates student $i$ failed the exam
• $y_i = 1$ indicates student $i$ passed the exam

Step 2: Initialize coefficients

We start with initial coefficient values. A common choice is to set all coefficients to zero:

where:

• $\boldsymbol{\beta}^{(0)}$ is the initial coefficient vector at iteration 0 (the starting point for optimization)
• $\beta_0^{(0)} = 0$ is the initial intercept (the baseline log-odds)
• $\beta_1^{(0)} = 0$ is the initial coefficient for study hours
• $\beta_2^{(0)} = 0$ is the initial coefficient for previous scores

This means our initial model predicts a 50% probability for all students (since $\eta = 0$ implies $p = 0.5$).

Step 3: Calculate the linear predictor

For the first iteration, with all coefficients at zero:

where:

• $\boldsymbol{\eta}^{(0)}$ is the vector of linear predictors at iteration 0 (a $7 \times 1$ vector, one value per student)
• Each element is computed as $\eta_i^{(0)} = 1 \cdot 0 + x_{i1} \cdot 0 + x_{i2} \cdot 0 = 0$ for student $i$

Step 4: Calculate predicted probabilities

Using the logistic function:

where:

• $\boldsymbol{p}^{(0)}$ is the vector of predicted probabilities at iteration 0 (a $7 \times 1$ vector, one probability per student)
• The logistic function is applied element-wise to each $\eta_i^{(0)}$
• Since $e^0 = 1$, we have $p_i^{(0)} = \frac{1}{1+1} = \frac{1}{2} = 0.5$ for all students

Step 5: Calculate the log-likelihood

The log-likelihood for logistic regression is:

where:

• $\ell(\boldsymbol{\beta})$ is the log-likelihood function (measures how well the model fits the data)
• $n$ is the number of observations (in our case, $n = 7$ students)
• $y_i$ is the actual outcome for observation $i$ (either 0 or 1)
• $p_i$ is the predicted probability for observation $i$ (from the logistic function)
• The sum aggregates the log-likelihood contributions from all observations

For our first iteration:

Breaking this down by observation:

Simplifying (since each term equals $\log(0.5)$):

Step 6: Update coefficients using gradient ascent

The gradient of the log-likelihood with respect to the coefficients is:

where:

• $\frac{\partial \ell}{\partial \boldsymbol{\beta}}$ is the gradient vector (a $(p+1) \times 1$ vector containing the partial derivatives with respect to each coefficient)
• $\mathbf{X}^T$ is the transpose of the design matrix (a $(p+1) \times n$ matrix, in our case $3 \times 7$)
• $\mathbf{y}$ is the vector of actual outcomes (a $n \times 1$ vector, in our case $7 \times 1$)
• $\mathbf{p}$ is the vector of predicted probabilities (a $n \times 1$ vector, in our case $7 \times 1$)
• $(\mathbf{y} - \mathbf{p})$ is the residual vector (the difference between actual and predicted values)

For our first iteration, we compute $\mathbf{X}^T(\mathbf{y} - \mathbf{p})$:

First, calculate the residuals $(\mathbf{y} - \mathbf{p})$:

Then, multiply by $\mathbf{X}^T$:

Computing each element of the gradient:

Step 7: Update coefficients

Using gradient ascent with learning rate $\alpha = 0.01$:

where:

• $\boldsymbol{\beta}^{(1)}$ is the updated coefficient vector after iteration 1
• $\alpha = 0.01$ is the learning rate (controls the step size in the direction of the gradient)
• The update rule moves the coefficients in the direction that increases the log-likelihood

Step 8: Iterate until convergence

We repeat steps 3-7 until the coefficients stop changing significantly. After several iterations, the algorithm converges to:

where:

• $\hat{\boldsymbol{\beta}}$ is the final estimated coefficient vector (the maximum likelihood estimates)
• $\hat{\beta}_0 = -15.2$ is the estimated intercept (the log-odds when study hours and previous scores are both zero)
• $\hat{\beta}_1 = 0.8$ is the estimated coefficient for study hours (the change in log-odds per additional hour of study)
• $\hat{\beta}_2 = 0.1$ is the estimated coefficient for previous scores (the change in log-odds per additional point on previous scores)

Let's visualize how the optimization process converges during training:

These visualizations demonstrate the iterative nature of logistic regression optimization:

Log-Likelihood Convergence: Shows how the model improves with each iteration, with rapid initial progress that gradually slows as it approaches the optimal solution. The log-likelihood is maximized (becomes less negative) as the model learns better parameters.

Coefficient Convergence: Illustrates how each coefficient evolves from its initial value to its final estimate. Different coefficients may converge at different rates depending on the data and feature scales. The convergence pattern shows that the optimization is working correctly: coefficients stabilize rather than oscillating or diverging.

Step 9: Final model interpretation

The fitted logistic regression model is:

where:

• $p$ is the probability of passing the exam (the outcome we're predicting)
• $\text{hours}$ is the number of study hours (feature $x_1$)
• $\text{scores}$ is the previous test score (feature $x_2$)

Interpretation:

Intercept ($\hat{\beta}_0 = -15.2$): When both study hours and previous scores are 0, the log-odds are -15.2, corresponding to a very low probability of passing

Study hours coefficient ($\hat{\beta}_1 = 0.8$): For each additional hour of study, the log-odds of passing increase by 0.8, holding previous scores constant

Previous scores coefficient ($\hat{\beta}_2 = 0.1$): For each additional point on previous scores, the log-odds of passing increase by 0.1, holding study hours constant

Step 10: Make predictions

For a student with 5 hours of study and a previous score of 75:

First, calculate the linear predictor:

Then, apply the logistic function to get the probability:

Since $e^{3.7} \approx 40.45$:

This student has approximately a 2.4% chance of passing.

For a student with 7 hours of study and a previous score of 85:

First, calculate the linear predictor:

Then, apply the logistic function:

Since $e^{1.1} \approx 3.00$:

This student has a 25% chance of passing.