Multinomial Logistic Regression › Example

Let's work through a concrete example to understand how multinomial logistic regression works with actual numbers. We'll use a simple dataset with two predictor variables and three classes to make the calculations manageable.

Suppose we have a dataset with two features (age and income) and three possible outcomes (low, medium, high risk). We'll use "low" as our reference category and calculate the probabilities for "medium" and "high" risk.

As mentioned in the formula section earlier, the coefficients in multinomial logistic regression are found by solving an optimization problem that maximizes the likelihood of the observed data. In practice, this is done using numerical algorithms such as Newton-Raphson or gradient-based methods, which iteratively adjust the coefficients to best fit the data.

For this example, let's assume we have already obtained the estimated coefficients. We'll now use these given coefficients to demonstrate how to calculate the predicted probabilities for a new observation.

Now let's manually calculate the multinomial logistic regression for an observation with age = 25 and income = 30 using these estimated coefficients:

Coefficients for medium vs low (reference):

• $\beta_{\text{medium},0} = 0.5$ (intercept)
• $\beta_{\text{medium},\text{age}} = 0.1$ (age coefficient)
• $\beta_{\text{medium},\text{income}} = -0.05$ (income coefficient)

Coefficients for high vs low (reference):

• $\beta_{\text{high},0} = 1.2$ (intercept)
• $\beta_{\text{high},\text{age}} = 0.15$ (age coefficient)
• $\beta_{\text{high},\text{income}} = -0.08$ (income coefficient)

Step 1: Calculate Linear Predictors

For our observation with age = 25 and income = 30, we calculate the linear predictor for each category:

$$\eta_{\text{medium}} = \beta_{\text{medium},0} + \beta_{\text{medium},\text{age}} \times \text{age} + \beta_{\text{medium},\text{income}} \times \text{income}$$

where:

• $\eta_{\text{medium}}$ is the linear predictor for the medium risk category (the log-odds of medium risk relative to low risk before normalization)
• $\beta_{\text{medium},0} = 0.5$ is the intercept for medium risk (the baseline log-odds when age and income are both zero)
• $\beta_{\text{medium},\text{age}} = 0.1$ is the age coefficient for medium risk (the change in log-odds for each one-year increase in age)
• $\beta_{\text{medium},\text{income}} = -0.05$ is the income coefficient for medium risk (the change in log-odds for each one-unit increase in income)

Substituting the values:

$$\eta_{\text{medium}} = 0.5 + 0.1 \times 25 + (-0.05) \times 30 = 0.5 + 2.5 - 1.5 = 1.5$$

Similarly for high risk:

$$\eta_{\text{high}} = \beta_{\text{high},0} + \beta_{\text{high},\text{age}} \times \text{age} + \beta_{\text{high},\text{income}} \times \text{income}$$

where:

• $\eta_{\text{high}}$ is the linear predictor for the high risk category (the log-odds of high risk relative to low risk before normalization)
• $\beta_{\text{high},0} = 1.2$ is the intercept for high risk (the baseline log-odds when age and income are both zero)
• $\beta_{\text{high},\text{age}} = 0.15$ is the age coefficient for high risk (the change in log-odds for each one-year increase in age)
• $\beta_{\text{high},\text{income}} = -0.08$ is the income coefficient for high risk (the change in log-odds for each one-unit increase in income)

Substituting the values:

$$\eta_{\text{high}} = 1.2 + 0.15 \times 25 + (-0.08) \times 30 = 1.2 + 3.75 - 2.4 = 2.55$$

For the reference category (low risk):

$$\eta_{\text{low}} = 0$$

where:

• $\eta_{\text{low}}$ is the linear predictor for the low risk category (set to 0 because low risk is the reference category)

Step 2: Calculate Exponentials

We apply the exponential function to each linear predictor:

$$e^{\eta_{\text{medium}}} = e^{1.5} \approx 4.482$$

where:

• $e$ is Euler's number (approximately 2.71828)
• $e^{\eta_{\text{medium}}}$ is the exponential of the linear predictor for medium risk (used in the softmax numerator)

$$e^{\eta_{\text{high}}} = e^{2.55} \approx 12.807$$

where:

• $e^{\eta_{\text{high}}}$ is the exponential of the linear predictor for high risk (used in the softmax numerator)

$$e^{\eta_{\text{low}}} = e^{0} = 1.000$$

where:

• $e^{\eta_{\text{low}}}$ is the exponential of the linear predictor for low risk (equals 1 because $e^0 = 1$)

Step 3: Calculate Denominator

The denominator of the softmax function is the sum of all exponentials:

$$\sum_{k=1}^{K} e^{\eta_k} = e^{\eta_{\text{low}}} + e^{\eta_{\text{medium}}} + e^{\eta_{\text{high}}}$$

where:

• $\sum_{k=1}^{K} e^{\eta_k}$ is the sum of exponentials across all $K$ categories (the normalization constant in the softmax function)
• $K = 3$ is the total number of categories (low, medium, high)

Substituting the values:

$$\sum_{k=1}^{K} e^{\eta_k} = 1.000 + 4.482 + 12.807 = 18.289$$

Step 4: Calculate Probabilities

Using the softmax formula, we calculate the probability for each category:

$$P(\text{low}) = \frac{e^{\eta_{\text{low}}}}{\sum_{k=1}^{K} e^{\eta_k}} = \frac{1.000}{18.289} \approx 0.055$$

where:

• $P(\text{low})$ is the predicted probability of low risk for this observation (computed by dividing the exponential of the low risk linear predictor by the sum of all exponentials)

$$P(\text{medium}) = \frac{e^{\eta_{\text{medium}}}}{\sum_{k=1}^{K} e^{\eta_k}} = \frac{4.482}{18.289} \approx 0.245$$

where:

• $P(\text{medium})$ is the predicted probability of medium risk for this observation (computed by dividing the exponential of the medium risk linear predictor by the sum of all exponentials)

$$P(\text{high}) = \frac{e^{\eta_{\text{high}}}}{\sum_{k=1}^{K} e^{\eta_k}} = \frac{12.807}{18.289} \approx 0.700$$

where:

• $P(\text{high})$ is the predicted probability of high risk for this observation (computed by dividing the exponential of the high risk linear predictor by the sum of all exponentials)

Step 5: Verification

We verify that all probabilities sum to 1, as required by the softmax function:

$$P(\text{low}) + P(\text{medium}) + P(\text{high}) = 0.055 + 0.245 + 0.700 = 1.000 \,\checkmark$$

The calculations show that for a 25-year-old with income 30, the model predicts:

• 5.5% probability of low risk
• 24.5% probability of medium risk
• 70.0% probability of high risk

The predicted class would be "high" since it has the highest probability. Notice how the probabilities sum to exactly 1, which is a fundamental property of the multinomial logistic regression model.