Generalized Linear Models › Maximum Likelihood Estimation

Unlike linear regression, GLMs typically don't have closed-form solutions and require iterative optimization to find the maximum likelihood estimates of the parameters. The likelihood function for a GLM is:

$$L(\boldsymbol{\beta}) = \prod_{i=1}^n f(y_i | \boldsymbol{\beta}, \mathbf{x}_i)$$

where:

• $L(\boldsymbol{\beta})$ is the likelihood function (the probability of observing the data given the parameters)
• $\boldsymbol{\beta}$ is the vector of regression coefficients ($\boldsymbol{\beta} = [\beta_0, \beta_1, \ldots, \beta_p]^T$)
• $n$ is the number of observations in the dataset
• $f(y_i | \boldsymbol{\beta}, \mathbf{x}_i)$ is the probability density function (for continuous responses) or probability mass function (for discrete responses) of the chosen distribution for observation $i$
• $y_i$ is the observed response value for observation $i$
• $\mathbf{x}_i$ is the vector of predictor values for observation $i$ (i.e., $\mathbf{x}_i = [1, x_{i1}, x_{i2}, \ldots, x_{ip}]^T$ where the first element is 1 for the intercept)
• $\prod_{i=1}^n$ denotes the product over all $n$ observations (reflecting the assumption of independent observations)

The log-likelihood is:

$$\ell(\boldsymbol{\beta}) = \sum_{i=1}^n \log f(y_i | \boldsymbol{\beta}, \mathbf{x}_i)$$

where:

• $\ell(\boldsymbol{\beta})$ is the log-likelihood function (the logarithm of the likelihood)
• $\log$ denotes the natural logarithm
• $\sum_{i=1}^n$ denotes the sum over all $n$ observations
• Taking the logarithm converts the product to a sum, which is computationally more stable and prevents numerical underflow when dealing with many small probabilities

The maximum likelihood estimates are found by solving:

$$\hat{\boldsymbol{\beta}} = \arg\max_{\boldsymbol{\beta}} \ell(\boldsymbol{\beta})$$

where:

• $\hat{\boldsymbol{\beta}}$ is the maximum likelihood estimate of the coefficient vector (the value that makes the observed data most probable)
• $\arg\max_{\boldsymbol{\beta}}$ finds the value of $\boldsymbol{\beta}$ that maximizes the log-likelihood function $\ell(\boldsymbol{\beta})$

This is typically done using Iteratively Reweighted Least Squares (IRLS), which is an iterative algorithm that solves a sequence of weighted least squares problems.