Lasso Regularization (L1 Regularization) › Example

Let's work through a simple example to understand how LASSO regularization works. Suppose we have a dataset with 3 features and we want to predict house prices. Our model has the following coefficients:

• $\beta_1 = 0.8$ (feature 1: square footage)
• $\beta_2 = -0.3$ (feature 2: distance from city center)
• $\beta_3 = 0.1$ (feature 3: number of bathrooms)

The L1 norm is the sum of absolute values of the coefficients:

$$\sum_{j=1}^3 |\beta_j| = |0.8| + |-0.3| + |0.1| = 0.8 + 0.3 + 0.1 = 1.2$$

where:

-$j = 1, 2, 3$: index over the three features

• $|\beta_j|$: absolute value of coefficient $j$
• $1.2$: total L1 penalty (sum of absolute coefficient values)

Let's see how different values of $\lambda$ affect the model. Start with understanding the optimization process. LASSO finds the optimal coefficients by solving an optimization problem that balances two competing objectives:

1. Fit the data well: Minimize the Sum of Squared Errors (SSE)
2. Keep the model simple: Minimize the L1 penalty term $\lambda \sum_{j=1}^p |\beta_j|$

The larger $\lambda$ is, the more the model prioritizes simplicity over fitting the training data perfectly. This trade-off is what prevents overfitting.

Think of it this way: LASSO is like having a "budget" for how large your coefficients can be. The penalty term $\lambda \sum_{j=1}^p |\beta_j|$ acts like a "cost" that increases as coefficients get larger.

• Small $\lambda$: Low cost for large coefficients → model can afford to keep coefficients large to fit data well
• Large $\lambda$: High cost for large coefficients → model needs to make coefficients smaller to stay within "budget"

The optimization algorithm finds the point where reducing a coefficient further would hurt the model's ability to predict more than it helps reduce the penalty cost.

Case 1: $\lambda = 0$ (No regularization)

• Penalty term: $0 \times 1.2 = 0$, so when $\lambda = 0$, the objective function becomes: $$\min_{\beta} \left\{ \text{SSE} + 0 \cdot \sum_{j=1}^3 |\beta_j| \right\} = \min_{\beta} \left\{ \text{SSE} \right\}$$ The penalty term disappears completely, so LASSO becomes identical to ordinary least squares regression. Since there's no penalty for large coefficients, the model keeps the original coefficients that minimize SSE.
• These are the final coefficients: $\beta_1 = 0.8$, $\beta_2 = -0.3$, $\beta_3 = 0.1$ (unchanged)

---

Case 2: $\lambda = 0.5$ (Moderate regularization)

• Penalty term: $0.5 \times 1.2 = 0.6$, so LASSO solves: $$\min_{\beta} \left\{ \text{SSE} + 0.5 \sum_{j=1}^3 |\beta_j| \right\}$$ The penalty affects the coefficients as follows (conceptually):
  • $\beta_1$ (square footage): Original magnitude = 0.8, penalty cost = $0.5 \times 0.8 = 0.4$, so the model reduces it to 0.6 (still useful, but cheaper)
  • $\beta_2$ (distance): Original magnitude = 0.3, penalty cost = $0.5 \times 0.3 = 0.15$, so the model reduces it from -0.3 to -0.2 (smaller penalty, still captures the negative relationship)
  • $\beta_3$ (bathrooms): Original magnitude = 0.1, penalty cost = $0.5 \times 0.1 = 0.05$, so the model reduces it from 0.1 to 0.05 (smallest coefficient, reduced proportionally) This happens because the penalty creates a "cost-benefit analysis" for each coefficient. Important features get reduced less than unimportant ones. (Exact values depend on the data; numbers here are illustrative.)

---

Case 3: $\lambda = 2.0$ (Strong regularization)

• Penalty term: $2.0 \times 1.2 = 2.4$, so LASSO solves: $$\min_{\beta} \left\{ \text{SSE} + 2.0 \sum_{j=1}^3 |\beta_j| \right\}$$ The strong penalty affects each coefficient as follows (conceptually):
  • $\beta_1$ (square footage): Original magnitude = 0.8, penalty cost = $2.0 \times 0.8 = 1.6$. The model asks: "Is keeping $\beta_1 = 0.8$ worth the penalty cost of 1.6?" Since square footage strongly affects house prices, the model keeps it but reduces it significantly: $0.8 \rightarrow 0.4$. The new penalty cost = $2.0 \times 0.4 = 0.8$ (much cheaper!)
  • $\beta_2$ (distance): Original magnitude = 0.3, penalty cost = $2.0 \times 0.3 = 0.6$. The model asks: "Is keeping $\beta_2 = -0.3$ worth the penalty cost of 0.6?" Distance is less important than square footage, so the model sets it to exactly zero. The new penalty cost = $2.0 \times 0 = 0$ (free!)
  • $\beta_3$ (bathrooms): Original magnitude = 0.1, penalty cost = $2.0 \times 0.1 = 0.2$. The model asks: "Is keeping $\beta_3 = 0.1$ worth the penalty cost of 0.2?" Bathrooms have minimal impact, so the model sets it to exactly zero. The new penalty cost = $2.0 \times 0 = 0$ (free!)
    (Exact values depend on the data; numbers here are illustrative.)

Let's trace through what happens mathematically. For each case, LASSO solves:

$$\min_{\beta_1, \beta_2, \beta_3} \left\{ \text{SSE}(\beta_1, \beta_2, \beta_3) + \lambda (|\beta_1| + |\beta_2| + |\beta_3|) \right\}$$

where:

-$\beta_1, \beta_2, \beta_3$: coefficients for the three features

• $\text{SSE}(\beta_1, \beta_2, \beta_3)$: sum of squared errors as a function of the coefficients
• $\lambda$: regularization parameter
• $|\beta_j|$: absolute value of coefficient $j$

For $\lambda = 0.5$:

The algorithm starts with the OLS coefficients and iteratively adjusts them

• It finds that reducing $\beta_1$ from 0.8 to 0.6 reduces the penalty by $0.5 \times 0.2 = 0.1$ while only slightly increasing SSE
• It finds that reducing $\beta_2$ from -0.3 to -0.2 reduces the penalty by $0.5 \times 0.1 = 0.05$ while only slightly increasing SSE
• It finds that reducing $\beta_3$ from 0.1 to 0.05 reduces the penalty by $0.5 \times 0.05 = 0.025$ while only slightly increasing SSE

For $\lambda = 2.0$:

The algorithm finds that the penalty cost for keeping $\beta_2$ and $\beta_3$ is too high

• Setting $\beta_2 = 0$ eliminates a penalty cost of $2.0 \times 0.3 = 0.6$
• Setting $\beta_3 = 0$ eliminates a penalty cost of $2.0 \times 0.1 = 0.2$
• The total penalty reduction (0.8) outweighs the increase in SSE from removing these features

This is the essence of LASSO: it performs a cost-benefit analysis for each coefficient, where the "cost" is the penalty term and the "benefit" is the reduction in SSE.