LIME (Local Interpretable Model-agnostic Explanations) › Formula

The mathematical foundation of LIME begins with a fundamental question: How do we fairly attribute a model's prediction to its input features when the model itself is a complex, opaque function? We need to explain which features matter and precisely how much each feature contributes to a specific prediction in a way that is both accurate and interpretable.

Explaining a complex model globally is often impossible. Feature interactions, non-linear transformations, and complex decision boundaries resist simple explanations. LIME takes a different approach. Instead of trying to explain the entire model, we focus on understanding its behavior in a small neighborhood around a specific prediction.

Think of it this way. If you're standing in mountainous terrain, the terrain as a whole is complex and hard to describe. But if you look at the ground right around your feet, it appears nearly flat. LIME applies this same logic to machine learning models. Locally, even complex models often behave approximately linearly.

We need a simple model $g$ that approximates our complex model $f$ in the local neighborhood. What makes a model both simple and interpretable? LIME chooses a linear model because linear models provide clear, additive feature contributions. We can write this as:

$$g(z') = w_g \cdot z'$$

where:

• $g$: Local explanation model (linear)
• $f$: Complex model we want to explain
• $z'$: Instance in simplified interpretable representation (typically binary vector)
• $w_g$: Weight vector containing attribution scores for each feature

Each weight $w_g[i]$ directly answers the question: "How much does feature $i$ contribute to this prediction?"

The linear form is key. It lets us understand the total prediction as a sum of individual feature contributions. If we say the prediction is 0.7, we can say "Feature 1 contributed +0.2, Feature 2 contributed +0.3, Feature 3 contributed +0.2." This additive structure makes the explanation naturally interpretable.

Having a linear model isn't enough. We need to ensure it accurately represents the complex model's behavior where it matters most: near our instance of interest. Not all nearby instances should matter equally. An instance very similar to our original should influence the explanation much more than one that's only vaguely similar.

We capture this with a proximity measure $\pi_x(z)$ that quantifies how "local" each perturbed instance $z$ is to our original instance $x$. We typically use an exponential kernel:

$$\pi_x(z) = \exp(-D(x, z)^2 / \sigma^2)$$

where:

• $\pi_x(z)$: Proximity weight for instance $z$ relative to original instance $x$
• $D(x, z)$: Distance between original instance $x$ and perturbed instance $z$
• $\sigma$: Bandwidth parameter controlling the size of the local neighborhood

This formula captures our intuition well. When a perturbed instance $z$ is very close to $x$, the distance $D(x, z)$ is small. The exponential function returns a weight close to 1, giving that instance full influence. As $z$ moves farther away, the weight decays exponentially. Only truly local instances shape our explanation.

The bandwidth parameter $\sigma$ controls the size of our local neighborhood. Choose $\sigma$ too small, and we might overfit to noise in the immediate vicinity. Choose it too large, and we include instances outside the local region, violating our assumption that the model behaves linearly nearby.

Now we can define what makes our simple explanation $g$ a good approximation of the complex model $f$. We want $g$ to match $f$'s predictions on nearby instances, with more emphasis on instances closer to our original. This gives us a weighted loss function:

$$L(f, g, \pi_x) = \sum_{z, z'} \pi_x(z) (f(z) - g(z'))^2$$

where:

• $L(f, g, \pi_x)$: Weighted loss function measuring approximation quality
• $z$: Perturbed instance in original feature space
• $z'$: Perturbed instance in interpretable representation
• $f(z)$: Complex model's prediction for instance $z$
• $g(z')$: Local explanation model's prediction for instance $z'$

The squared term $(f(z) - g(z'))^2$ measures how well our simple explanation matches the complex model's prediction for each perturbed instance. The proximity weight $\pi_x(z)$ scales this error. Matching accuracy matters more for nearby instances. By minimizing this weighted loss, we force $g$ to approximate $f$ accurately in the local neighborhood, with precision increasing as we approach the instance we're explaining.

Accuracy alone doesn't guarantee a useful explanation. Imagine our local linear model closely matches $f$ on all nearby instances, but does so using all features with non-zero weights, including marginally relevant ones. This explanation would be technically accurate but practically useless. It fails to identify the truly important features.

We need sparse attributions that highlight only the features that matter. This motivates a complexity penalty $\Omega(g)$ that encourages simpler explanations:

$$\Omega(g) = \lambda \|w_g\|_0$$

where:

• $\Omega(g)$: Complexity penalty encouraging sparse explanations
• $\lambda$: Regularization parameter balancing accuracy and simplicity
• $\|w_g\|_0$: L0 norm counting the number of non-zero weights

The term $\|w_g\|_0$ counts the number of non-zero weights. The parameter $\lambda$ balances accuracy against simplicity. This penalty embodies Occam's razor: among equally accurate explanations, prefer the simpler one. By penalizing the number of features, we force the explanation to focus on the most important contributions, filtering out noise and marginal effects.

We can now state LIME's complete optimization problem:

$$\xi(x) = \arg\min_{g \in G} L(f, g, \pi_x) + \Omega(g)$$

where:

• $\xi(x)$: The explanation for instance $x$
• $G$: Class of interpretable models (linear models in LIME)
• $L(f, g, \pi_x)$: Weighted loss measuring local fidelity
• $\Omega(g)$: Complexity penalty for sparsity

This formula represents LIME's solution to the attribution problem. We're looking for an explanation $g$ that minimizes two things:

1. Weighted loss $L(f, g, \pi_x)$: Ensures the explanation accurately represents the model's behavior in the local neighborhood. The proximity weights make sure accuracy is prioritized near the instance we're explaining.

2. Complexity penalty $\Omega(g)$: Ensures the explanation remains interpretable by highlighting only the most important feature contributions.

These two objectives pull in different directions. The loss term wants explanations that closely match the complex model's behavior. The complexity penalty wants simpler explanations with fewer features. The proximity measure ensures both objectives are evaluated in the context that matters: the local neighborhood.

Together, these components create an optimization problem whose solution gives us clear, interpretable attribution of feature contributions. The result accurately reflects how the complex model behaves for this specific instance while remaining simple enough for humans to understand.

The LIME optimization problem has several properties that affect how we use it in practice:

Non-convexity and practical solutions. The L0 penalty in $\Omega(g)$ makes the optimization problem non-convex, which means finding the global optimum is hard. In practice, most implementations use L1 regularization instead, replacing $\|w_g\|_0$ with $\|w_g\|_1$ (the sum of absolute values). This L1 penalty still encourages sparsity but creates a convex optimization problem we can solve efficiently. The tradeoff is acceptable: we lose theoretical guarantees about finding the sparsest solution, but we gain computational tractability and reliable convergence.

Sensitivity to bandwidth choice. The bandwidth parameter $\sigma$ in the proximity measure significantly affects explanation quality. Choose it carefully to balance local accuracy and generalization. If $\sigma$ is too small, we overfit to noise in the immediate vicinity. The explanation captures random fluctuations rather than meaningful patterns. If $\sigma$ is too large, we include instances outside the local region, violating the assumption that the model behaves linearly in the neighborhood. A good practice is to experiment with different values and check that explanations remain stable.

The linearity assumption. LIME assumes the complex model's behavior is approximately linear in the local neighborhood. This assumption doesn't always hold. Models with highly non-linear decision boundaries or complex feature interactions in the local region can violate it. When this happens, the linear explanation misses important aspects of the model's behavior, leading to misleading attributions. We should validate this assumption by checking the weighted loss. If it's consistently high, the linear approximation isn't capturing the model's local behavior well, and the explanations may not be trustworthy.