SHAP (SHapley Additive exPlanations) › Formula

At its heart, SHAP answers a deceptively simple question: How much does each feature contribute to this specific prediction? Yet answering this question fairly requires accounting for a fundamental challenge: the contribution of a feature depends on what other features are present. A feature might have a large impact when considered alone, but a smaller impact when other correlated features are already in the model. SHAP solves this attribution problem by drawing on a concept from cooperative game theory called Shapley values, which provide a mathematically principled way to fairly distribute credit among contributors.

The key insight is that we cannot simply ask "what happens when we add feature $i$?" because the answer depends on the context, the set of other features that are already present. Instead, we should consider feature $i$'s contribution across all possible contexts, then average these contributions in a way that ensures fairness. This journey from intuitive question to rigorous formula guides our understanding of how SHAP works.

To understand why SHAP's formula is structured the way it is, we should first appreciate the fundamental problem it addresses. Imagine you're trying to explain why a machine learning model predicted a house price of $235,000. You have three features: square footage, number of bedrooms, and age of the house. If you simply ask "how much does square footage contribute?", the answer isn't straightforward.

If we look at the difference between the prediction with all features ($f(\text{all}) = 235$) and the prediction with square footage removed ($f(\text{bedrooms, age}) = 110$), we might conclude square footage contributes $235 - 110 = 125$. But what if we had started from a different baseline? What if we removed bedrooms instead, or age? The contribution would appear different depending on what context we assume.

This reveals the core challenge: a feature's marginal contribution depends on what other features are already in the model. A feature might have a large impact when added to an empty set, but a smaller impact when added to a set that already contains correlated features. To be fair, we need to consider every possible way the feature could have been added to the model, across all possible sequences and contexts.

SHAP solves this by averaging the feature's marginal contribution across all possible subsets of other features, weighted by the probability of encountering each subset in a random ordering. This ensures that the attribution doesn't depend on the arbitrary order in which features are considered, but rather reflects the feature's true average contribution across all possible contexts.

The foundation of SHAP lies in understanding marginal contributions, the change in prediction that occurs when we add feature $i$ to a specific subset of other features. Formally, for a given subset $S$ that doesn't include feature $i$, the marginal contribution is:

$$[f(S \cup \{i\}) - f(S)]$$

This captures a simple but important idea: what does feature $i$ add when it's included alongside the features in $S$, compared to what the prediction would have been with just $S$?

However, this single marginal contribution tells only part of the story. The same feature might contribute differently depending on what other features are present. For example, square footage might contribute more when considered alone (because it's picking up information that would otherwise be captured by bedrooms), but contribute less when bedrooms are already in the model (because some of the information overlaps). To get a fair attribution, we need to look at the feature's marginal contribution across all possible subsets $S$.

Simply averaging the marginal contributions would be unfair because not all subsets are equally likely or important. Some subsets might be more representative of how the feature typically appears in practice. This is where the weighting factor comes in. It ensures that each marginal contribution is weighted appropriately based on how likely we are to encounter that particular context.

The SHAP value for feature $i$ in a prediction is given by:

$$\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!(|F|-|S|-1)!}{|F|!} [f(S \cup \{i\}) - f(S)]$$

This formula combines three essential components:

The Summation: $\sum_{S \subseteq F \setminus \{i\}}$ ensures we consider every possible subset $S$ of features that doesn't include feature $i$. This exhaustive enumeration guarantees that we've examined the feature's contribution across all conceivable contexts, leaving no scenario unaccounted for.

The Marginal Contribution: $[f(S \cup \{i\}) - f(S)]$ represents how much feature $i$ adds when included alongside subset $S$. This is the raw signal of the feature's impact in each specific context. Notice that this value can differ dramatically depending on $S$. A feature might have a large positive contribution when added to an empty set, but a smaller or even negative contribution when added to a set containing correlated features.

The Weighting Factor: $\frac{|S|!(|F|-|S|-1)!}{|F|!}$ is the sophisticated piece that ensures fairness. This combinatorial weight doesn't just count subsets equally. It accounts for the probability that, if we randomly ordered all features, we would encounter subset $S$ before feature $i$, with all remaining features coming after. This probability-based weighting ensures that the final attribution doesn't depend on any arbitrary ordering of features, but reflects the feature's average contribution across all possible orderings.

These three components work together to produce a fair, mathematically principled attribution: we enumerate all possible contexts (summation), measure the feature's impact in each context (marginal contribution), and then average these impacts weighted by how naturally each context arises (weighting factor). The result doesn't depend on the order in which features are considered or the baseline we happen to choose.

To make this concrete, let's walk through a visual example that demonstrates how all these pieces fit together. The visualization below shows the complete calculation process for computing the SHAP value of feature $x_1$ in a model with three features.

You'll see four subsets $S$ (shown in blue circles): the empty set $\emptyset$, the singleton sets $\{x_2\}$ and $\{x_3\}$, and the pair $\{x_2, x_3\}$. For each subset, the diagram shows:

• The marginal contribution $[f(S \cup \{x_1\}) - f(S)]$ (shown as red-bordered boxes with values like "125")
• The weighting factor that applies to that marginal contribution (shown in green boxes)
• How these weighted contributions combine to produce the final SHAP value $\phi_1$

In this particular example, all marginal contributions happen to equal 125, which simplifies the calculation. In general, these values can vary depending on the model's structure and how features interact with each other. The key insight to take away is how the formula systematically considers every possible context, weights each appropriately, and combines them to produce a single, fair attribution value.

The visualization makes visible what the formula encodes: fair attribution requires us to consider all contexts and weight them by how naturally they arise in random orderings of features.

The weighting factor $\frac{|S|!(|F|-|S|-1)!}{|F|!}$ is central to ensuring fairness in SHAP's attribution scheme. While it might look intimidating at first, it encodes a simple idea: we should weight each marginal contribution by how likely we are to encounter that particular context in a random ordering of features.

To understand why this matters, consider what happens if we simply averaged all marginal contributions equally. Some subsets $S$ would be over-represented (like the empty set or the full set), while others would be under-represented. But more fundamentally, different subsets represent different "stages" in the feature addition process. If we imagine features being added one by one in some order, then subset $S$ represents all the features that would come before feature $i$ in that ordering.

The weighting factor captures this probabilistic interpretation: if we randomly shuffle all features, what's the probability that exactly the features in subset $S$ come before feature $i$, with all remaining features coming after?

Let's break down the formula piece by piece:

• $|S|!$: This counts the number of ways to arrange the features in subset $S$ among themselves. If $S$ contains $k$ features, there are $k!$ ways to order them, but what matters here is their position relative to feature $i$, not their internal ordering.

• $(|F|-|S|-1)!$: This counts the number of ways to arrange the features that come after feature $i$. Since we have $|F|$ total features, $|S|$ features come before $i$, feature $i$ itself is in the middle, leaving $(|F|-|S|-1)$ features to come after, which can be arranged in $(|F|-|S|-1)!$ ways.

• $|F|!$: This is the total number of possible orderings of all features, the denominator that normalizes our probability calculation.

The ratio $\frac{|S|!(|F|-|S|-1)!}{|F|!}$ therefore represents the probability that, in a random ordering, we find exactly this configuration: all features in $S$ before feature $i$, all remaining features after feature $i$.

Why is this the right weight? Because it ensures that each marginal contribution is weighted by how "typical" or "representative" that context is. Subsets that are more likely to appear in random orderings (like medium-sized subsets) get appropriate weight, while extreme subsets (like the empty set or nearly complete sets) get weights that reflect their relative rarity. This probabilistic weighting guarantees that the final SHAP value doesn't depend on any arbitrary ordering we might choose. It's the average contribution across all possible orderings, weighted by how naturally each ordering arises.

This probabilistic interpretation of the weighting factor completes our understanding of how SHAP achieves fairness. By weighting each marginal contribution by the probability of encountering its context in a random ordering, we ensure that the final SHAP value is independent of any arbitrary feature ordering. The formula doesn't favor one subset over another arbitrarily. It weights them according to how naturally they arise, creating an attribution scheme that is both mathematically principled and intuitively fair.

Now that we understand the individual components of the SHAP formula, we can see how they work together to create a coherent whole. The efficiency property provides an important sanity check that demonstrates how the formula's structure ensures completeness:

$$\sum_{i=1}^{|F|} \phi_i = f(F) - f(\emptyset)$$

This equation states that the sum of all SHAP values equals the difference between the model's prediction using all features and the baseline prediction (using no features).

Why is this important? It guarantees that SHAP provides a complete decomposition of the prediction. Every unit of difference between the full prediction and the baseline is accounted for. There are no "orphaned" contributions floating around, and nothing is double-counted. This is a mathematical guarantee that emerges naturally from the way SHAP averages marginal contributions across all subsets.

To see why this holds, consider what happens when we sum all the SHAP values. Each feature's SHAP value is a weighted average of its marginal contributions across all subsets. When we sum these up, we're essentially reconstructing the path from baseline ($f(\emptyset)$) to full prediction ($f(F)$) by adding features one at a time, across all possible orderings. The mathematical structure ensures that this reconstruction is exact. The weighted sum of all marginal contributions perfectly captures the total difference between baseline and final prediction.

This efficiency property is what makes SHAP explanations interpretable and actionable: if you start with the baseline and add each feature's SHAP contribution, you arrive exactly at the model's prediction. There's no missing piece and no unexplained remainder. Just a clean, complete decomposition.

The efficiency property is just one of several fundamental guarantees that SHAP values provide. These properties aren't arbitrary design choices. They emerge naturally from the mathematical structure of Shapley values and give SHAP explanations a consistent and principled basis:

Efficiency: As we've just seen, the sum of all SHAP values equals $f(F) - f(\emptyset)$. This ensures complete attribution with no missing pieces. Every unit of difference between prediction and baseline is accounted for.

Symmetry: If two features $i$ and $j$ have identical marginal contributions in all possible contexts (i.e., $f(S \cup \{i\}) - f(S) = f(S \cup \{j\}) - f(S)$ for all subsets $S$), then they receive identical SHAP values: $\phi_i = \phi_j$. This guarantees fair treatment of equivalent features. Features that contribute equally in all contexts are attributed equally.

Dummy: If a feature never changes the prediction regardless of what other features are present (i.e., $f(S \cup \{i\}) - f(S) = 0$ for all subsets $S$), then its SHAP value is zero: $\phi_i = 0$. This ensures that irrelevant features get zero credit. The mathematical structure naturally filters out features that don't contribute.

Additivity: For models that decompose as a sum of sub-models (e.g., $f(x) = f_1(x) + f_2(x)$), the SHAP values of the combined model equal the sum of the SHAP values from the individual models: $\phi_i^{f_1+f_2} = \phi_i^{f_1} + \phi_i^{f_2}$. This means SHAP explanations compose naturally, making them useful for understanding ensemble models and complex systems.

Together, these properties form a mathematical foundation for SHAP explanations. They ensure that the attributions are not arbitrary or dependent on implementation details, but rather reflect fundamental properties of how features contribute to predictions. This mathematical rigor is what distinguishes SHAP from simpler attribution methods. It provides guarantees that hold regardless of the specific model or data, making the explanations reliable when interpreting models, debugging predictions, or explaining results to stakeholders.