Sparse Models › Efficiency Analysis: Decoupling Parameters from Compute

The power of sparse models becomes clear when we quantify the efficiency gains. We can compare a dense model to a sparse model with equivalent computational cost to see the capacity advantage. This analysis reveals the mathematical foundation for why sparse models can reach better performance per unit of compute than their dense counterparts.

Consider a dense FFN with intermediate dimension $d_{\text{ff}}$. This layer has two weight matrices: $W_1 \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$ and $W_2 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$. These matrices define the expansion into a higher-dimensional space and the subsequent projection back to the model dimension.

The dense FFN has the following computational characteristics:

• Total parameters: $2 \times d_{\text{model}} \times d_{\text{ff}}$. This count comes from two weight matrices: $W_1 \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$ (containing $d_{\text{ff}} \times d_{\text{model}}$ parameters) and $W_2 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$ (containing $d_{\text{model}} \times d_{\text{ff}}$ parameters)
• FLOPs per token: $4 \times d_{\text{model}} \times d_{\text{ff}}$, because each of the two matrix multiplications requires approximately $2 \times d_{\text{model}} \times d_{\text{ff}}$ FLOPs (standard approximation that a matrix multiplication of dimensions $m \times n$ by $n \times p$ requires $2mnp$ FLOPs).

where:

• $d_{\text{model}}$: the dimension of the model's hidden representations (the size of token embeddings throughout the network)
• $d_{\text{ff}}$: the intermediate dimension of the feed-forward network, typically set to $4 \times d_{\text{model}}$
• $2mnp$: the FLOP count for matrix multiplication comes from computing $m \times p$ output elements, each requiring $n$ multiplications and $n$ additions, giving approximately $2mnp$ FLOPs

Now consider a sparse layer with $N$ experts, where each expert has intermediate dimension $d_{\text{ff}}/N$, using top-2 routing. This configuration divides the total parameters across multiple smaller experts, each specialized for different types of inputs. The top-2 routing means each token will be processed by exactly two experts, combining their outputs with learned weights.

For this sparse configuration, we can compute the total parameter count, active parameters per token, and FLOPs per token. These derivations illuminate exactly where the efficiency gains originate.

Total parameters: Each expert is a smaller FFN with intermediate dimension $d_{\text{ff}}/N$ instead of the full $d_{\text{ff}}$. Since each expert has two weight matrices, it contains $2 \times d_{\text{model}} \times (d_{\text{ff}}/N)$ parameters. The expert's reduced size means it specializes in a narrower range of computations, but collectively the experts cover the same representational space as the original dense layer. Summing across all $N$ experts:

$$\begin{aligned} \text{Total sparse parameters} &= N \times 2 \times d_{\text{model}} \times (d_{\text{ff}}/N) && \text{(sum over all experts)} \\ &= 2 \times d_{\text{model}} \times d_{\text{ff}} \times \frac{N}{N} && \text{(rearrange)} \\ &= 2 \times d_{\text{model}} \times d_{\text{ff}} && (\text{cancel } N) \end{aligned}$$

This matches the dense model exactly, since the $N$ terms cancel. This result is important because it shows that we haven't reduced capacity: the total number of learnable parameters remains the same. The parameters are simply distributed across multiple experts rather than concentrated in a single network.

Active parameters per token: The key efficiency gain comes from activating only a subset of experts. With top-2 routing, only 2 out of $N$ experts activate for any given token. Each active expert contributes its $2 \times d_{\text{model}} \times (d_{\text{ff}}/N)$ parameters to the computation. This selective activation is the heart of sparse efficiency:

$$\begin{aligned} \text{Active parameters} &= \text{top-k} \times 2 \times d_{\text{model}} \times (d_{\text{ff}}/N) && \text{(k active experts)} \\ &= 2 \times 2 \times d_{\text{model}} \times (d_{\text{ff}}/N) && \text{(substitute top-k = 2)} \\ &= \frac{4 \times d_{\text{model}} \times d_{\text{ff}}}{N} && \text{(simplify)} \end{aligned}$$

FLOPs per token: The computational cost follows the active parameters, since we only perform computations for the experts that are activated. Each active expert must perform two matrix multiplications (expansion and contraction), costing $4 \times d_{\text{model}} \times (d_{\text{ff}}/N)$ FLOPs. With 2 active experts:

$$\begin{aligned} \text{Sparse FLOPs} &= \text{top-k} \times 4 \times d_{\text{model}} \times (d_{\text{ff}}/N) && \text{(k active experts, each costs } 4 \times d_{\text{model}} \times (d_{\text{ff}}/N) \text{ FLOPs)} \\ &= 2 \times 4 \times d_{\text{model}} \times (d_{\text{ff}}/N) && \text{(substitute top-k = 2)} \\ &= \frac{8 \times d_{\text{model}} \times d_{\text{ff}}}{N} && \text{(simplify)} \end{aligned}$$

With $N = 8$ experts, the sparse model uses only $1/4$ the FLOPs per token while maintaining the same total parameter count. This is a remarkable result: we get the same capacity at a fraction of the computational cost. We can verify this by computing the ratio:

$$\begin{aligned} \text{Sparse FLOPs ratio} &= \frac{\text{Sparse FLOPs per token}}{\text{Dense FLOPs per token}} && \text{(definition)} \\ &= \frac{8 \times d_{\text{model}} \times d_{\text{ff}}/N}{4 \times d_{\text{model}} \times d_{\text{ff}}} && \text{(substitute formulas)} \\ &= \frac{8/N}{4} && (\text{cancel } d_{\text{model}} \times d_{\text{ff}}) \\ &= \frac{2}{N} && \text{(simplify)} \\ &= \frac{2}{8} = \frac{1}{4} && \text{(substitute } N = 8\text{)} \end{aligned}$$

This $1/4$ ratio reveals the key insight: with 8 experts and top-2 routing, we use only 25% of the dense model's compute per token. The general formula $2/N$ shows that computational cost decreases linearly with the number of experts, while total model capacity (parameter count) increases linearly with $N$. This inverse relationship between experts and compute cost is the mathematical foundation for sparse model efficiency.

A significant advantage of sparse models is the ability to increase total parameters without increasing compute. Rather than keeping the same total parameter count as the dense model, we can make each expert the same size as the original dense FFN, giving us $N$ times more parameters while only paying $k/N$ times the compute. This is the real power of sparse architectures: they let us scale capacity independently of computational cost.

The sparse model reaches materially higher parameter count while maintaining reasonable computational cost. With 8 experts and top-2 routing, the total parameter count is 8x larger than the dense baseline, but FLOPs per token only double because each token activates just 2 of the 8 experts. This creates a 4x improvement in capacity per unit of compute. The dense model's single FFN processes every token, while the sparse model distributes capacity across multiple experts and activates only a subset per token. This basic trade-off allows sparse models to scale capacity beyond what would be computationally feasible with dense architectures.

In practice, this lets a new training and deployment paradigm:

The sparse model processes each token with similar cost to a ~12B dense model but has the knowledge capacity of a much larger model. This has measurable implications for the scaling laws we discussed in Part XXIII: we can push the Pareto frontier of performance vs. compute. The sparse architecture in effect lets us move along a different efficiency curve, one where we can trade increased memory (for storing more expert parameters) for improved quality at fixed compute.