Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Sparse Models › Dense Models and Their Limitations

Michael BrenndoerferUpdated December 31, 20257 min read

Language AI HandbookAll chapters

Part 33 of 67Mixture of Experts

Chapter 239Sparse Models › Dense Models and Their Limitations

Reading section 2 of 10 · Dense Models and Their Limitations · 7 minSection 2 of 10 · 7 min

Section 2 of 10 · 7 min read

Sparse Models › Dense Models and Their LimitationsOne section · Sections in this chapter2 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Every transformer architecture we've studied so far follows the same pattern. When a token enters the model, it flows through every layer, every attention head, and every feed-forward network. The model's full parameter count participates in every computation. This uniformity of computation, where the same operations apply regardless of the input's content, has been both a strength and a limitation of the transformer architecture.

Dense Model

A model where all parameters are activated and participate in the forward pass for every input. The computational cost scales directly with the total parameter count.

The "dense" property means every parameter contributes to every prediction, creating a direct relationship between model size and computational cost. If we double the parameters, we double the compute required per token. This tight coupling between capacity and computation is a design choice rather than a basic constraint of neural networks.

Consider the feed-forward network we examined in Part XV. For a model with hidden dimension $d_{\text{model}}$ and FFN intermediate dimension $d_{\text{ff}}$ , each token passes through two linear transformations. The feed-forward network is where sparse models make their intervention.

The feed-forward network transforms each token through an expansion to a larger intermediate dimension, applies a non-linearity, then projects back to the original dimension. This two-stage transformation allows the network to learn complex non-linear mappings. The network first projects into a higher-dimensional space where patterns are easier to separate, then projects back to the model dimension. This expansion and contraction pattern is remarkably effective at capturing complex relationships in the data. Given an input token $x$ , the FFN computes:

\text{FFN}(x) = W_2 \cdot \sigma(W_1 x + b_1) + b_2

where:

$x$ : input token representation with dimension $d_{\text{model}}$
$W_1 \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$ : first weight matrix that projects to the intermediate dimension
$b_1 \in \mathbb{R}^{d_{\text{ff}}}$ : bias vector for the first transformation
$\sigma$ : activation function (typically ReLU or GELU) applied element-wise
$W_2 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$ : second weight matrix that projects back to $d_{\text{model}}$
$b_2 \in \mathbb{R}^{d_{\text{model}}}$ : bias vector for the output transformation

The activation function has two key properties that make it ideal for the transformation. It ensures non-linearity in the transformation and allows the network to learn complex patterns by projecting into a space where features can be more easily separated before projecting back to the model dimension. Without this non-linearity, the composition of two linear transformations would simply collapse into a single linear transformation, severely limiting the network's expressive power.

We can derive the total parameter count for these two weight matrices as follows. This derivation helps us understand exactly where the computational burden lies in a transformer layer:

\begin{aligned} \text{Total FFN parameters} &= |W_1| + |W_2| && \text{(sum of both weight matrices)} \\ &= (d_{\text{ff}} \times d_{\text{model}}) + (d_{\text{model}} \times d_{\text{ff}}) && \text{(dimensions of each matrix)} \\ &= 2 \times d_{\text{ff}} \times d_{\text{model}} && \text{(combine terms)} \end{aligned}

In a typical transformer where $d_{\text{ff}} = 4 \times d_{\text{model}}$ , substituting into our formula gives $2 \times d_{\text{ff}} \times d_{\text{model}} = 2 \times (4 \times d_{\text{model}}) \times d_{\text{model}} = 8 \times d_{\text{model}}^2$ parameters per FFN layer. To put this in concrete terms, for GPT-3 with $d_{\text{model}} = 12288$ , that's over 1.2 billion parameters per layer, and every single parameter gets used for every single token. This enormous parameter count per layer, multiplied across all layers, explains why large language models require such substantial hardware resources.

This creates a coupling between three quantities that we might prefer to scale independently:

Model capacity: The ability to store and process complex patterns, which is tied to parameter count
Training compute: The FLOPs required to update parameters during training
Inference compute: The FLOPs required to process each token at runtime

In dense models, these three quantities move together in lockstep. Want more capacity? Add parameters. More parameters means more compute for both training and inference. The Chinchilla scaling laws help us balance compute and data efficiently, but they don't break this basic coupling. This coupling is a significant constraint: we cannot increase the model's knowledge capacity without simultaneously increasing the cost of using that knowledge.

The practical impact is severe. A 175 billion parameter dense model requires roughly 350 billion FLOPs per token (using the approximation that FLOPs = 2 times parameters per forward pass). At scale, this determines your hardware requirements, inference latency, and operating costs. For organizations deploying these models to millions of users, even small inefficiencies translate into substantial infrastructure expenses.

In[3]:

Code

def estimate_dense_model_cost(
    params_billions: float, tokens_per_second: float = 1000
) -> dict:
    """Estimate computational costs for a dense model."""
    params = params_billions * 1e9

    # FLOPs per token: forward pass only, approximately 2x parameters
    flops_per_token = 2 * params

    # Total FLOPs per second
    flops_per_second = flops_per_token * tokens_per_second

    # A100 GPU theoretical peak: 312 TFLOPS (FP16)
    a100_tflops = 312e12

    # Realistic utilization typically ranges from 30-50%
    utilization = 0.4
    effective_tflops = a100_tflops * utilization

    # GPUs needed for real-time inference
    gpus_needed = flops_per_second / effective_tflops

    return {
        "params_billions": params_billions,
        "flops_per_token": flops_per_token,
        "tflops_per_second": flops_per_second / 1e12,
        "a100_gpus_needed": gpus_needed,
    }


model_sizes = [7, 70, 175, 540]

Out[4]:

Console

7B parameters: 1.40e+10 FLOPs/token, 0.1 A100 GPUs needed
70B parameters: 1.40e+11 FLOPs/token, 1.1 A100 GPUs needed
175B parameters: 3.50e+11 FLOPs/token, 2.8 A100 GPUs needed
540B parameters: 1.08e+12 FLOPs/token, 8.7 A100 GPUs needed

Out[5]:

Visualization

Line chart on log-log axes showing TFLOPs per token versus model size in billions of parameters, with values rising linearly from 7B to 540B. — FLOPs per token scaling with model size, reaching approximately 1.1 TFLOPs for 540B parameter models, showing the tight coupling between capacity and compute.

Line chart on log-log axes showing the number of A100 GPUs required for real-time inference versus model size, scaling from under 1 GPU at 7B to nearly 9 GPUs at 540B. — Hardware requirements scaling with model size, requiring nearly 9 A100 GPUs for real-time inference at 1,000 tokens per second.

The results reveal the scaling problem. As model size increases from 7B to 540B parameters, computational requirements grow proportionally. The 7B model processes 1000 tokens per second with approximately 1 A100 GPU, while the 540B model requires nearly 9 GPUs for the same throughput. This linear scaling between parameters and compute creates a basic efficiency barrier. Even with optimized hardware running at 40% utilization, the infrastructure costs become prohibitive at large scales. This growth in hardware requirements motivates the search for sparse architectures that can decouple model capacity from computational cost.