Mixtral › Efficiency Analysis
Language AI HandbookAll chapters Mixtral › Efficiency AnalysisOne section · Sections in this chapter9 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The efficiency case for Mixtral centers on the ratio between total and active parameters. While the model requires storing all 46.7B parameters in memory, each forward pass only activates 12.9B; this results in a 72% reduction in compute compared to a hypothetical model that activated all parameters. This section quantifies these efficiency gains precisely.
Efficiency is often discussed loosely in the LLM literature, with claims about parameter counts, FLOPs, and inference speed sometimes conflated or used interchangeably. For a precise analysis of Mixtral, we need to distinguish between three different metrics that each tell a different part of the story. First, total parameter count determines how much GPU memory you need to load the model. Second, active parameter count per forward pass determines compute cost and therefore throughput: this is the metric that makes Mixtral fast despite being large. Third, training compute is the product of active parameters, training tokens, and training steps: MoE models are efficient at inference but not necessarily at training, because every parameter must be updated during training even if it is rarely active.
Think of the distinction between total and active parameters as the difference between the size of a city's library system and the number of books you check out per visit. The library needs physical shelf space for every book (total parameters), but you only read a few books at a time (active parameters). A larger library with more specialized books does not slow down your reading if you are selective about what you check out. Mixtral's efficiency comes from maintaining a large, specialized library while only "reading" a small, carefully chosen subset for each token.
The memory-compute gap is Mixtral's most important practical characteristic. It creates a situation where Mixtral is slow to load (requires ~93GB in FP16 to store all weights) but fast to run once loaded (processes tokens with 12.9B active parameters worth of compute). This makes Mixtral well-suited for workloads where the model stays loaded and handles many requests, but poorly suited for occasional-use scenarios where the model is loaded and unloaded frequently.
Computational Cost Breakdown
Let's calculate the actual FLOPs for a forward pass.
def calculate_mixtral_flops(
seq_len: int,
hidden_dim: int = 4096,
ffn_dim: int = 14336,
num_layers: int = 32,
num_experts: int = 8,
top_k: int = 2,
vocab_size: int = 32000,
num_heads: int = 32,
head_dim: int = 128,
) -> dict:
"""Calculate FLOPs for Mixtral forward pass.
For matrix multiplication C = AB with A of shape (m, k) and B of shape (k, n),
each output element requires k multiply-add operations. With m×n output elements,
the total FLOP count is approximately 2mkn: the factor of 2 accounts for
one multiply and one add per operation. We simplify to mkn for order-of-magnitude
estimates, omitting the constant factor of 2.
"""
# Embedding lookup (negligible for FLOP counting)
embed_flops = 0
# Per-layer attention FLOPs (approximate)
# QKV projection: 3 matrices of (seq_len, hidden_dim) @ (hidden_dim, hidden_dim)
qkv_flops = 3 * seq_len * hidden_dim * hidden_dim
# Attention scores: (seq_len, hidden_dim) @ (hidden_dim, seq_len) simplified
attn_score_flops = seq_len * seq_len * hidden_dim
# Output projection: (seq_len, hidden_dim) @ (hidden_dim, hidden_dim)
out_proj_flops = seq_len * hidden_dim * hidden_dim
attention_flops_per_layer = qkv_flops + attn_score_flops + out_proj_flops
# Per-layer MoE FLOPs
# Router: (seq_len, hidden_dim) @ (hidden_dim, num_experts)
router_flops = seq_len * hidden_dim * num_experts
# Each active expert: gate, up, down projections
# Gate: (seq_len, hidden_dim) @ (hidden_dim, ffn_dim)
# Up: (seq_len, hidden_dim) @ (hidden_dim, ffn_dim)
# Down: (seq_len, ffn_dim) @ (ffn_dim, hidden_dim)
# Total: 2 * seq_len * hidden_dim * ffn_dim + seq_len * ffn_dim * hidden_dim
expert_flops = 3 * seq_len * hidden_dim * ffn_dim
# Total for top_k experts
moe_flops_per_layer = router_flops + top_k * expert_flops
# Total for all layers
total_flops = num_layers * (attention_flops_per_layer + moe_flops_per_layer)
# Output projection
output_flops = seq_len * hidden_dim * vocab_size
total_flops += output_flops
# For comparison: equivalent dense model
dense_ffn_dim = top_k * ffn_dim # Same compute as top-2 routing
dense_flops_per_layer = (
attention_flops_per_layer + 3 * seq_len * hidden_dim * dense_ffn_dim
)
dense_total = num_layers * dense_flops_per_layer + output_flops
return {
"mixtral_total_gflops": total_flops / 1e9,
"attention_fraction": (num_layers * attention_flops_per_layer)
/ total_flops,
"moe_fraction": (num_layers * moe_flops_per_layer) / total_flops,
"equivalent_dense_gflops": dense_total / 1e9,
}
# Calculate for typical sequence length
flops = calculate_mixtral_flops(seq_len=2048)Mixtral 8x7B Computational Analysis (2048 tokens): Total forward pass: 28308.1 GFLOPs Attention fraction: 17.5% MoE fraction: 81.6% Equivalent dense model: 28306.0 GFLOPs Ratio (dense/sparse): 1.00x
The compute is dominated by the MoE layers, which is expected given the large FFN intermediate dimension. The reason is that Mixtral reaches its quality by having large individual experts with 14,336 intermediate dimension, rather than many small ones. Each active expert is in effect a 7B-class FFN. This gives large capacity per expert.
Memory Requirements
While compute scales with active parameters, memory requirements include all parameters. This distinction is important for deployment planning; you can reach fast inference with MoE, but you still need enough memory to store the full model.
def estimate_memory(
total_params_b: float,
active_params_b: float,
batch_size: int,
seq_len: int,
hidden_dim: int,
num_layers: int,
precision_bytes: int = 2, # FP16/BF16
) -> dict:
"""Estimate memory requirements for inference."""
# Model weights
weight_memory_gb = total_params_b * 1e9 * precision_bytes / 1e9
# KV cache per layer stores keys and values for all positions
# Standard: 2 × batch_size × seq_len × hidden_dim × precision_bytes
# The factor of 2 accounts for both keys and values
# With GQA (8 KV heads vs 32 Q heads), we reduce by 32/8 = 4×
# Each KV head serves multiple query heads: with 32 query heads and 8 KV heads,
# each KV head serves 4 query heads, so we only need to cache 1/4 of the
# key-value pairs compared to standard MHA.
kv_cache_per_layer = (
2 * batch_size * seq_len * hidden_dim * precision_bytes / 4
)
kv_cache_total_gb = (num_layers * kv_cache_per_layer) / 1e9
# Activation memory (rough estimate)
activation_gb = (
batch_size * seq_len * hidden_dim * precision_bytes * 2 / 1e9
)
return {
"weights_gb": weight_memory_gb,
"kv_cache_gb": kv_cache_total_gb,
"activations_gb": activation_gb,
"total_gb": weight_memory_gb + kv_cache_total_gb + activation_gb,
}
# Mixtral memory requirements
mixtral_mem = estimate_memory(
total_params_b=46.7,
active_params_b=12.9,
batch_size=1,
seq_len=4096,
hidden_dim=4096,
num_layers=32,
)
# Equivalent dense model (Llama 2 70B)
llama_mem = estimate_memory(
total_params_b=70.0,
active_params_b=70.0,
batch_size=1,
seq_len=4096,
hidden_dim=8192,
num_layers=80,
)Memory Requirements Comparison: Component Mixtral 8x7B Llama 2 70B -------------------------------------------------- Model weights 93.4 GB 140.0 GB KV cache (4K ctx) 0.54 GB 2.68 GB Activations 0.07 GB 0.13 GB -------------------------------------------------- Total 94.0 GB 142.8 GB
Mixtral requires materially less memory than Llama 2 70B, primarily due to fewer total parameters and a smaller KV cache, thanks to GQA and smaller hidden dimension. At FP16, the estimate still exceeds a single 80 GB accelerator; quantization or multiple devices are required. The lower total makes Mixtral easier to deploy than Llama 2 70B, but not a 48 GB single-card model at this precision.
Inference Throughput
The efficiency gains translate directly to inference speed. Mixtral typically reaches 3 to 6 times higher throughput than Llama 2 70B for equivalent quality outputs.
The scatter plot reveals the key efficiency insight. Mixtral occupies a unique position in the quality-speed tradeoff space. Llama 2 13B offers similar throughput but much lower quality. Llama 2 70B matches Mixtral's quality but at 4 times lower throughput. Mixtral effectively breaks the linear quality-compute tradeoff that dense models exhibit, achieving high quality at materially lower compute cost.