Mixtral › Architecture Overview

Mixtral builds directly on the Mistral 7B foundation. If you recall from Part XXIX, Mistral 7B introduced several efficiency-focused innovations: sliding window attention for long contexts, Grouped Query Attention (GQA) for faster inference, and RoPE for position encoding. Mixtral preserves all these components in its attention layers while replacing the dense feed-forward networks with MoE layers. This choice shows a key architectural insight: the attention mechanism handles sequence-level reasoning and information routing, while the feed-forward layers serve as the model's knowledge storage and feature transformation engine. By making only the feed-forward layers sparse, Mixtral maintains the attention mechanism's ability to integrate information across the full sequence and gains efficiency benefits from conditional computation in the knowledge-intensive FFN components.

Think of the attention layers as the brain's working memory, holding all relevant context simultaneously while deciding what matters. The feed-forward layers, by contrast, act more like long-term memory, translating a momentary representation into stored knowledge about language and facts, including reasoning patterns. This analogy motivates the architectural choice precisely: it makes sense to specialize long-term memory access (the FFN) while keeping working memory (attention) general-purpose and sequence-wide. Replacing only the FFN with sparse experts gives you specialized knowledge retrieval at inference time while preserving the model's ability to attend to the full context globally.

Another way to appreciate this design is to consider what would happen if attention were made sparse instead. Sparse attention would limit which tokens can communicate with which others, potentially breaking the global context understanding that lets tasks like question answering and summarization. Sparse feed-forward computation carries no such penalty: each token is processed independently by the FFN, so routing different tokens through different experts introduces no information loss. The MoE transformation of each token is computed in isolation, meaning the router can freely assign different tokens to different experts without creating dependencies that would break the attention computation.

The high-level architecture uses the decoder-only transformer pattern we've seen throughout the GPT family. Each Mixtral layer consists of the following components:

1. Multi-head attention with sliding window masking, GQA, and RoPE, identical to Mistral 7B
2. MoE feed-forward layer replacing the standard FFN, with 8 experts and top-2 routing

The key hyperparameters define the model's scale:

• Layers: 32 transformer blocks
• Hidden dimension: 4,096
• Attention heads: 32, with 8 key-value heads via GQA
• Experts per layer: 8
• Active experts per token: 2
• Expert FFN intermediate dimension: 14,336
• Vocabulary size: 32,000
• Context length: 32,768 tokens (with 4,096 sliding window)

These hyperparameters were not chosen arbitrarily: they reflect a careful study of the quality-efficiency frontier. The choice of 8 experts with top-2 routing, for instance, means 25% of expert capacity is activated per token. Fewer experts (say, 4) would reduce memory requirements but limit specialization. More experts (say, 64) would allow finer specialization but exponentially increase memory while giving diminishing returns in quality. The intermediate dimension of 14,336 is particularly noteworthy: it is substantially larger than the 11,008 used in Llama 2 7B's dense FFN. This shows the insight that each expert should be large enough to function as a capable specialist rather than a weak classifier. Smaller experts would require more of them to reach the same quality, worsening the memory footprint without improving efficiency.

The parameter count calculation reveals why Mixtral is simultaneously large and efficient. To understand this, we trace through where the parameters reside. Each expert FFN contains roughly 5.6 billion parameters. This count arises from the three projections (gate, up, and down) that map from hidden dimension 4,096 to intermediate dimension 14,336 and back, using the SwiGLU activation function. Specifically, each projection matrix contributes approximately $4096 \times 14336 \approx 58.7$ million parameters, and with three projections per expert and 8 experts across 32 layers, the total FFN parameters reach approximately 45 billion. The attention layers, embeddings, and output projections add another 1.7 billion, bringing the total to about 46.7 billion parameters.

However, since only 2 of 8 experts activate per token, the effective parameter count during inference is closer to 12.9 billion, comparable to running a 13B dense model. This distinction between total and active parameters is central to MoE efficiency: the model stores knowledge across all experts, but the computational cost per forward pass depends only on the subset that processes each token.