Mistral Architecture › Architectural Foundation
Language AI HandbookAll chapters Mistral Architecture › Architectural FoundationOne section · Sections in this chapter2 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Mistral 7B inherits the core structure of LLaMA while introducing targeted improvements. Understanding these changes requires first recognizing what stays the same: Mistral uses a decoder-only transformer with RMSNorm for layer normalization, SwiGLU for the feed-forward network, and rotary positional embeddings (RoPE) for encoding position information. These components, proven effective in LLaMA, form the foundation upon which Mistral builds.
The key architectural parameters for Mistral 7B are:
- Hidden dimension: 4096
- Number of layers: 32
- Number of attention heads: 32
- Number of KV heads: 8 (grouped query attention)
- Feed-forward hidden dimension: 14336
- Vocabulary size: 32000
- Context length: 8192 tokens (sliding window of 4096)
- Total parameters: 7.3 billion
Three innovations distinguish Mistral from its predecessors: sliding window attention (SWA), rolling buffer KV cache, and pre-fill chunking. Together, these enable efficient processing of long sequences while maintaining strong performance.
# Mistral 7B configuration
mistral_config = {
"hidden_size": 4096,
"num_layers": 32,
"num_attention_heads": 32,
"num_kv_heads": 8,
"intermediate_size": 14336,
"vocab_size": 32000,
"max_position_embeddings": 8192,
"sliding_window": 4096,
"rope_theta": 10000.0,
}Mistral 7B Configuration: ---------------------------------------- Hidden Size 4,096 Num Layers 32 Num Attention Heads 32 Num Kv Heads 8 Intermediate Size 14,336 Vocab Size 32,000 Max Position Embeddings 8,192 Sliding Window 4,096 Rope Theta 10000.0
The configuration reveals Mistral's efficiency-focused design. The 8 KV heads (compared to 32 query heads) indicate grouped query attention, while the 4096-token sliding window enables the model to handle 8K context lengths without quadratic memory growth.