Mistral Architecture › Mistral vs LLaMA: Architectural Comparison

Michael BrenndoerferUpdated August 6, 20252 min read
Language AI HandbookAll chapters
Part 29 of 67Modern Decoder Models
Chapter 211Mistral Architecture › Mistral vs LLaMA: Architectural Comparison
Reading section 8 of 14 · Mistral vs LLaMA: Architectural Comparison · 2 minSection 8 of 14 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 211 of 513

Section 8 of 14 · 2 min read

Mistral Architecture › Mistral vs LLaMA: Architectural ComparisonOne section · Sections in this chapter8 / 14 ↓
Sections in this chapter14
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's directly compare the architectural choices between Mistral 7B and LLaMA 2 7B to understand what changed:

In[28]:
Code
llama2_7b_config = {
    "hidden_size": 4096,
    "num_layers": 32,
    "num_attention_heads": 32,
    "num_kv_heads": 32,  # Full MHA in base LLaMA 2 7B
    "intermediate_size": 11008,
    "vocab_size": 32000,
    "max_position_embeddings": 4096,
    "sliding_window": None,  # Full attention
}


def compare_configs(config1, config2, name1, name2):
    """Compare two model configurations."""
    all_keys = set(config1.keys()) | set(config2.keys())

    comparison = []
    for key in sorted(all_keys):
        val1 = config1.get(key, "N/A")
        val2 = config2.get(key, "N/A")
        if val1 != val2:
            comparison.append((key, val1, val2, "←"))
        else:
            comparison.append((key, val1, val2, ""))

    return comparison
Out[29]:
Console
Architecture Comparison: LLaMA 2 7B vs Mistral 7B

Parameter                          LLaMA 2 7B     Mistral 7B  Diff
--------------------------------------------------------------------
Hidden Size                             4,096          4,096      
Intermediate Size                      11,008         14,336     ←
Max Position Embeddings                 4,096          8,192     ←
Num Attention Heads                        32             32      
Num Kv Heads                               32              8     ←
Num Layers                                 32             32      
Rope Theta                                N/A        10000.0     ←
Sliding Window                           Full          4,096     ←
Vocab Size                             32,000         32,000

The key differences are:

  1. KV heads: Mistral uses 8 KV heads (GQA) vs LLaMA's 32 (MHA), reducing KV cache by 4x
  2. FFN size: Mistral uses a larger intermediate dimension (14336 vs 11008), adding capacity
  3. Context length: Mistral supports 8K tokens vs LLaMA's 4K
  4. Sliding window: Mistral uses a 4096-token window, LLaMA uses full attention

These changes create a more efficient model that can handle longer contexts with less memory.