Mistral Architecture › Key Parameters

The following parameters control Mistral's efficiency-capability trade-offs:

• sliding_window (int, default=4096): The number of previous tokens each position can attend to. Larger values provide more direct context access but increase memory usage linearly. Mistral uses 4096, which balances efficiency with sufficient local context. For tasks requiring precise long-range retrieval, consider larger windows or full attention.

• num_kv_heads (int, default=8): The number of key-value heads in grouped query attention. Must divide num_attention_heads evenly. Lower values reduce KV cache size but may limit representational diversity. Mistral's ratio of 32:8 (4:1) provides a good balance.

• num_attention_heads (int, default=32): The number of query heads. Each group of num_attention_heads / num_kv_heads query heads shares one KV head pair. More heads enable finer-grained attention patterns.

• hidden_size (int, default=4096): The model dimension. Determines the size of embeddings and the input/output dimension for each layer. Head dimension is computed as hidden_size / num_attention_heads.

• intermediate_size (int, default=14336): The hidden dimension of the feed-forward network. Mistral uses a larger FFN (3.5x hidden_size) than LLaMA (2.7x), trading some of the memory savings from GQA for additional model capacity.

• max_position_embeddings (int, default=8192): The maximum context length. With sliding window attention, sequences can technically extend beyond this, but RoPE positional embeddings are optimized for this range.

• rope_theta (float, default=10000.0): The base frequency for rotary positional embeddings. Higher values extend the effective context length but may reduce precision for nearby positions.