KV Cache › Key Parameters

Understanding each parameter's role helps you reason about deployment decisions when configuring inference servers:

• d_model: The dimensionality of the model's hidden states. This appears directly in the memory formula and in the projection computation cost.
• num_heads: The number of attention heads. Along with head_dim, it determines how the model dimension is partitioned. More heads mean more granular attention patterns but the same total cache size (since $H \times d_h = d_{\text{model}}$).
• head_dim: The dimension of each attention head, equal to $d_{\text{model}} / \text{num\_heads}$. Smaller head dimensions mean more heads for the same model width.
• max_seq_len: The maximum sequence length the cache can store. This determines the upper bound on context length and the maximum cache memory for a given model.
• batch_size: The number of sequences processed simultaneously. Larger batches improve GPU utilization and throughput but multiply cache memory linearly.