KV Cache Memory › Key Parameters

The following parameters are essential for KV cache memory calculation:

• num_layers: Number of transformer layers in the model. Each layer maintains its own key and value cache, so cache memory scales linearly with the number of layers.
• num_kv_heads: Number of key-value attention heads, also written as $H_{\text{kv}}$. In standard Multi-Head Attention, this equals the number of query heads. Grouped Query Attention uses fewer KV heads to reduce memory.
• head_dim: Dimension of each attention head, also written as $D_h$. Typically 64, 96, or 128. Combined with number of heads, this determines the model dimension.
• batch_size: Number of sequences processed simultaneously. Each sequence maintains its own independent cache, so memory scales linearly with batch size.
• seq_len: Context length in tokens. Cache memory grows linearly as more tokens are processed and cached.
• dtype_bytes: Bytes per element based on data type. FP16/BF16 use 2 bytes, FP32 uses 4 bytes, INT8 uses 1 byte, INT4 uses 0.5 bytes.