Cross-Attention › Key Parameters

When implementing cross-attention in encoder-decoder transformers, several parameters control the mechanism's behavior and capacity. Choosing these parameters well is important for model quality and training efficiency.

• d_model: The model dimension, which is the size of input token representations. Both encoder outputs and decoder states should have this dimension. Common values range from 256 to 1024, with larger models using 2048 or more. The model dimension is the primary parameter controlling the overall expressiveness of the attention computation.

• d_k (query/key dimension): The dimension of the projected queries and keys. This controls the capacity of the attention scoring mechanism. Typically set to d_model // n_heads in multi-head attention, giving each head a portion of the full dimensionality. Smaller $d_k$ produces lower-variance attention scores even without the scaling factor, while larger $d_k$ gives more capacity but requires scaling more aggressively.

• d_v (value dimension): The dimension of the projected values. Often equal to $d_k$, but can differ. This determines the size of information transmitted when attention flows from encoder to decoder. In practice, $d_v = d_k = d_{\text{model}} / n_{\text{heads}}$ is the standard choice.

• n_heads: The number of parallel attention heads in multi-head cross-attention. More heads allow the model to attend to different aspects of the source sequence simultaneously. Typical values are 8, 12, or 16 heads. The original transformer used 8 heads with $d_{\text{model}} = 512$, giving $d_k = d_v = 64$ per head.

• encoder_mask: A boolean mask indicating which encoder positions are valid (True) versus padding (False). Essential for batched processing of variable-length source sequences to prevent attending to meaningless padding tokens. Without this mask, shorter sequences in a batch would receive spurious attention from the padding positions.

• scale factor: The scaling factor $1/\sqrt{d_k}$ applied before softmax. This is automatically determined by $d_k$ and prevents attention scores from becoming too large in high dimensions, which would cause softmax to saturate and gradients to vanish. This simple fix has a large practical impact on training stability.