Attention Intuition › Key Parameters

When implementing attention mechanisms, several parameters significantly impact model behavior:

• Hidden dimension (hidden_dim): The size of encoder and decoder hidden states. Larger values (256 to 1024) can represent finer distinctions but increase memory and computation. For dot product scoring, encoder and decoder dimensions must match.

• Attention dimension (d_a): For additive attention, this controls the size of the intermediate projection space. Typical values range from 64 to 512. Smaller values reduce parameters but may limit the model's ability to learn complex alignment patterns.

• Number of attention heads: In multi-head attention (covered in transformer chapters), this parameter controls how many parallel attention computations run simultaneously. Common values are 4, 8, or 16 heads, with the hidden dimension divided equally among heads.

• Dropout rate: Applied to attention weights during training to prevent the model from relying too heavily on specific positions. Values of 0.1 to 0.3 are typical. Higher dropout encourages more distributed attention patterns.

• Temperature scaling: An optional parameter that divides attention scores before softmax. Values below 1.0 sharpen the distribution (more focused attention), while values above 1.0 flatten it (more uniform attention). Scaled dot product attention uses $\sqrt{d_k}$ as an automatic temperature calibrated to the embedding dimension. When fine-tuning or probing attention behavior, manually adjusting temperature can be a useful diagnostic tool.

• Coverage mechanism: An optional extension that penalizes attention to positions that have already received high attention across previous decoder steps. Coverage was important for early machine translation models to avoid the "over-translation" problem, where certain source phrases received excessive attention and were incorrectly translated multiple times in the output. A coverage vector tracks the cumulative attention each encoder position has received, and a coverage penalty discourages the model from repeatedly attending to positions that have already contributed heavily.