Luong Attention › Key Parameters

When implementing Luong attention, several parameters significantly impact model behavior and performance:

• Attention method (method): Chooses between "dot", "general", or "concat" score functions. Dot product is fastest and parameter-free but requires matching encoder/decoder dimensions. General attention adds a learned projection matrix, enabling different dimensions and learned similarity. Concat attention is most expressive but slowest. Start with dot product for most applications.

• Hidden dimension (hidden_dim): The dimensionality of the decoder hidden state. Larger values (256-512) provide more representational capacity but increase computation. For attention, this determines the space in which similarity is computed. Values of 256-512 work well for most sequence-to-sequence tasks.

• Encoder dimension (encoder_dim): The dimensionality of encoder hidden states. Can differ from hidden_dim when using general or concat attention. For bidirectional encoders, this is typically $2 \times \text{hidden\_dim}$ since forward and backward states are concatenated.

• Local attention window (D): For local attention, controls the half-width of the attention window. Positions outside $[p_t - D, p_t + D]$ receive zero attention. Larger windows (D=10-20) provide more flexibility but increase computation. Smaller windows (D=2-5) work well when alignments are expected to be monotonic.

• Dropout (dropout): Applied to embeddings and the attentional hidden state before output projection. Values of 0.1-0.3 help prevent overfitting, especially important for attention weights which can become overly peaked during training.