Attention Mechanism (2015) › Limitations

Encoder-decoder attention computes a score for each source-target position pair. With source length $T$ and target length $T'$, this requires $T \times T'$ alignment scores. The cost is quadratic when both lengths grow together and becomes expensive for long sequences.

The score calculations for a given decoder step can be parallelized across source positions. The surrounding RNN decoder, however, still generated target states sequentially, and the growing score matrix increased computation and memory use. Later self-attention architectures removed recurrence but retained the all-pairs cost.

Attention mechanisms also struggled with certain types of linguistic phenomena. While they excelled at word-level alignments, they had difficulty handling phrase-level or syntactic-level correspondences. A complex source phrase might need to be translated as a single target word, or vice versa, and attention mechanisms sometimes failed to capture these multi-word correspondences effectively. The mechanism worked best when alignments were roughly one-to-one or one-to-many, but struggled with many-to-one or many-to-many alignments that required more complex coordination.

Another limitation was the lack of explicit modeling of attention history. The standard attention mechanism computed weights independently for each decoding step, without explicitly tracking which source positions had already been attended to. This could lead to problems like repetition, where the model would attend to the same source words multiple times, or omission, where important source words were never attended to. While the decoder's hidden state implicitly tracked some of this information, explicit coverage mechanisms would later be developed to address these issues.

The attention mechanism also required storing all encoder hidden states in memory throughout the decoding process. For long sequences, this memory requirement could become prohibitive, especially when processing batches of sequences in parallel. Unlike simpler encoder-decoder models that only needed the final encoder hidden state, attention-based models needed to maintain all intermediate states, increasing memory requirements linearly with sequence length.

Finally, while attention provided interpretable alignments, these alignments were not always linguistically meaningful. The model learned attention patterns that improved translation quality, but these patterns did not necessarily correspond to semantic or syntactic relationships in ways that linguists would recognize. Attention weights could be noisy or spread across multiple source positions when a single focused alignment would be more appropriate. This reflected the model's optimization for translation accuracy rather than linguistic interpretability.