Encoder-Decoder Architecture › Limitations and Practical Considerations

Encoder-decoder models offer powerful capabilities for sequence transformation, but they come with trade-offs that affect when and how to use them.

The encoder and decoder are distinct modules that must be trained and deployed together. This doubles the parameter count compared to using just an encoder or decoder of the same size. For translation between N languages, you need either N(N-1) models (one per direction) or a multilingual model that handles all directions.

Cross-attention adds computational overhead. Each decoder layer computes attention over the full encoder output, adding $O(m \times n)$ operations per layer, where:

• $m$: the target sequence length (number of tokens generated so far)
• $n$: the source sequence length (number of encoder positions)
• $O(m \times n)$: the complexity of computing attention scores between all decoder-encoder position pairs

For long source sequences, this cross-attention cost can dominate computation, especially as the decoder generates many tokens.

Despite encoder-decoder models' strong performance on structured tasks, recent trends have favored decoder-only architectures at scale. GPT-4, Claude, and Llama are all decoder-only. Several factors drive this:

• Simplicity: One architecture handles both understanding and generation
• Scaling: Easier to scale one component than two
• Flexibility: The same model handles diverse tasks through prompting
• Training efficiency: Next-token prediction provides dense supervision

However, encoder-decoder models remain valuable for production systems where task structure is fixed. Translation services, summarization APIs, and document processing pipelines often use encoder-decoder models because the structured input-output relationship benefits from specialized processing.

The encoder-decoder architecture offers clear advantages in specific scenarios:

• Repeated access to input: When the decoder needs to reference the input multiple times during generation, encoding once is more efficient than reprocessing.
• Length mismatch: When output is much shorter than input (summarization) or much longer (elaboration), the asymmetric architecture is natural.
• Non-monotonic alignment: When output order differs from input order, cross-attention learns the complex alignment better than a decoder trying to handle both tasks.