Transformer Architecture (2017) › Limitations

The Transformer architecture had several limitations that would shape subsequent research directions. Perhaps the most significant was the quadratic computational complexity of self-attention relative to sequence length. Computing attention over a sequence of length $n$ requires computing attention scores for all $n^2$ pairs of positions, which becomes computationally expensive for very long sequences. This quadratic complexity limited the practical sequence lengths that transformers could handle effectively, forcing researchers to truncate or segment long documents.

The memory requirements of storing attention matrices for long sequences also became problematic. For a sequence of length $n$, the attention mechanism requires storing an $n \times n$ matrix of attention scores, which grows quadratically with sequence length. For very long sequences, this could exhaust available GPU memory, limiting the practical applicability of transformers to long documents or contexts. This memory limitation would drive research into more memory-efficient attention mechanisms and sparse attention patterns.

The lack of built-in inductive biases about sequence structure was both a strength and a limitation. Unlike RNNs, which have an inherent bias toward processing sequences sequentially, transformers have no built-in assumptions about sequence order beyond what they learn from positional encodings. While this flexibility is powerful, it means that transformers must learn all sequence-related patterns from data, requiring more training data and computation than architectures with stronger inductive biases. For tasks with clear sequential structure, this lack of bias could be inefficient.

Training instability could also be an issue, particularly for deep transformer models. While residual connections and layer normalization help, very deep transformers could still suffer from vanishing or exploding gradients during training. This instability limited the depth of early transformer models and required careful initialization and learning rate scheduling. The problem became more pronounced as researchers scaled to larger models with more layers.

The fixed-size positional encodings used in the original Transformer could also be limiting. Sinusoidal encodings work well for sequences up to the maximum length seen during training, but their effectiveness can degrade for much longer sequences. While learned positional embeddings can adapt to the training data, they still don't generalize perfectly to sequences much longer than those in the training set. This limitation affects the model's ability to handle arbitrarily long sequences effectively.

Transformer models also require substantial computational resources for both training and inference. While training is more efficient than for sequential models due to parallelization, transformer models are still computationally expensive. Inference, while faster than training, still requires computing attention over the entire sequence, which can be slow for long sequences or when serving many requests simultaneously. The computational cost has driven research into model compression and distillation, along with more efficient attention mechanisms.