BART Architecture › Limitations and Impact

BART's architecture has several important limitations that practitioners should understand before deploying it or comparing it to more recent alternatives. These limitations are not defects so much as consequences of the design choices that gave BART its strengths in 2019, but which have been superseded by architectural advances in the years since.

The post-norm design that BART inherited from the original transformer makes training less stable at large scales compared to pre-norm architectures. When training BART-like models at the scale of hundreds of billions of parameters, the post-norm placement requires extremely careful tuning of learning rate warmup schedules and initialization strategies to prevent training instability. Pre-norm architectures, by contrast, are more forgiving: the normalization at the input of each sublayer ensures that residual connections add well-scaled increments regardless of depth. This practical advantage explains why T5 and virtually all subsequent large language models have adopted pre-norm. For BART at its original scales of 140M and 400M parameters, post-norm works adequately, but scaling further requires addressing this limitation.

The use of learned absolute position embeddings creates a hard context length limit of 1024 tokens. This was a reasonable constraint in 2019, when available compute made processing longer sequences prohibitive and most benchmark tasks used shorter inputs. But it has become a significant practical limitation as use cases requiring long-document understanding have grown. Summarizing a 10-page research paper, answering questions about a book chapter, or processing lengthy code files all require context lengths well beyond 1024 tokens. Extending BART to longer contexts requires either interpolating or extrapolating the position embeddings, which degrades performance in ways that are difficult to fully recover through fine-tuning. More modern architectures using rotary position embeddings (RoPE) or ALiBi can extrapolate to longer sequences with much less degradation, as we covered in Part XIV.

Computationally, BART's pretraining objective requires reconstructing the entire input document, making pretraining more expensive than T5's span corruption approach. For downstream applications, however, this difference disappears since both models fine-tune and generate output tokens autoregressively at the same cost. The pretraining cost difference matters primarily when training new BART-style models from scratch, a scenario that only large research labs or organizations with substantial compute budgets will encounter. For the large majority of practitioners, who use pretrained model weights, this limitation is invisible.

BART also lacks mechanisms for efficient long-sequence processing. Its quadratic attention complexity means processing 4,096 tokens requires 16 times more attention computation than processing 1,024 tokens. Modern efficient attention mechanisms such as FlashAttention, sparse attention, and linear attention variants have significantly reduced this computational bottleneck, but the original BART architecture predates these developments. Fine-tuned BART models used in production today often incorporate these efficiency improvements, but they were not part of the original design.

Despite these limitations, BART's impact on NLP research and practice has been substantial. It demonstrated that combining BERT-style encoding with GPT-style decoding produces a model that handles both understanding and generation more effectively than either family alone. The denoising pretraining framework established that text reconstruction under arbitrary corruption is a powerful general-purpose objective, opening research directions into what kinds of corruption are most beneficial and why. BART also provided the first clear evidence that encoder-decoder architectures could match or exceed encoder-only models on understanding-heavy tasks while substantially outperforming them on generation tasks, a finding that influenced the subsequent development of models like mBART, BART-large fine-tuned on summarization (Pegasus, ProphetNet), and the mT5 family.

BART also showed that encoder-decoder architectures work well for conditional generation tasks. While decoder-only models like GPT have since dominated many applications due to their simplicity and scalability, encoder-decoder models like BART remain competitive for tasks where the input and output differ structurally, such as document summarization, data-to-text generation, and machine translation. The key advantage of encoder-decoder models for these tasks is that the encoder can build a complete, contextualized representation of the entire input before generation begins, rather than treating the input tokens as just the first part of a unified sequence. For tasks where the input is semantically distinct from the output, this separation of understanding and generation continues to provide meaningful benefits.