BART Architecture › Summary

BART combines a BERT-style bidirectional encoder with a GPT-style autoregressive decoder, creating a model designed for both understanding and generation. The design philosophy is explicit: inherit proven components rather than innovating architecturally, and let the pretraining objective (denoising reconstruction) unify the two inherited components into a coherent whole.

The three attention mechanisms in BART each serve a distinct role. Encoder bidirectional self-attention, represented by an all-ones mask matrix, allows every input token to incorporate context from every other token, building rich contextual representations. Decoder causal self-attention, represented by a lower-triangular mask, enforces the sequential discipline required for autoregressive generation. Cross-attention, with queries from the decoder and keys and values from the encoder, connects the understanding and generation phases, letting the decoder to dynamically focus on relevant parts of the input as it generates each output token.

Compared to T5, BART makes different architectural choices that reflect its design philosophy. GeLU activation instead of ReLU, post-norm instead of pre-norm, learned absolute positions instead of relative positions, and separate embedding matrices for encoder and decoder: these differences reflect BART's decision to directly combine BERT and GPT rather than developing novel architectural components. Each choice has practical implications for training stability, sequence length handling, and task-specific performance.

BART comes in base (140M parameters, 6+6 layers) and large (400M parameters, 12+12 layers) configurations. Both use the same architectural blueprint, with the large model giving greater representational capacity at higher computational cost. Compared to contemporary models, BART-large sits between BERT-large and T5-base in parameter count. This reflects the overhead of maintaining both a full encoder and a full decoder.

The limitations of BART, including post-norm instability at scale, fixed context length, and expensive pretraining, reflect the state of the field in 2019 and have been addressed in subsequent work. Its impact, establishing denoising encoder-decoder pretraining as a powerful framework for generation tasks, remains evident in the architectures and training objectives of models developed in its wake.

The next chapter explores BART's pretraining in detail, examining the various noising functions that teach the model to reconstruct corrupted text and how these objectives shape the model's capabilities for downstream tasks.