BART Architecture › BART Architecture

BART (Bidirectional and Auto-Regressive Transformers) represents one of the most thoughtfully designed models to emerge from the 2019 generation of large pretrained language models. When Facebook AI Research introduced BART, the NLP field was defined by a clear division: encoder-only models like BERT excelled at understanding tasks, applying bidirectional attention to build rich contextual representations, while decoder-only models like GPT dominated text generation, applying causal attention to produce fluent, coherent continuations. Both families had demonstrated remarkable capabilities, but they had also exposed a basic tension. The architectures optimized for understanding were poorly suited to generation, and those optimized for generation could not exploit full bidirectional context. BART asked a simple but powerful question: what if we designed a model that inherently did both?

The answer was to take the encoder-decoder framework, which had been introduced for machine translation in the original "Attention Is All You Need" transformer paper, and adapt it specifically for the pretraining paradigm. Rather than treating the encoder and decoder as novel components designed from scratch, BART's designers made an elegant choice: make the encoder as close to BERT as possible, and make the decoder as close to GPT as possible. Then connect them through cross-attention, and pretrain the resulting model as a denoising autoencoder, teaching it to reconstruct original text from arbitrarily corrupted versions. This combination of established components with a flexible pretraining objective produced a model that achieved state-of-the-art performance on summarization benchmarks and opened new research directions in sequence-to-sequence learning.

What makes BART interesting is its performance and the clarity of its design philosophy. Every architectural choice can be traced back to a principled decision: inherit from BERT for understanding, inherit from GPT for generation, and let the pretraining objective unify them. This transparency makes BART an excellent model for understanding encoder-decoder architectures more broadly. By studying what BART does and why, you gain insight into the design space that all sequence-to-sequence models inhabit. The choices BART makes, which positions it adopts from its predecessors and which it modifies, illuminate the tradeoffs that any practitioner working with language models must eventually consider.

Where T5, which we covered in previous chapters, takes an encoder-decoder approach with a specific span corruption objective, BART uses a more flexible and philosophically distinct approach. T5 reformulates every NLP task as text-to-text mapping and uses a uniform span corruption objective across all tasks. BART, by contrast, directly inherits BERT's bidirectional encoder and GPT's autoregressive decoder, and explores a much richer space of pretraining noise functions, including token masking, deletion, infilling, sentence permutation, and document rotation. These differences matter in practice, leading to distinct strengths for different applications, and understanding them builds your ability to choose the right architecture for a given problem.

This chapter examines BART's architecture in detail: how its encoder and decoder are structured, how the three attention mechanisms work and why each is configured as it is, how BART compares to T5 in design and performance, and what its limitations are. The next chapter covers BART's pretraining objectives in depth. Together, these two chapters complete your understanding of one of the foundational encoder-decoder models in modern NLP.