BART Architecture › BART vs T5 Comparison

BART and T5 were developed around the same time and share the encoder-decoder architecture, but they differ in key ways that reflect fundamentally different design philosophies. T5 was designed around the principle of maximum unification: by reformulating every NLP task as text-to-text mapping and using a single uniform pretraining objective (span corruption), the T5 authors aimed to create a general-purpose framework applicable to any task without architectural modifications. BART was designed around a different principle: maximum fidelity to proven components. By directly inheriting BERT's encoder and GPT's decoder, the BART authors accepted greater architectural specificity in exchange for the ability to use the extensive knowledge already accumulated about those architectures' behavior and optimization. Understanding these differences helps you choose the right model and reveals the design space of encoder-decoder models.

The differences between BART and T5 are not arbitrary: each choice in BART can be traced to a corresponding choice in BERT or GPT, and each choice in T5 can be traced to the text-to-text unification principle. When you see that BART uses post-norm while T5 uses pre-norm, you are seeing the difference between BERT's original design and T5's architectural innovation for stability. When you see that BART uses BPE tokenization while T5 uses SentencePiece, you are seeing BART inherit GPT-2's tokenizer versus T5 developing its own. These differences accumulate into models with distinct strengths and weaknesses in practice.

The table below summarizes the key architectural distinctions between BART and T5:

Pre-norm versus post-norm affects training stability in important ways. Pre-norm normalizes the input to each sublayer before the sublayer's computation. This keeps the residual connection adds a well-scaled update to a normalized representation. Post-norm normalizes after adding the residual connection, which means the residual pathway can accumulate unnormalized updates across layers. Pre-norm tends to produce more stable gradients at initialization. This enables training of very deep models with fewer numerical difficulties. Post-norm models often achieve slightly better final performance when training is successful, because the normalization at the output of each sublayer constrains representations more tightly. The practical effect is that most modern large language models, from T5 onward, have adopted pre-norm because it makes training more predictable, especially at large scale.

BART's use of learned absolute position embeddings means it has a fixed maximum sequence length, typically 1024 tokens, while T5's relative position encoding theoretically generalizes to longer sequences. With learned absolute embeddings, each position from 1 to 1024 has a dedicated embedding vector that the model learns during training. A sequence of length 1025 would require a position-1025 embedding that does not exist. T5's relative position encoding computes compatibility between positions based on their relative distance rather than their absolute positions, and uses a small number of learned bias terms corresponding to distance buckets. This approach can extrapolate to distances not seen during training, though with some performance degradation. In practice, both models require additional techniques for handling very long contexts, as we covered in Part XVIII.

The vocabulary difference is also practically significant. BART inherits GPT-2's byte-pair encoding (BPE) tokenizer with a vocabulary of approximately 50,000 tokens, while T5 uses SentencePiece with a vocabulary of 32,000 tokens. BPE operates on bytes and can represent any Unicode text without unknown tokens, which is particularly valuable for handling code and other unusual inputs. SentencePiece provides language-agnostic subword tokenization and is somewhat more compact. These tokenization differences affect how each model handles out-of-vocabulary words, multilingual inputs, and special character sequences.

The most noticeable practical difference between BART and T5 is how input and output are structured during fine-tuning. This reflects the two models' different design philosophies about task generalization.

T5 uses a text-to-text format where every task is framed as mapping an input text to an output text. Tasks are specified through prefixes that tell the model what operation to perform:

This uniformity is elegant: the model architecture and loss function are identical for every task, and tasks can even be mixed during fine-tuning. The task prefix teaches the model to condition its behavior on the instruction, prefiguring the instruction-following capability that later became central to large language model development.

BART treats tasks more naturally by using its encoder-decoder separation. For sequence-to-sequence tasks like summarization, the input goes to the encoder and the output comes from the decoder. No task prefix is needed because the architecture itself implies the operation: encode the source, decode the target. For classification tasks, a special token's representation from the decoder is used for prediction, which is closer to how BERT handles classification with its [CLS] token. This means BART can handle classification without reformulating it as text generation, an efficiency advantage for discriminative tasks.

The practical implication is that T5's text-to-text format is more flexible for multi-task learning and instruction following, while BART's format is more natural for generation tasks and slightly more efficient for classification. Neither is universally superior: the choice depends on the application, available computational resources, and the specific mix of tasks you need to support.

The pretraining objectives of BART and T5 differ in both philosophy and practical effect. While we will cover BART's pretraining in detail in the next chapter, the high-level contrast is important for understanding why these models behave differently in practice.

T5 uses span corruption, replacing random spans of tokens with sentinel tokens (like <extra_id_0>, <extra_id_1>, etc.) and training the model to generate the missing spans in sequence. The key properties of span corruption are its computational efficiency: since the target sequence contains only the masked spans (not the entire input), each training example requires generating far fewer tokens than the input length. For a 512-token input where 15% of tokens are masked in spans of average length 3, the target is roughly 25 tokens, making each training step roughly 20 times more efficient in terms of decoding computation.

BART explores multiple noise functions and trains the model to reconstruct the full original document. The target sequence is the complete uncorrupted text, which is the same length as the input. This means each BART pretraining step requires generating significantly more tokens than the corresponding T5 step. However, this more demanding objective provides a stronger training signal for generation tasks: the model must fill in masked regions and reconstruct the complete coherent text. The noise functions BART explores, including token deletion (which forces the model to determine how many tokens are missing), text infilling (which forces it to generate multiple tokens in place of a single mask), and sentence permutation (which forces it to reorder scrambled sentences), each train different aspects of text reconstruction ability.

T5's span corruption is more computationally efficient because the target sequence is much shorter than the input. BART's document reconstruction means the target is the same length as the uncorrupted input, requiring more computation per training step but giving a richer learning signal for generation-heavy downstream tasks. For applications where generation quality is important, such as abstractive summarization or dialogue generation, BART's more demanding pretraining objective tends to pay off at fine-tuning time.

In benchmarks, both models show strong performance with different strengths that reflect their pretraining objectives and architectural choices:

• Summarization: BART tends to perform better, particularly on abstractive summarization benchmarks like CNN/DailyMail and XSum. The reason is that BART's pretraining requires generating coherent, full-length text rather than filling in short spans, which directly exercises the skills needed for summarization.
• Translation: Performance is similar, with both models achieving strong results when fine-tuned on parallel data. For low-resource translation where fine-tuning data is scarce, the pretraining differences become more pronounced.
• Question answering: Both perform well, with specific results depending on the dataset and fine-tuning setup. For extractive QA, BERT-style encoder models remain competitive. For abstractive or open-domain QA, encoder-decoder models have an advantage.
• Classification: BART can perform classification by using the decoder's representation of a special token, though encoder-only models remain competitive for pure classification tasks where generation is not required.

The practical difference often comes down to implementation details and fine-tuning approach rather than architecture alone. For most practitioners, the availability of pretrained model weights, the quality of existing fine-tuned checkpoints for the target task, and compatibility with the application's requirements will matter more than the raw architectural differences.