BART Architecture › BART Model Sizes

BART was released in two primary sizes, following the naming convention established by BERT. This two-tier approach was standard for 2019-era models: a "base" configuration suitable for research and resource-constrained applications, and a "large" configuration that maximizes performance at the cost of greater computational requirements. Understanding the differences between these sizes and how they compare to contemporary models helps you calibrate the computational demands of working with BART.

The design principle behind BART's sizing is that the base model should be large enough to demonstrate the architecture's capabilities convincingly, while remaining tractable for research experiments on single GPUs. The large model should push capabilities further while remaining feasible for fine-tuning on multi-GPU research servers. Both sizes use the same architectural blueprint, differing only in depth (number of layers) and width (hidden dimension size), a standard practice for scaling transformer models.

The base model provides a good balance between capability and computational requirements. Its configuration reflects the original BERT-base design, with 6 encoder layers and 6 decoder layers instead of BERT's 12 layers. This reduction is necessary because BART's decoder adds approximately the same number of parameters as the encoder, so a 12+12 layer BART would be much larger than BERT-base. By halving the depth to 6+6, BART-base achieves a total parameter count similar to BERT-base and GPT-2 Small, which makes it comparable in cost to those models.

• Encoder layers: 6
• Decoder layers: 6
• Hidden dimension: 768
• Attention heads: 12
• Feed-forward dimension: 3072
• Parameters: approximately 140 million

BART-base works well when computational resources are limited or fast inference is required. It can run on consumer GPUs and provides strong performance on tasks such as summarization and question answering. For development and experimentation, starting with BART-base is advisable: it converges faster during fine-tuning and allows you to iterate on hyperparameters before committing to the more expensive BART-large.

The large model increases capacity significantly by doubling both depth and width. Each of the 12 encoder layers and 12 decoder layers contains wider attention heads and larger feed-forward networks, compounding the capacity increase. The result is a model with nearly three times the parameters of BART-base.

• Encoder layers: 12
• Decoder layers: 12
• Hidden dimension: 1024
• Attention heads: 16
• Feed-forward dimension: 4096
• Parameters: approximately 400 million

BART-large performs better on most benchmarks, particularly for complex generation tasks requiring fine-grained understanding of long documents. It requires more memory and computation, typically requiring a GPU with at least 16GB of memory for fine-tuning. For production summarization systems where quality is important and computational cost is acceptable, BART-large is generally the preferred choice.

The hidden dimension increase from 768 to 1024 has a cascading effect on all components: the attention key, query, and value dimensions scale up, the feed-forward dimensions scale proportionally, and the position embeddings become larger. The number of attention heads increases from 12 to 16, and since the hidden dimension also grows, each individual attention head becomes slightly wider as well (64 dimensions per head in both base and large, since $1024 / 16 = 64$). This consistency in per-head dimension is intentional: keeping the per-head dimension constant ensures that the scaled dot-product attention mechanism has consistent properties across model sizes.

The following table contextualizes BART's sizes relative to related models:

BART's parameter count is somewhat lower than T5 for the same size label because T5 uses more layers. BART-large with 12+12 layers is closer to T5-base with 12+12 layers in terms of depth, though T5 uses parameter-efficient relative position encodings while BART has separate learned embeddings for each position. The naming conventions across model families are inconsistent: a "base" BART is comparable in depth to a "base" T5, but BART-large is not comparable to T5-large in terms of parameter count. When comparing models across families, examining layer counts and hidden dimensions directly is more informative than relying on size labels.