BART Architecture: Encoder-Decoder Design for NLP

BART (Bidirectional and Auto-Regressive Transformers) represents one of the most thoughtfully designed models to emerge from the 2019 generation of large pretrained language models. When Facebook AI Research introduced BART, the NLP field was defined by a clear division: encoder-only models like BERT excelled at understanding tasks, applying bidirectional attention to build rich contextual representations, while decoder-only models like GPT dominated text generation, applying causal attention to produce fluent, coherent continuations. Both families had demonstrated remarkable capabilities, but they had also exposed a basic tension. The architectures optimized for understanding were poorly suited to generation, and those optimized for generation could not exploit full bidirectional context. BART asked a simple but powerful question: what if we designed a model that inherently did both?

The answer was to take the encoder-decoder framework, which had been introduced for machine translation in the original "Attention Is All You Need" transformer paper, and adapt it specifically for the pretraining paradigm. Rather than treating the encoder and decoder as novel components designed from scratch, BART's designers made an elegant choice: make the encoder as close to BERT as possible, and make the decoder as close to GPT as possible. Then connect them through cross-attention, and pretrain the resulting model as a denoising autoencoder, teaching it to reconstruct original text from arbitrarily corrupted versions. This combination of established components with a flexible pretraining objective produced a model that achieved state-of-the-art performance on summarization benchmarks and opened new research directions in sequence-to-sequence learning.

What makes BART interesting is its performance and the clarity of its design philosophy. Every architectural choice can be traced back to a principled decision: inherit from BERT for understanding, inherit from GPT for generation, and let the pretraining objective unify them. This transparency makes BART an excellent model for understanding encoder-decoder architectures more broadly. By studying what BART does and why, you gain insight into the design space that all sequence-to-sequence models inhabit. The choices BART makes, which positions it adopts from its predecessors and which it modifies, illuminate the tradeoffs that any practitioner working with language models must eventually consider.

Where T5, which we covered in previous chapters, takes an encoder-decoder approach with a specific span corruption objective, BART uses a more flexible and philosophically distinct approach. T5 reformulates every NLP task as text-to-text mapping and uses a uniform span corruption objective across all tasks. BART, by contrast, directly inherits BERT's bidirectional encoder and GPT's autoregressive decoder, and explores a much richer space of pretraining noise functions, including token masking, deletion, infilling, sentence permutation, and document rotation. These differences matter in practice, leading to distinct strengths for different applications, and understanding them builds your ability to choose the right architecture for a given problem.

This chapter examines BART's architecture in detail: how its encoder and decoder are structured, how the three attention mechanisms work and why each is configured as it is, how BART compares to T5 in design and performance, and what its limitations are. The next chapter covers BART's pretraining objectives in depth. Together, these two chapters complete your understanding of one of the foundational encoder-decoder models in modern NLP.

BART follows the encoder-decoder framework we discussed in Part XII and Part XVI, but its design philosophy draws explicitly from BERT and GPT. To appreciate this design choice, consider the basic trade-off in language modeling: understanding requires seeing the full context (including what comes before and after a word), while generation must proceed sequentially since you cannot use words you have not yet produced. These two requirements seem contradictory, yet both are needed for tasks like summarization where you must deeply understand a document before creating a coherent condensed version. A model that processes input bidirectionally can build richer representations, but it cannot generate text one token at a time, because generation requires a strictly left-to-right discipline. A model that processes input left-to-right can generate text naturally, but it builds weaker representations because each token can only incorporate information from tokens to its left.

BART resolves this tension through architectural separation. The encoder is essentially a BERT-style transformer, using bidirectional self-attention over the input sequence and letting each token to attend to all other tokens. This bidirectional view means that when the encoder processes the word "bank" in a sentence, it can simultaneously consider both the preceding context ("walked along the") and the following context ("of the river") to determine that we are discussing a riverbank rather than a financial institution. The decoder, in contrast, is essentially a GPT-style transformer, using causal (left-to-right) self-attention to ensure the model can only use previously generated tokens when predicting the next one. This constraint is not a limitation but a necessity: during generation, future tokens simply do not exist yet, and a model trained to exploit them would fail at inference time.

Think of BART's architecture as two specialist consultants working in sequence. The encoder is a research analyst who reads the entire input document with complete access to all its parts, building a complete, context-aware understanding of the content. The decoder is a writer who can consult the analyst's notes (through cross-attention) but must produce words one at a time without knowing what will come next in the output. The writer's sequential discipline ensures that what gets produced is valid text, and the analyst's complete view ensures that what the writer produces is grounded in the input. The key insight is that separating these two roles into distinct architectural components allows each to be optimized for its particular function, avoiding the compromises that would arise from forcing a single architecture to serve both purposes simultaneously.

The architecture can be summarized as:

This equation describes the architecture directly. The BART authors explicitly designed the encoder to match BERT's architecture and the decoder to match GPT's, then connected them with cross-attention. The "+" here represents architectural composition: the encoder and decoder are separate components linked through a cross-attention mechanism that allows the decoder to query the encoder's representations as it generates each token.

This design means BART inherits the bidirectional contextual understanding that made BERT successful for classification and extraction tasks, while also gaining the autoregressive generation capabilities that made GPT successful for text generation. The result is a model that can first build a rich, context-aware representation of the input, then use that representation to generate fluent, coherent output. This combination is precisely what tasks such as summarization and translation require. It also supports question-answering.

The BART encoder processes the input sequence using standard transformer encoder blocks, transforming raw token embeddings into contextualized representations. Think of each encoder layer as a refinement pass: the first layer turns bare token embeddings into slightly contextualized representations, the second layer enriches those with broader context, and by the final layer each token's representation encodes a sophisticated understanding of its role within the full input sequence. The encoder performs this transformation with complete access to all input positions at every layer, a capability that distinguishes it from the decoder and that makes it capable of resolving complex ambiguities that require long-range context.

These representations capture the meaning of each token within its full surrounding context. Each block contains:

1. Multi-head self-attention with bidirectional (non-causal) masking. This is the mechanism that allows each token to "see" every other token in the input, gathering information from both directions to build context-aware representations. With multiple attention heads, the model can simultaneously attend to different aspects of the context: one head might focus on syntactic dependencies, another on semantic similarity, and another on positional proximity.

2. Feed-forward network with GeLU activation. After attention aggregates information across positions, this two-layer neural network transforms each position's representation independently, letting the model to compute complex non-linear functions of the attended information. The feed-forward network is where much of the model's factual knowledge is believed to be stored, based on analysis of what information can be retrieved from its weights.

3. Residual connections around both sublayers. These skip connections add the input of each sublayer to its output, creating direct gradient pathways that facilitate training deep networks and letting the model to learn incremental refinements rather than complete transformations. Without residual connections, gradients in deep networks tend to vanish or explode, making training unstable.

4. Layer normalization applied after each sublayer (post-norm). This normalizes the activations to have zero mean and unit variance, stabilizing training by preventing the hidden representations from growing too large or too small as they pass through many layers.

The encoder produces a sequence of hidden states, one for each input token. These states capture rich bidirectional context, as each encoder representation can incorporate information from the entire input sequence. When you pass a sentence through BART's encoder, the representation for each word includes its meaning within the sentence context. This is the foundation upon which the decoder builds its generation: rather than generating from a blank slate, the decoder starts with a complete, contextually-rich picture of the input.

The decoder generates output tokens autoregressively, one at a time, using the encoder's representations to understand the input. This process mirrors how a human might write a summary: first, you read and understand the source document (the encoder), then you compose the summary word by word while referring back to the original (the decoder with cross-attention). Each decoder block contains:

1. Causal self-attention over previously generated tokens. This allows each position to attend only to earlier positions in the output sequence, building up a representation of what has been generated so far. The causal constraint ensures the model cannot "cheat" by looking at tokens it has not yet produced. During training, the model processes the entire target sequence at once, but the mask enforces the discipline of sequential generation: each position can only see positions to its left.

2. Cross-attention over encoder hidden states. This is the bridge between understanding and generation. It allows the decoder to query the encoder's representations, focusing on different parts of the input as needed for generating each output token. Without cross-attention, the decoder would be a purely unconditional language model with no grounding in the input. With cross-attention, the decoder can dynamically focus on whatever parts of the source are most relevant to the word it is currently generating.

3. Feed-forward network with GeLU activation. Just as in the encoder, this transforms the combined self-attention and cross-attention information through a non-linear function, computing complex features of the attended representations.

4. Residual connections around all three sublayers. With three sublayers instead of two, the decoder has even more opportunity to benefit from these direct pathways that preserve information and facilitate gradient flow.

5. Layer normalization after each sublayer (post-norm). This maintains training stability across the decoder's deeper structure.

The causal masking in self-attention prevents the decoder from "cheating" by looking at future tokens during training. When training on a target sequence like "The study found significant results," the decoder predicting "found" can see "The" and "study" but not "significant" or "results." This ensures that the learned conditional distribution $P(\text{token}_t \mid \text{token}_1, \ldots, \text{token}_{t-1})$ is trained with exactly the same information available during inference. Cross-attention allows each decoder position to attend to all encoder positions, letting the decoder to ground its generation in the input context. When generating "found," the decoder can look back at the relevant parts of the source document to decide what verb is appropriate.

BART makes several design decisions that distinguish it from other encoder-decoder models. These choices reflect its heritage from BERT and GPT and affect training dynamics and model behavior in ways that practitioners should understand.

GeLU activation is the first key choice. Unlike T5, which uses ReLU in its original form, BART follows BERT and GPT-2 in using the GeLU (Gaussian Error Linear Unit) activation function in feed-forward layers. The GeLU function is defined as $\text{GeLU}(x) = x \cdot \Phi(x)$, where $\Phi(x)$ is the standard Gaussian cumulative distribution function. In practice, a common approximation is used:

where:

• $x$: the pre-activation value at a given neuron
• $\Phi(x)$: the Gaussian CDF, which determines what fraction of a Gaussian distribution lies below $x$
• The product $x \cdot \Phi(x)$: scales the input by its probability under a standard normal, smoothly transitioning between near-zero for negative inputs and near-linear for positive inputs

GeLU provides smoother gradients than ReLU because it does not have a sharp transition at zero. Instead, it smoothly interpolates between passing and blocking signals. This smoothness can lead to more stable optimization and has become the preferred activation for many modern language models. Why does this formula make sense? Notice that for very negative $x$, $\Phi(x) \approx 0$, so GeLU outputs near zero, similar to ReLU. For very positive $x$, $\Phi(x) \approx 1$, so GeLU is approximately linear, also like ReLU. The difference is in the transition region near $x = 0$, where GeLU changes smoothly rather than abruptly.

Post-layer normalization is the second key choice. BART applies layer normalization after residual connections (known as post-norm), following the original transformer design. T5 uses pre-norm, applying layer normalization before each sublayer. The placement of normalization affects gradient flow: pre-norm tends to produce more stable gradients at initialization, which makes it easier to train very deep models. Post-norm can achieve slightly better final performance when training succeeds, but requires more careful tuning of learning rate schedules. This trade-off explains why both conventions persist in practice.

Learned positional embeddings are the third key choice. BART uses learned absolute position embeddings, similar to BERT and GPT-2, rather than the relative position encodings used in T5. With learned embeddings, the model maintains a separate embedding vector for each position (1, 2, 3, and so on up to some maximum), and these vectors are learned during training just like token embeddings. This approach is simple and effective but creates a hard limit on sequence length: the model has no embedding for position 1025 if it was trained with a maximum of 1024.

No parameter sharing is the fourth key choice. Unlike T5, BART does not tie encoder and decoder embeddings by default. The encoder and decoder maintain separate embedding matrices. This increases parameter count but allows the encoder and decoder to learn specialized representations suited to their different roles. The encoder's embeddings are tuned for building contextual representations of input text, while the decoder's embeddings are tuned for the language modeling objective of predicting the next token. These roles are related but distinct, and separate embedding matrices give the model the freedom to specialize each.

BART's attention patterns show how information flows through the model. Attention is the mechanism by which transformers route information between positions, and the pattern of allowed attention, meaning which positions can attend to which other positions, fundamentally shapes what the model can learn and compute. There are three distinct attention mechanisms in BART, each with its own configuration and purpose: bidirectional self-attention in the encoder, causal self-attention in the decoder, and cross-attention connecting the decoder to the encoder. Understanding these three patterns and why each is configured the way it is gives you a deep understanding of how information moves through the model.

Think of the attention mechanisms as a communication network within the model. Bidirectional encoder attention is like a town hall meeting where everyone can hear everyone else simultaneously. This allows rich information exchange. Causal decoder attention is like a relay race where each runner can only receive information from those who ran before. This keeps valid sequential generation. Cross-attention is like a telephone line from the decoder to the encoder. This allows the decoder to query the encoder's representations at any moment during generation. Together, these three communication patterns implement the full pipeline from input understanding to output generation.

In the encoder, every token can attend to every other token. This bidirectional attention gives the encoder the power to build contextual representations, because a word's meaning often depends on context that appears both before and after it. Consider disambiguating "The bank was eroding" versus "The bank was closing": you need both the subject and the verb to understand which sense of "bank" is intended. With bidirectional attention, the token "bank" can simultaneously look at "eroding" or "closing" to resolve this ambiguity. In a unidirectional model, "bank" would have to represent its meaning before seeing the disambiguating word that follows it.

To compute bidirectional self-attention, we construct a mask that permits all token pairs. For an input sequence of length $n$, the attention mask is an $n \times n$ matrix of ones:

where:

• $\mathbf{M}_{\text{encoder}}$: the attention mask matrix of shape $n \times n$
• $n$: the length of the input sequence
• Each entry of 1 indicates that attention is permitted between that query-key pair; a 0 would block that pair

Reading this matrix, row $i$ describes which positions token $i$ can attend to: a 1 in column $j$ means token $i$ can attend to token $j$. Since every entry is 1, every token can attend to every other token, including itself. This complete connectivity allows information to flow freely through the sequence, letting the encoder to build representations that incorporate arbitrarily distant context.

Why does this formula make sense? Notice that the matrix is symmetric: if token $i$ can attend to token $j$, then token $j$ can also attend to token $i$. This symmetry reflects the bidirectional nature of the encoder: attention flows in both directions simultaneously. The uniformity of the matrix, with every entry equal to 1, reflects the fact that there are no structural constraints on encoder attention. Any token might be relevant to any other token, and the model learns through training which connections are useful.

This bidirectional attention is what gives encoder-only models like BERT their power for understanding tasks. Each token's representation is informed by the complete context, not just preceding tokens. A word at the beginning of a sentence can be influenced by words at the end, and vice versa, letting the rich contextual representations that make BERT effective for tasks like sentiment analysis and named entity recognition. BART inherits this power in its encoder, giving it the same contextual understanding capability that made BERT so successful.

The decoder uses causal (or autoregressive) masking in its self-attention layers. The word "causal" refers to the structure of language generation: each token is caused by (depends on) the tokens that came before it, not those that come after. This constraint is not artificial. During generation, future tokens do not yet exist. A model that attended to future tokens during training would be learning a task that is fundamentally different from the task it faces at inference time, where future tokens are simply unavailable.

To enforce this constraint, we use a lower-triangular mask. For a sequence of length $m$, the attention mask is:

where:

• $\mathbf{M}_{\text{decoder}}$: the causal attention mask matrix of shape $m \times m$
• $m$: the length of the decoder sequence (the number of tokens generated so far)
• Entry $(i, j) = 1$ if $j \leq i$, meaning position $i$ can attend to position $j$
• Entry $(i, j) = 0$ if $j > i$, meaning position $i$ cannot attend to the future position $j$

The lower-triangular structure emerges directly from the causality constraint. Position 1 can only attend to position 1 (itself), so the first row has a single 1. Position 2 can attend to positions 1 and 2, giving two 1s. Position $i$ can attend to all positions from 1 through $i$, creating the triangular pattern. The zeros above the diagonal represent the blocked attention to future positions, which is information that the model must not use because it will not be available at generation time.

Why does this formula make sense? Consider the diagonal of the mask: every position can attend to itself, which makes sense because a token's representation should at minimum encode information about the token itself. The lower triangle means that as we move to later positions in the sequence, more context becomes available: position 1 has only itself, position 2 has itself and position 1, and so on. This is precisely the structure of autoregressive generation, where each new token is produced with access to all previously generated tokens.

This ensures that when generating token $t$, the model can only attend to tokens at positions $1, 2, \ldots, t$. This causal structure makes autoregressive generation possible. During training, we can compute attention for all positions in parallel (the mask handles the constraints), but the model learns to predict each token using only its predecessors, exactly as it will during generation. This training efficiency, combined with the discipline of causal masking, is one of the key innovations of transformer-based language models over earlier recurrent architectures.

Cross-attention connects the decoder to the encoder, bridging the understanding and generation phases. Without cross-attention, the decoder would not know what input to generate output for. It would become an unconditional language model, generating plausible text without grounding in any specific input. With cross-attention, the decoder can dynamically consult the encoder's representations, focusing on whatever parts of the input are most relevant for generating the current output token.

The mechanism of cross-attention is identical to standard scaled dot-product attention, but with a important difference: the queries come from the decoder, while the keys and values come from the encoder. This asymmetry is meaningful. The decoder is asking questions (represented as queries) about the input, and the encoder provides the answers (as values) along with ways to determine relevance (through keys). For each decoder position, queries come from the decoder hidden states while keys and values come from the encoder hidden states. There is no masking in cross-attention: every decoder position can attend to every encoder position.

We compute cross-attention as:

where:

• $\mathbf{Q}_{\text{dec}}$: query vectors derived from decoder hidden states, shape $(m, d_k)$. These represent "what the decoder is looking for" at each position.
• $\mathbf{K}_{\text{enc}}$: key vectors derived from encoder hidden states, shape $(n, d_k)$. These represent "what information each encoder position offers" and are compared against queries.
• $\mathbf{V}_{\text{enc}}$: value vectors derived from encoder hidden states, shape $(n, d_v)$. These contain the actual information that will be retrieved and combined.
• $d_k$: the dimension of query and key vectors, used for scaling
• $d_v$: the dimension of value vectors
• $\sqrt{d_k}$: a scaling factor that prevents dot products from growing too large, which would push softmax into regions with vanishing gradients

The computation proceeds in three stages. First, the dot product $\mathbf{Q}_{\text{dec}} \mathbf{K}_{\text{enc}}^T$ computes a compatibility score between each decoder position and each encoder position, resulting in an $m \times n$ matrix of raw attention scores. Second, the softmax operation converts these scaled scores into attention weights that sum to 1 across encoder positions, creating a proper probability distribution over the input that indicates how much attention each output position pays to each input position. Third, these weights are used to compute a weighted combination of encoder values, where positions with higher attention weights contribute more to the final representation.

Why does this formula make sense? The dot product $\mathbf{Q}_{\text{dec}} \mathbf{K}_{\text{enc}}^T$ measures alignment between what the decoder is looking for (queries) and what each encoder position offers (keys). High scores mean the decoder's needs align well with what that encoder position provides. After scaling and softmax, these alignment scores become a probability distribution that the decoder uses to blend together the encoder's value vectors. The result is a context vector: a weighted mixture of all encoder outputs, where the weights reflect relevance to the current decoding step.

This lets the decoder focus on different parts of the input as it generates each output token. When summarizing a document, the decoder might focus on the introduction when generating the first sentence of the summary, then shift attention to specific details when describing particular findings, and attend to the conclusion when wrapping up. This dynamic alignment is what makes encoder-decoder models so effective for conditional generation tasks, a mechanism we explored thoroughly in our discussion of Bahdanau and Luong attention in Part XII.

To understand how information flows through BART, consider processing an input sequence and generating an output. The flow follows a clear two-phase structure that separates understanding from generation while connecting them through cross-attention. The encoder and decoder do not operate completely independently: the encoder runs first, creating a complete set of hidden states, and the decoder then uses those hidden states at every layer as it generates output tokens one by one.

The encoder phase processes the entire input once. Input tokens are transformed into embeddings, positional embeddings are added, and the combined representations are passed through $L$ encoder layers. At each layer, bidirectional self-attention allows information to flow freely between all positions. By the end of the encoder, each token's representation includes context from the entire input. This phase processes the entire input once, creating a fixed set of representations that the decoder will query during generation. The computational cost of this phase is $O(n^2 d)$ where $n$ is the sequence length and $d$ is the hidden dimension, dominated by the self-attention operation.

The decoder phase generates one token at a time. For each output token, the decoder performs a sequence of operations that integrate three sources of information:

• Causal self-attention integrates information from previously generated tokens, building a representation of the output generated so far. This allows the decoder to maintain coherence across the output sequence: later tokens can attend to earlier ones. This keeps the generated text flows logically.
• Cross-attention retrieves relevant information from the encoder, grounding the generation in the input. At each step, the decoder issues queries against all encoder hidden states and receives a weighted combination of their values as a context signal.
• The feed-forward network transforms the combined representation, computing complex functions of the attended information.
• The output projection produces a probability distribution over the vocabulary, from which the next token is selected.

This two-phase structure is efficient for tasks with long inputs and shorter outputs. The expensive encoder computation happens once, and the decoder reuses those representations for every generated token. If generating a 100-word summary from a 1000-word document, the encoder runs once over 1000 tokens, and the decoder runs 100 times, each time performing cross-attention over the 1000 encoder states. The encoder's computation is amortized across all generated tokens.

BART and T5 were developed around the same time and share the encoder-decoder architecture, but they differ in key ways that reflect fundamentally different design philosophies. T5 was designed around the principle of maximum unification: by reformulating every NLP task as text-to-text mapping and using a single uniform pretraining objective (span corruption), the T5 authors aimed to create a general-purpose framework applicable to any task without architectural modifications. BART was designed around a different principle: maximum fidelity to proven components. By directly inheriting BERT's encoder and GPT's decoder, the BART authors accepted greater architectural specificity in exchange for the ability to use the extensive knowledge already accumulated about those architectures' behavior and optimization. Understanding these differences helps you choose the right model and reveals the design space of encoder-decoder models.

The differences between BART and T5 are not arbitrary: each choice in BART can be traced to a corresponding choice in BERT or GPT, and each choice in T5 can be traced to the text-to-text unification principle. When you see that BART uses post-norm while T5 uses pre-norm, you are seeing the difference between BERT's original design and T5's architectural innovation for stability. When you see that BART uses BPE tokenization while T5 uses SentencePiece, you are seeing BART inherit GPT-2's tokenizer versus T5 developing its own. These differences accumulate into models with distinct strengths and weaknesses in practice.

The table below summarizes the key architectural distinctions between BART and T5:

Pre-norm versus post-norm affects training stability in important ways. Pre-norm normalizes the input to each sublayer before the sublayer's computation. This keeps the residual connection adds a well-scaled update to a normalized representation. Post-norm normalizes after adding the residual connection, which means the residual pathway can accumulate unnormalized updates across layers. Pre-norm tends to produce more stable gradients at initialization. This enables training of very deep models with fewer numerical difficulties. Post-norm models often achieve slightly better final performance when training is successful, because the normalization at the output of each sublayer constrains representations more tightly. The practical effect is that most modern large language models, from T5 onward, have adopted pre-norm because it makes training more predictable, especially at large scale.

BART's use of learned absolute position embeddings means it has a fixed maximum sequence length, typically 1024 tokens, while T5's relative position encoding theoretically generalizes to longer sequences. With learned absolute embeddings, each position from 1 to 1024 has a dedicated embedding vector that the model learns during training. A sequence of length 1025 would require a position-1025 embedding that does not exist. T5's relative position encoding computes compatibility between positions based on their relative distance rather than their absolute positions, and uses a small number of learned bias terms corresponding to distance buckets. This approach can extrapolate to distances not seen during training, though with some performance degradation. In practice, both models require additional techniques for handling very long contexts, as we covered in Part XVIII.

The vocabulary difference is also practically significant. BART inherits GPT-2's byte-pair encoding (BPE) tokenizer with a vocabulary of approximately 50,000 tokens, while T5 uses SentencePiece with a vocabulary of 32,000 tokens. BPE operates on bytes and can represent any Unicode text without unknown tokens, which is particularly valuable for handling code and other unusual inputs. SentencePiece provides language-agnostic subword tokenization and is somewhat more compact. These tokenization differences affect how each model handles out-of-vocabulary words, multilingual inputs, and special character sequences.

The most noticeable practical difference between BART and T5 is how input and output are structured during fine-tuning. This reflects the two models' different design philosophies about task generalization.

T5 uses a text-to-text format where every task is framed as mapping an input text to an output text. Tasks are specified through prefixes that tell the model what operation to perform:

This uniformity is elegant: the model architecture and loss function are identical for every task, and tasks can even be mixed during fine-tuning. The task prefix teaches the model to condition its behavior on the instruction, prefiguring the instruction-following capability that later became central to large language model development.

BART treats tasks more naturally by using its encoder-decoder separation. For sequence-to-sequence tasks like summarization, the input goes to the encoder and the output comes from the decoder. No task prefix is needed because the architecture itself implies the operation: encode the source, decode the target. For classification tasks, a special token's representation from the decoder is used for prediction, which is closer to how BERT handles classification with its [CLS] token. This means BART can handle classification without reformulating it as text generation, an efficiency advantage for discriminative tasks.

The practical implication is that T5's text-to-text format is more flexible for multi-task learning and instruction following, while BART's format is more natural for generation tasks and slightly more efficient for classification. Neither is universally superior: the choice depends on the application, available computational resources, and the specific mix of tasks you need to support.

The pretraining objectives of BART and T5 differ in both philosophy and practical effect. While we will cover BART's pretraining in detail in the next chapter, the high-level contrast is important for understanding why these models behave differently in practice.

T5 uses span corruption, replacing random spans of tokens with sentinel tokens (like <extra_id_0>, <extra_id_1>, etc.) and training the model to generate the missing spans in sequence. The key properties of span corruption are its computational efficiency: since the target sequence contains only the masked spans (not the entire input), each training example requires generating far fewer tokens than the input length. For a 512-token input where 15% of tokens are masked in spans of average length 3, the target is roughly 25 tokens, making each training step roughly 20 times more efficient in terms of decoding computation.

BART explores multiple noise functions and trains the model to reconstruct the full original document. The target sequence is the complete uncorrupted text, which is the same length as the input. This means each BART pretraining step requires generating significantly more tokens than the corresponding T5 step. However, this more demanding objective provides a stronger training signal for generation tasks: the model must fill in masked regions and reconstruct the complete coherent text. The noise functions BART explores, including token deletion (which forces the model to determine how many tokens are missing), text infilling (which forces it to generate multiple tokens in place of a single mask), and sentence permutation (which forces it to reorder scrambled sentences), each train different aspects of text reconstruction ability.

T5's span corruption is more computationally efficient because the target sequence is much shorter than the input. BART's document reconstruction means the target is the same length as the uncorrupted input, requiring more computation per training step but giving a richer learning signal for generation-heavy downstream tasks. For applications where generation quality is important, such as abstractive summarization or dialogue generation, BART's more demanding pretraining objective tends to pay off at fine-tuning time.

In benchmarks, both models show strong performance with different strengths that reflect their pretraining objectives and architectural choices:

• Summarization: BART tends to perform better, particularly on abstractive summarization benchmarks like CNN/DailyMail and XSum. The reason is that BART's pretraining requires generating coherent, full-length text rather than filling in short spans, which directly exercises the skills needed for summarization.
• Translation: Performance is similar, with both models achieving strong results when fine-tuned on parallel data. For low-resource translation where fine-tuning data is scarce, the pretraining differences become more pronounced.
• Question answering: Both perform well, with specific results depending on the dataset and fine-tuning setup. For extractive QA, BERT-style encoder models remain competitive. For abstractive or open-domain QA, encoder-decoder models have an advantage.
• Classification: BART can perform classification by using the decoder's representation of a special token, though encoder-only models remain competitive for pure classification tasks where generation is not required.

The practical difference often comes down to implementation details and fine-tuning approach rather than architecture alone. For most practitioners, the availability of pretrained model weights, the quality of existing fine-tuned checkpoints for the target task, and compatibility with the application's requirements will matter more than the raw architectural differences.

BART was released in two primary sizes, following the naming convention established by BERT. This two-tier approach was standard for 2019-era models: a "base" configuration suitable for research and resource-constrained applications, and a "large" configuration that maximizes performance at the cost of greater computational requirements. Understanding the differences between these sizes and how they compare to contemporary models helps you calibrate the computational demands of working with BART.

The design principle behind BART's sizing is that the base model should be large enough to demonstrate the architecture's capabilities convincingly, while remaining tractable for research experiments on single GPUs. The large model should push capabilities further while remaining feasible for fine-tuning on multi-GPU research servers. Both sizes use the same architectural blueprint, differing only in depth (number of layers) and width (hidden dimension size), a standard practice for scaling transformer models.

The base model provides a good balance between capability and computational requirements. Its configuration reflects the original BERT-base design, with 6 encoder layers and 6 decoder layers instead of BERT's 12 layers. This reduction is necessary because BART's decoder adds approximately the same number of parameters as the encoder, so a 12+12 layer BART would be much larger than BERT-base. By halving the depth to 6+6, BART-base achieves a total parameter count similar to BERT-base and GPT-2 Small, which makes it comparable in cost to those models.

• Encoder layers: 6
• Decoder layers: 6
• Hidden dimension: 768
• Attention heads: 12
• Feed-forward dimension: 3072
• Parameters: approximately 140 million

BART-base works well when computational resources are limited or fast inference is required. It can run on consumer GPUs and provides strong performance on tasks such as summarization and question answering. For development and experimentation, starting with BART-base is advisable: it converges faster during fine-tuning and allows you to iterate on hyperparameters before committing to the more expensive BART-large.

The large model increases capacity significantly by doubling both depth and width. Each of the 12 encoder layers and 12 decoder layers contains wider attention heads and larger feed-forward networks, compounding the capacity increase. The result is a model with nearly three times the parameters of BART-base.

• Encoder layers: 12
• Decoder layers: 12
• Hidden dimension: 1024
• Attention heads: 16
• Feed-forward dimension: 4096
• Parameters: approximately 400 million

BART-large performs better on most benchmarks, particularly for complex generation tasks requiring fine-grained understanding of long documents. It requires more memory and computation, typically requiring a GPU with at least 16GB of memory for fine-tuning. For production summarization systems where quality is important and computational cost is acceptable, BART-large is generally the preferred choice.

The hidden dimension increase from 768 to 1024 has a cascading effect on all components: the attention key, query, and value dimensions scale up, the feed-forward dimensions scale proportionally, and the position embeddings become larger. The number of attention heads increases from 12 to 16, and since the hidden dimension also grows, each individual attention head becomes slightly wider as well (64 dimensions per head in both base and large, since $1024 / 16 = 64$). This consistency in per-head dimension is intentional: keeping the per-head dimension constant ensures that the scaled dot-product attention mechanism has consistent properties across model sizes.

The following table contextualizes BART's sizes relative to related models:

BART's parameter count is somewhat lower than T5 for the same size label because T5 uses more layers. BART-large with 12+12 layers is closer to T5-base with 12+12 layers in terms of depth, though T5 uses parameter-efficient relative position encodings while BART has separate learned embeddings for each position. The naming conventions across model families are inconsistent: a "base" BART is comparable in depth to a "base" T5, but BART-large is not comparable to T5-large in terms of parameter count. When comparing models across families, examining layer counts and hidden dimensions directly is more informative than relying on size labels.

To make the architecture concrete, let's trace a short input through BART's encoder and decoder step by step. This numerical walkthrough illustrates exactly how each component turns representations and where the three attention mechanisms come into play.

Suppose we want to summarize the sentence "Scientists discovered a new planet." We want BART to generate the summary "New planet found." Let's trace the processing at a conceptual level, using simplified numbers to illustrate the key transformations.

Step 1: Tokenization. The tokenizer first splits the input into BPE subword tokens. For our sentence, the result might be: <s>, Scientists, discovered, a, new, planet, ., </s>. Call this sequence of 8 tokens $\mathbf{x} = [x_1, x_2, x_3, x_4, x_5, x_6, x_7, x_8]$. Similarly, the target output tokenizes as: <s>, New, planet, found, ., </s>, giving 6 tokens $\mathbf{y} = [y_1, y_2, y_3, y_4, y_5, y_6]$.

Step 2: Input embeddings. Each input token $x_i$ is looked up in BART's embedding table to produce an embedding vector $\mathbf{e}_i \in \mathbb{R}^{768}$ (for BART-base). Then the corresponding positional embedding $\mathbf{p}_i \in \mathbb{R}^{768}$ for position $i$ is added, giving the initial encoder input:

where:

• $\mathbf{h}_i^{(0)}$: the initial representation for token $i$, combining token identity and position
• $\mathbf{e}_i$: the token embedding for $x_i$, encoding the token's meaning
• $\mathbf{p}_i$: the positional embedding for position $i$, encoding the token's location in the sequence

After this step, we have 8 vectors, each of dimension 768, representing the combined token-and-position information for each input token. These are the inputs to the first encoder layer.

Step 3: Encoder processing. The 8 vectors pass through 6 encoder layers. In each layer, multi-head self-attention allows all 8 tokens to interact with each other simultaneously. Consider what happens to the token "planet" at position 6. In the first encoder layer, "planet" can already attend to "Scientists" and "discovered," picking up contextual signals that this is a scientific discovery context. In later layers, its representation becomes increasingly contextualized, incorporating information about the full sentence structure. After 6 layers, the encoder produces 8 hidden state vectors $\mathbf{h}_i^{(L)} \in \mathbb{R}^{768}$, one for each input token, where each vector encodes that token's meaning in the full sentence context.

Step 4: Decoder initialization. The decoder starts with the beginning-of-sequence token <s> as its first input. This is mapped to an embedding and positional embedding just as in the encoder, giving an initial decoder hidden state $\mathbf{d}_1^{(0)} \in \mathbb{R}^{768}$.

Step 5: Generating the first output token. The decoder processes $\mathbf{d}_1^{(0)}$ through 6 decoder layers. In each layer, three operations occur. First, causal self-attention: since we only have one decoder token so far, this simply returns the token's own representation (trivially causal with one token). Second, cross-attention: the decoder issues a query against all 8 encoder hidden states, computing compatibility scores for each. The decoder's query asks, in effect, "given that I'm about to generate the first word of the summary, which encoder states are most relevant?" The cross-attention mechanism computes a weighted combination of encoder values, likely focusing heavily on the "new planet" part of the input since that is the most content-bearing phrase. Third, the feed-forward network processes the combined representation. After all 6 layers, the decoder projects the final hidden state onto the vocabulary dimension (50,265 dimensions for BART's BPE vocabulary) and applies softmax to get a probability distribution. The highest-probability token is selected (assuming greedy decoding): "New."

Step 6: Generating subsequent tokens. The decoder now has two tokens: <s> and "New." It processes both through the decoder layers, with causal self-attention letting "New" to attend to <s> but not vice versa. Cross-attention again queries the encoder, but now from the perspective of generating the second word given that "New" was already generated. The model selects "planet" as the next token. This process continues, generating "found," ".", and finally </s>, at which point generation stops.

The key insight from this trace is that the encoder runs once and the decoder runs once per output token. For our 6-token output, the cross-attention mechanism queries the same 8 encoder hidden states 6 times, each time from a different perspective as the output sequence grows. The encoder's computation is reused across all generation steps, which is the basic efficiency advantage of the encoder-decoder architecture over pure decoder architectures for conditional generation.

Let's explore BART's architecture using the Hugging Face Transformers library. We'll examine the model structure, inspect attention patterns, and see how the encoder and decoder interact. The goal is to verify the architectural details we've discussed and develop intuition for how BART behaves in practice.

Understanding BART's configuration parameters is important for adapting the model to new tasks and for interpreting the behavior of different model variants. Each parameter controls a specific aspect of the architecture's capacity and behavior.

The key parameters for BART's architecture are:

• d_model: The hidden dimension size, 768 for base and 1024 for large. This is the basic width of the model: it determines the dimensionality of all token representations throughout the network. Larger values increase the model's representational capacity but proportionally increase computation and memory requirements across all layers.
• encoder_layers / decoder_layers: The number of transformer blocks in each component. For BART-base, both are 6; for BART-large, both are 12. More layers increase model capacity by letting more refinement passes over the representations, but also increase training time and inference latency proportionally to depth.
• encoder_attention_heads: The number of attention heads, 12 for base and 16 for large. Multiple heads allow the model to attend to different aspects of the input simultaneously. Each head operates on a slice of the hidden dimension ($d_k = d_{\text{model}} / \text{heads}$, so 64 dimensions per head in both base and large), letting the model to learn multiple distinct attention patterns.
• encoder_ffn_dim: The dimension of the feed-forward network's hidden layer, typically 4 times the hidden dimension (3072 for base, 4096 for large). This controls the capacity of the non-linear transformations within each block. A larger FFN dimension allows the model to represent more complex functions of the attended information.
• max_position_embeddings: The maximum sequence length the model can process, 1024 for BART. This is a hard architectural limit imposed by the learned absolute position embeddings. Attempting to process longer sequences requires extrapolation beyond the trained embedding vectors, which typically degrades performance.
• activation_function: The non-linearity used in feed-forward layers, GeLU for BART. This affects the local curvature of the loss function and the speed of training convergence.

These parameters interact in important ways. Doubling d_model roughly quadruples the computation in attention layers (due to the $O(n^2 d)$ complexity) and the computation in feed-forward layers (due to the $O(n d^2)$ complexity of the two linear projections). Doubling encoder_layers doubles the depth and thus the computation linearly. When scaling BART to larger sizes, both dimensions are typically increased together, leading to the super-linear scaling behavior that characterizes large language model training costs.

BART's architecture has several important limitations that practitioners should understand before deploying it or comparing it to more recent alternatives. These limitations are not defects so much as consequences of the design choices that gave BART its strengths in 2019, but which have been superseded by architectural advances in the years since.

The post-norm design that BART inherited from the original transformer makes training less stable at large scales compared to pre-norm architectures. When training BART-like models at the scale of hundreds of billions of parameters, the post-norm placement requires extremely careful tuning of learning rate warmup schedules and initialization strategies to prevent training instability. Pre-norm architectures, by contrast, are more forgiving: the normalization at the input of each sublayer ensures that residual connections add well-scaled increments regardless of depth. This practical advantage explains why T5 and virtually all subsequent large language models have adopted pre-norm. For BART at its original scales of 140M and 400M parameters, post-norm works adequately, but scaling further requires addressing this limitation.

The use of learned absolute position embeddings creates a hard context length limit of 1024 tokens. This was a reasonable constraint in 2019, when available compute made processing longer sequences prohibitive and most benchmark tasks used shorter inputs. But it has become a significant practical limitation as use cases requiring long-document understanding have grown. Summarizing a 10-page research paper, answering questions about a book chapter, or processing lengthy code files all require context lengths well beyond 1024 tokens. Extending BART to longer contexts requires either interpolating or extrapolating the position embeddings, which degrades performance in ways that are difficult to fully recover through fine-tuning. More modern architectures using rotary position embeddings (RoPE) or ALiBi can extrapolate to longer sequences with much less degradation, as we covered in Part XIV.

Computationally, BART's pretraining objective requires reconstructing the entire input document, making pretraining more expensive than T5's span corruption approach. For downstream applications, however, this difference disappears since both models fine-tune and generate output tokens autoregressively at the same cost. The pretraining cost difference matters primarily when training new BART-style models from scratch, a scenario that only large research labs or organizations with substantial compute budgets will encounter. For the large majority of practitioners, who use pretrained model weights, this limitation is invisible.

BART also lacks mechanisms for efficient long-sequence processing. Its quadratic attention complexity means processing 4,096 tokens requires 16 times more attention computation than processing 1,024 tokens. Modern efficient attention mechanisms such as FlashAttention, sparse attention, and linear attention variants have significantly reduced this computational bottleneck, but the original BART architecture predates these developments. Fine-tuned BART models used in production today often incorporate these efficiency improvements, but they were not part of the original design.

Despite these limitations, BART's impact on NLP research and practice has been substantial. It demonstrated that combining BERT-style encoding with GPT-style decoding produces a model that handles both understanding and generation more effectively than either family alone. The denoising pretraining framework established that text reconstruction under arbitrary corruption is a powerful general-purpose objective, opening research directions into what kinds of corruption are most beneficial and why. BART also provided the first clear evidence that encoder-decoder architectures could match or exceed encoder-only models on understanding-heavy tasks while substantially outperforming them on generation tasks, a finding that influenced the subsequent development of models like mBART, BART-large fine-tuned on summarization (Pegasus, ProphetNet), and the mT5 family.

BART also showed that encoder-decoder architectures work well for conditional generation tasks. While decoder-only models like GPT have since dominated many applications due to their simplicity and scalability, encoder-decoder models like BART remain competitive for tasks where the input and output differ structurally, such as document summarization, data-to-text generation, and machine translation. The key advantage of encoder-decoder models for these tasks is that the encoder can build a complete, contextualized representation of the entire input before generation begins, rather than treating the input tokens as just the first part of a unified sequence. For tasks where the input is semantically distinct from the output, this separation of understanding and generation continues to provide meaningful benefits.

BART combines a BERT-style bidirectional encoder with a GPT-style autoregressive decoder, creating a model designed for both understanding and generation. The design philosophy is explicit: inherit proven components rather than innovating architecturally, and let the pretraining objective (denoising reconstruction) unify the two inherited components into a coherent whole.

The three attention mechanisms in BART each serve a distinct role. Encoder bidirectional self-attention, represented by an all-ones mask matrix, allows every input token to incorporate context from every other token, building rich contextual representations. Decoder causal self-attention, represented by a lower-triangular mask, enforces the sequential discipline required for autoregressive generation. Cross-attention, with queries from the decoder and keys and values from the encoder, connects the understanding and generation phases, letting the decoder to dynamically focus on relevant parts of the input as it generates each output token.

Compared to T5, BART makes different architectural choices that reflect its design philosophy. GeLU activation instead of ReLU, post-norm instead of pre-norm, learned absolute positions instead of relative positions, and separate embedding matrices for encoder and decoder: these differences reflect BART's decision to directly combine BERT and GPT rather than developing novel architectural components. Each choice has practical implications for training stability, sequence length handling, and task-specific performance.

BART comes in base (140M parameters, 6+6 layers) and large (400M parameters, 12+12 layers) configurations. Both use the same architectural blueprint, with the large model giving greater representational capacity at higher computational cost. Compared to contemporary models, BART-large sits between BERT-large and T5-base in parameter count. This reflects the overhead of maintaining both a full encoder and a full decoder.

The limitations of BART, including post-norm instability at scale, fixed context length, and expensive pretraining, reflect the state of the field in 2019 and have been addressed in subsequent work. Its impact, establishing denoising encoder-decoder pretraining as a powerful framework for generation tasks, remains evident in the architectures and training objectives of models developed in its wake.

The next chapter explores BART's pretraining in detail, examining the various noising functions that teach the model to reconstruct corrupted text and how these objectives shape the model's capabilities for downstream tasks.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about BART's encoder-decoder architecture.