BART Architecture › Code Implementation
Language AI HandbookAll chapters BART Architecture › Code ImplementationOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's explore BART's architecture using the Hugging Face Transformers library. We'll examine the model structure, inspect attention patterns, and see how the encoder and decoder interact. The goal is to verify the architectural details we've discussed and develop intuition for how BART behaves in practice.
from transformers import BartConfig, BartModel, BartTokenizer
# Load BART-base model and tokenizer with attention output enabled in config
tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
config = BartConfig.from_pretrained(
"facebook/bart-base", output_attentions=True
)
model = BartModel.from_pretrained("facebook/bart-base", config=config)
model.eval()First, let's examine the model configuration to confirm the architectural details we discussed:
# Store configuration for inspection
model_config = model.configThe configuration confirms our discussion: 6 encoder and decoder layers, 768-dimensional hidden states, 12 attention heads, and GeLU activation. The maximum position embeddings of 1024 reflects BART's use of learned absolute position encoding, matching the architectural specifications from the BART paper.
Now let's pass an example through the model and examine the outputs:
import torch
# Prepare input
input_text = "BART is a denoising autoencoder for pretraining sequence-to-sequence models."
inputs = tokenizer(input_text, return_tensors="pt")
# Create decoder input (shifted right, starting with BOS token)
decoder_input_ids = torch.tensor([[tokenizer.bos_token_id]])# Forward pass through the full model
with torch.no_grad():
outputs = model(
input_ids=inputs["input_ids"],
attention_mask=inputs["attention_mask"],
decoder_input_ids=decoder_input_ids,
output_attentions=True,
)The output shapes reveal the flow of information. The encoder produces hidden states for each input token, while the decoder produces hidden states for each position in the output sequence (currently just one, for the BOS token). The number of attention outputs equals the number of layers (6 for BART-base), one set per layer.
Let's visualize the attention patterns from the last layer of each component:
# Extract attention weights (last layer, first head)
encoder_attn = (
outputs.encoder_attentions[-1][0, 0].detach().numpy()
) # [seq_len, seq_len]
decoder_self_attn = (
outputs.decoder_attentions[-1][0, 0].detach().numpy()
) # [1, 1]
cross_attn = outputs.cross_attentions[-1][0, 0].detach().numpy() # [1, seq_len]
# Get tokens for labeling
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
The encoder attention pattern shows the bidirectional nature of BART's encoder. Each token can attend to all other tokens, with the model learning to focus on contextually relevant positions. Notice that attention is not uniform: the model concentrates attention on positions that provide useful contextual information, such as subword tokens that together form a single word attending strongly to each other.
Now let's examine cross-attention by generating a longer output sequence:
import torch
from transformers import BartConfig, BartForConditionalGeneration
# Load model with config that enables attention outputs
config_gen = BartConfig.from_pretrained(
"facebook/bart-base", output_attentions=True
)
model_gen = BartForConditionalGeneration.from_pretrained(
"facebook/bart-base", config=config_gen
)
model_gen.eval()
# We'll manually decode a few steps to capture attention
decoder_ids = [tokenizer.bos_token_id]
cross_attentions_per_step = []
with torch.no_grad():
for step in range(5):
decoder_input = torch.tensor([decoder_ids])
gen_outputs = model_gen(
input_ids=inputs["input_ids"],
attention_mask=inputs["attention_mask"],
decoder_input_ids=decoder_input,
output_attentions=True,
)
# Get cross-attention from last layer, first head
cross_attn_step = (
gen_outputs.cross_attentions[-1][0, 0, -1, :].detach().numpy()
)
cross_attentions_per_step.append(cross_attn_step)
# Greedy decoding for next token
next_token = gen_outputs.logits[0, -1, :].argmax().item()
decoder_ids.append(next_token)
# Stack cross-attentions
import numpy as np
cross_attn_matrix = np.stack(cross_attentions_per_step)
generated_tokens = tokenizer.convert_ids_to_tokens(decoder_ids[1:]) # Skip BOS
The cross-attention visualization reveals how the decoder grounds its generation in the input. Each row shows which encoder positions a generated token attended to. This shows the dynamic alignment between input and output. Tokens like "BART" generated first focus attention on the corresponding "BART" in the input, while later tokens like "sequence" attend to the "sequence" span in the encoder input. This alignment is not hardcoded: it emerges from the model learning through training which input positions are relevant for generating each output position.
Let's also count parameters to verify the model sizes we discussed:
def count_parameters(model):
"""Count trainable parameters in model components."""
encoder_params = sum(p.numel() for p in model.model.encoder.parameters())
decoder_params = sum(p.numel() for p in model.model.decoder.parameters())
embed_params = sum(p.numel() for p in model.model.shared.parameters())
lm_head_params = sum(p.numel() for p in model.lm_head.parameters())
total_params = sum(p.numel() for p in model.parameters())
return (
encoder_params,
decoder_params,
embed_params,
lm_head_params,
total_params,
)The parameter count confirms that BART-base has approximately 140 million parameters, split roughly evenly between the encoder and decoder with significant additional parameters in the embedding matrices. This aligns with our earlier discussion of BART model sizes and demonstrates how the encoder-decoder architecture distributes capacity across both components.
