BERT Architecture › BERT Architecture

In October 2018, Google released a paper titled "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Within months, BERT had shattered performance records on eleven NLP benchmarks and fundamentally changed how the field approached language understanding tasks. The architecture that made this possible wasn't revolutionary in its components: it stacked transformer encoder blocks with multi-head self-attention and feed-forward networks. The difference lay in BERT's training regime and in the way it deployed these familiar pieces.

To understand why BERT mattered, consider the problem it solved. Before BERT, language models learned representations in a fundamentally asymmetric way. Autoregressive models like GPT read text from left to right, predicting each word based only on the words that came before. This means the model's representation of "bank" in "She went to the river bank" is built only from "She went to the river" with no knowledge of what comes after. For language generation, this makes sense: the next token is unknown. But for language understanding, it is a serious limitation. When a human reads that sentence, they use everything around "bank" to determine its meaning. They look left and right. BERT's central insight was to train a model that could do exactly that.

The technical mechanism BERT used to enable this bidirectional training is called Masked Language Modeling, or MLM. Rather than predicting the next token from all previous tokens, BERT randomly masks some fraction of the input tokens and asks the model to predict the masked values from the surrounding context in both directions. Think of it like a fill-in-the-blank exercise where you can see everything in the sentence except the blanked word. This change alters the attention architecture: because the model must predict masked tokens using both left and right context, it cannot rely on a causal attention mask. Every token attends to every other token. The entire attention mechanism becomes bidirectional.

The second pretraining task BERT introduced is Next Sentence Prediction, or NSP. Given two sentences A and B, the model must predict whether B follows A in the original text or was randomly selected. This task teaches the model to understand relationships between sentences, not just relationships within them. A model trained only on single-sentence tasks would struggle with question answering (where the question and context are separate sentences) or natural language inference (where premise and hypothesis must be compared). NSP gave BERT a mechanism to learn inter-sentence coherence.

These two pretraining objectives worked together to produce a model that, after fine-tuning on task-specific data, could achieve state-of-the-art performance across an unusually broad range of benchmarks simultaneously. The key insight is that language understanding tasks, despite their surface diversity, all require understanding relationships between words and sentences. BERT's bidirectional pretraining gave it exactly the right inductive bias to generalize across all of them.

This chapter examines the BERT architecture in detail. We'll explore the two model sizes (Base and Large) and their layer configurations, understand how BERT's three embedding types combine to represent input, examine the bidirectional attention patterns that distinguish BERT from autoregressive models, and analyze the output representations that downstream tasks use. By the end, you'll understand what BERT's architecture looks like and why each design choice matters.