BERT Architecture: Embeddings, Attention and Encoder Layers

In October 2018, Google released a paper titled "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Within months, BERT had shattered performance records on eleven NLP benchmarks and fundamentally changed how the field approached language understanding tasks. The architecture that made this possible wasn't revolutionary in its components: it stacked transformer encoder blocks with multi-head self-attention and feed-forward networks. The difference lay in BERT's training regime and in the way it deployed these familiar pieces.

To understand why BERT mattered, consider the problem it solved. Before BERT, language models learned representations in a fundamentally asymmetric way. Autoregressive models like GPT read text from left to right, predicting each word based only on the words that came before. This means the model's representation of "bank" in "She went to the river bank" is built only from "She went to the river" with no knowledge of what comes after. For language generation, this makes sense: the next token is unknown. But for language understanding, it is a serious limitation. When a human reads that sentence, they use everything around "bank" to determine its meaning. They look left and right. BERT's central insight was to train a model that could do exactly that.

The technical mechanism BERT used to enable this bidirectional training is called Masked Language Modeling, or MLM. Rather than predicting the next token from all previous tokens, BERT randomly masks some fraction of the input tokens and asks the model to predict the masked values from the surrounding context in both directions. Think of it like a fill-in-the-blank exercise where you can see everything in the sentence except the blanked word. This change alters the attention architecture: because the model must predict masked tokens using both left and right context, it cannot rely on a causal attention mask. Every token attends to every other token. The entire attention mechanism becomes bidirectional.

The second pretraining task BERT introduced is Next Sentence Prediction, or NSP. Given two sentences A and B, the model must predict whether B follows A in the original text or was randomly selected. This task teaches the model to understand relationships between sentences, not just relationships within them. A model trained only on single-sentence tasks would struggle with question answering (where the question and context are separate sentences) or natural language inference (where premise and hypothesis must be compared). NSP gave BERT a mechanism to learn inter-sentence coherence.

These two pretraining objectives worked together to produce a model that, after fine-tuning on task-specific data, could achieve state-of-the-art performance across an unusually broad range of benchmarks simultaneously. The key insight is that language understanding tasks, despite their surface diversity, all require understanding relationships between words and sentences. BERT's bidirectional pretraining gave it exactly the right inductive bias to generalize across all of them.

This chapter examines the BERT architecture in detail. We'll explore the two model sizes (Base and Large) and their layer configurations, understand how BERT's three embedding types combine to represent input, examine the bidirectional attention patterns that distinguish BERT from autoregressive models, and analyze the output representations that downstream tasks use. By the end, you'll understand what BERT's architecture looks like and why each design choice matters.

BERT comes in two standard configurations: BERT-Base and BERT-Large. These sizes were chosen deliberately to balance capability against practical deployment constraints. BERT-Base matches the hidden dimension of OpenAI's GPT (768), enabling direct comparisons, while BERT-Large tests whether larger models capture finer language patterns.

The choice to release two sizes reflects the practical reality that research and production have different requirements. BERT-Base was designed to be large enough to achieve strong performance while still being trainable and deployable on hardware available to most researchers and companies in 2018. BERT-Large was designed as a proof of concept: does more scale help? The answer was unambiguously yes. BERT-Large consistently outperformed BERT-Base by several points on most benchmarks, establishing a pattern that would define the field for years: more parameters and more layers generally produce better representations.

Notice that both models use a head dimension of exactly 64. This is not a coincidence. The head dimension determines the size of the QKV vectors within each attention head. At 64 dimensions, these vectors are large enough to express complex patterns but small enough that the dot products between query and key vectors remain numerically stable without aggressive scaling. The Vaswani et al. transformer paper argued that head dimensions should be in this range for stable training, and BERT adopted this guidance. When BERT-Large increases from 12 to 16 heads, it achieves greater representational capacity by running more parallel attention computations rather than by making each computation larger.

The architectural specifications for each variant are:

Notice that the head dimension remains constant at 64 across both variants. This means BERT-Large achieves more capacity by having more heads (16 vs 12) and more layers (24 vs 12), not by making each head larger. The feed-forward dimension follows the standard 4x multiplier relative to hidden size (768 × 4 = 3072 for Base, 1024 × 4 = 4096 for Large).

Let's compute the exact parameter counts to understand where capacity resides:

The vast majority of parameters reside in the transformer layers, particularly in the feed-forward networks. Token embeddings represent a significant portion due to the large vocabulary, but their contribution decreases proportionally as model depth increases. This distribution matters for understanding where BERT stores knowledge: factual information tends to concentrate in feed-forward weights, while attention patterns encode syntactic and semantic relationships.

In practice, the choice between BERT-Base and BERT-Large often comes down to deployment constraints rather than performance requirements. BERT-Large requires roughly 3x more memory and compute than BERT-Base. For applications where latency and cost are important, BERT-Base frequently offers 90-95% of BERT-Large's performance at one-third the cost. This tradeoff has motivated research into distilled versions like DistilBERT, which achieves about 97% of BERT-Base's performance at 40% of its size. The next chapter covers these variants in detail.

BERT's input representation is one of its most distinctive features. Unlike simpler models that use only token embeddings, BERT combines three embedding types to capture different aspects of the input. This design enables BERT to process sentence pairs for tasks like question answering and natural language inference.

Think of the three embedding types as answering three different questions about each token. The token embedding answers "what word is this?", storing the semantic and syntactic properties of the vocabulary item itself. The position embedding answers "where does this word appear?", encoding the sequential position within the input. The segment embedding answers "which sentence does this word belong to?", distinguishing sentence A from sentence B in paired inputs. All three answers are equally important, so BERT adds them together into a single dense vector rather than processing them separately.

The 30,522-token vocabulary is produced by WordPiece tokenization, the same subword tokenization scheme used in earlier Google systems. WordPiece splits rare or unknown words into smaller pieces. This ensures that even novel words can be represented as combinations of known subwords. A word like "embedding" might remain whole if it appears frequently in training, but "embeddings" might be split into "embedding" and "##s", where the "##" prefix indicates a continuation. This keeps the vocabulary manageable while still representing nearly any text.

Every input token receives three embeddings that are summed together:

1. Token embeddings: Standard learned embeddings that map each vocabulary token to a dense vector
2. Position embeddings: Learned embeddings for each position (0 to 511) encoding sequential order
3. Segment embeddings: Two learned embeddings (A and B) indicating which sentence a token belongs to

The segment embeddings deserve special attention. BERT was designed for tasks involving sentence pairs: given two sentences, determine if the second follows the first (next sentence prediction), if the first entails the second (natural language inference), or find the answer span (question answering). The segment embeddings allow the model to distinguish which tokens belong to which sentence even after they're concatenated.

The concatenation of two sentences into a single sequence with special tokens might seem like a strange design choice. Why not process each sentence independently and combine their representations afterward? The answer is that cross-sentence attention patterns are precisely what many tasks require. In question answering, the model needs to attend from words in the question to matching words in the context passage. In natural language inference, it needs to identify contradictions between specific words in the premise and hypothesis. By putting both sentences in the same context window and letting attention operate freely between them, BERT can learn these cross-sentence relationships directly. The segment embeddings ensure that the model can always tell which sentence each token came from, even while attending freely across both.

BERT uses several special tokens to structure its input:

• [CLS]: Prepended to every input. Its final representation is used for classification tasks
• [SEP]: Inserted between sentences and at the end of the input to mark boundaries
• [MASK]: Used during pretraining to indicate positions the model should predict
• [PAD]: Fills sequences shorter than the batch maximum length
• [UNK]: Represents tokens not in the vocabulary

Let's implement the embedding layer to see how these components combine:

The embedding layer maps our 8-token input into a tensor of shape (1, 8, 768), where each token now has a 768-dimensional representation that combines token identity with positional and segment information. This dense representation is what flows through the subsequent transformer layers.

The element-wise addition of three embedding types may seem unusual. Why not concatenate them? The answer lies in parameter efficiency and training dynamics. Addition keeps the hidden dimension fixed at 768, while concatenation would triple it. More subtly, addition forces the model to learn representations where token meaning, positional information, and segment identity can coexist in the same vector space. The layer normalization after addition rescales these combined representations to have consistent statistics.

In practice, this summation works because modern neural networks are remarkably good at learning linear decompositions within a shared vector space. During training, the model adjusts all three embedding tables simultaneously via gradient descent. The token embeddings learn to represent word meaning in a subspace that leaves room for positional and segment information to be added without collision. The layer normalization that follows stabilizes the combined signal by ensuring the activations entering the first transformer layer have zero mean and unit variance, which helps the subsequent attention computations behave consistently regardless of which specific tokens were summed.

BERT uses learned position embeddings rather than the sinusoidal encodings from the original transformer. Each of the 512 possible positions gets its own learned vector. This choice trades generalization for expressiveness: learned embeddings cannot extrapolate to positions beyond 512, but they can capture position-specific patterns that sinusoidal encodings cannot.

Recall from the transformer chapter that sinusoidal encodings use a fixed mathematical formula: position $p$ and dimension $i$ produce the value $\sin(p / 10000^{2i/d})$ for even dimensions and $\cos(p / 10000^{2i/d})$ for odd dimensions. This formula was designed so that the distance between any two position encodings depends only on their relative offset, not their absolute positions, which enables the model to generalize to sequence lengths it never saw during training. The key insight behind sinusoidal encodings is that any position can be expressed as a linear combination of nearby positions, which means a model trained on sequences up to length 512 can theoretically handle length 600.

Learned position embeddings abandon this mathematical guarantee. Each of BERT's 512 positions gets a 768-dimensional embedding vector that is randomly initialized and trained by gradient descent, exactly like word embeddings. This means position 37 and position 38 start with no particular relationship to each other; they only become related if the training data contains patterns that differ between those positions. In practice, this turns out to work well because NLP tasks rarely require generalizing to sequence lengths longer than those seen in training, and most downstream tasks use sequences well under 512 tokens. The tradeoff is that you cannot simply feed a 600-token sequence to BERT without modification, since positions 513-600 have no learned embeddings.

The learned embeddings appear less regular because they capture whatever positional patterns help with the pretraining objectives. The sinusoidal pattern's mathematical regularity means positions can be expressed as linear combinations of other positions, enabling some length generalization. BERT prioritized expressiveness over extrapolation since most downstream tasks don't require sequences longer than 512 tokens.

The core of BERT consists of stacked transformer encoder blocks. Each block applies multi-head self-attention followed by a position-wise feed-forward network, with residual connections and layer normalization around each sub-layer. This section covers both components in depth, including the subtle design choices that affect training stability and final performance.

Understanding each layer requires appreciating what information flows through it. The input to each layer is a matrix of shape (sequence length, 768), where each row is a 768-dimensional vector representing one token's current representation. The layer transforms this matrix to produce a new matrix of the same shape: same number of tokens, same dimensionality, but with updated representations that incorporate information from across the sequence via attention. The key design constraint is that this shape-preserving property must hold at every layer so that layers can be stacked arbitrarily.

Each transformer layer follows a consistent pattern:

1. Multi-head self-attention with residual connection
2. Layer normalization
3. Feed-forward network with residual connection
4. Layer normalization

BERT uses "post-norm" architecture, where layer normalization follows each sub-layer rather than preceding it. This differs from the "pre-norm" variant used in some later models like GPT-2.

The post-norm versus pre-norm distinction matters for training dynamics. In post-norm (used by BERT), the residual path and the transformed path are summed first, then normalized. This means the scale of the residual and the transformed signal can drift relative to each other during training, which sometimes leads to training instability in very deep networks. Pre-norm places the layer normalization before the transformation. This ensures each sub-layer sees normalized inputs, which tends to stabilize gradient flow in deep networks. The reason BERT used post-norm was simply that it followed the original Vaswani et al. transformer design without modification. Later work showed that pre-norm allows training of much deeper models without careful learning rate warmup, but for 12 or 24 layers, post-norm works well with appropriate hyperparameter tuning.

The histograms above demonstrate this effect. Raw dot products of 64-dimensional vectors have variance around 64, producing scores that span a wide range. After dividing by $\sqrt{64} = 8$, the variance drops to approximately 1, keeping scores in a range where softmax produces meaningful probability distributions rather than near-deterministic outputs.

Each layer's feed-forward network expands the representation to 4x the hidden dimension, applies a non-linearity, then projects back. The feed-forward network operates independently on each token position. Unlike the attention mechanism, which mixes information across positions, the feed-forward network applies the same transformation to each position separately. This makes it position-wise in the precise sense: the token at position 5 gets exactly the same function applied to it as the token at position 23, with no interaction between them.

The reason for the 4x expansion is not fully understood theoretically, but empirically it works well and the pattern has been reproduced across nearly every transformer architecture since. The expansion from 768 to 3,072 dimensions (or 1,024 to 4,096 in BERT-Large) creates a larger intermediate space where the model can express complex non-linear transformations before projecting back down. Some interpretability research suggests that feed-forward networks function as key-value memories: the first linear layer's weights act as keys that activate for specific input patterns, and the second linear layer's weights act as values that add specific information to the representation when those patterns appear. Under this view, the feed-forward network is where BERT stores factual knowledge it acquired during pretraining.

Combining attention and feed-forward with residual connections and layer normalization:

The layer maintains the same tensor shape between input and output, which is essential for stacking layers and enabling residual connections. The attention probabilities tensor indexes the batch, attention head, query position, and key position, confirming that each of the 12 heads computes its own attention pattern over the sequence.

Residual connections enable training of deep networks. They allow gradients to flow directly backward through the network, mitigating the vanishing gradient problem. Without residuals, training a 12 or 24-layer network would be extremely difficult. To understand why, consider what happens to a gradient as it backpropagates through a long chain of transformations. Each transformation multiplies the gradient by a Jacobian matrix. If these Jacobian matrices consistently have values less than 1, the gradient shrinks exponentially with depth, becoming too small to update early layers effectively. Residual connections create a "shortcut" in the gradient computation: the gradient of the loss with respect to early layers includes a direct path through the identity function (the residual) that does not pass through the potentially problematic Jacobian matrices of the transformer sub-layers. This is why you can train a 24-layer BERT-Large successfully while a 24-layer network without residuals would require extreme care with initialization and learning rates.

BERT's attention is bidirectional: every token can attend to every other token in the sequence. This contrasts sharply with the causal (left-to-right) attention used in GPT and other autoregressive models. The difference changes which patterns BERT can learn.

To appreciate the difference concretely, consider the sentence "The animal didn't cross the street because it was too tired." A language model reading left to right must decide what "it" refers to based only on "The animal didn't cross the street because." The causal model might correctly infer that "it" refers to "animal" based on the prior context, but it is working against itself: the word "tired" that resolves any ambiguity appears after "it" and is invisible during the processing of "it". BERT, by contrast, has access to the entire sentence simultaneously when computing the representation of "it", including the word "tired" that strongly suggests "animal" is the antecedent. This access to both preceding and following context is what makes BERT dramatically better at coreference resolution, semantic role labeling, and other tasks that require integrating distant context.

The bidirectionality also enables more powerful contextual disambiguation. The word "bank" means different things in "river bank" and "bank account". A left-to-right model processing "bank account" knows the word before "bank" is "the" and perhaps several earlier words about finance. A bidirectional model processing "bank" in either phrase sees the disambiguating word on both sides, making disambiguation significantly easier. Research on probing BERT's representations has confirmed that BERT develops rich, context-sensitive word senses that outperform previous static word embedding approaches by a wide margin.

Let's examine what attention patterns look like in practice. We'll create sample attention weights and visualize how different heads might specialize:

Research on BERT's attention patterns reveals consistent specialization:

• Early layers tend to attend broadly, with some heads focusing on the [CLS] token
• Middle layers develop syntactic patterns, with heads tracking subject-verb relationships, dependency arcs, and constituent boundaries
• Later layers show more semantic attention, with heads focusing on semantically related tokens

This layered specialization is not explicitly programmed. It emerges from pretraining on the MLM and NSP objectives. The fact that syntactic patterns appear in middle layers and semantic patterns in later layers mirrors how linguists think about the relationship between syntax and semantics: syntax provides the structural scaffolding on which semantic interpretation is built. BERT appears to recapitulate this hierarchy through gradient descent alone, which is one of the most striking findings from early BERT interpretability research.

The specialization of individual heads within layers is also notable. Clark et al. (2019) analyzed BERT-Base's 144 attention heads and found that different heads specialize in different syntactic relationships. One head tracks direct objects, another tracks possessives, another tracks coreference links. Some heads have a "previous token" pattern, attending primarily to the immediately preceding token. Others have a "first token" pattern, routing most attention to [CLS]. This suggests that multi-head attention decomposes contextual processing into simpler, specialized operations.

When processing batches of variable-length sequences, BERT uses attention masks to prevent attending to padding tokens. The mask is applied as a large negative value before softmax, effectively zeroing out attention to padded positions:

The large negative value (-10000) becomes approximately zero after softmax. This ensures padded positions receive no attention weight.

BERT produces contextualized representations at every position. Different downstream tasks use these representations in different ways. Understanding the structure of these outputs is essential for knowing how to apply BERT effectively to new tasks.

After all 12 (or 24) encoder layers have processed the input, you have a matrix of shape (sequence length, 768). Each row of this matrix is a contextualized representation of the corresponding input token. The representation at position 5 has been updated by every attention head in every layer, incorporating information from every other token in the sequence. This is fundamentally different from a static word embedding: the representation of the word "bank" depends on which other words appeared in the input. The same word gets a different 768-dimensional vector in every different context.

This contextuality is the source of BERT's power, but it also creates a practical question: how do you extract task-relevant information from a sequence of 512 potentially useful vectors? The answer depends on what your task requires.

The [CLS] token's final hidden state is designed to aggregate sequence-level information. During pretraining with the Next Sentence Prediction task, the [CLS] representation must contain enough information to determine whether two sentences are consecutive. This encourages [CLS] to capture global semantic content.

The [CLS] token is special in that it is prepended to every input regardless of the task. It has no inherent meaning as a word; its entire representation is built up through attention during the forward pass. Because [CLS] can attend to every other token in every attention head, it has the opportunity to aggregate information from the entire sequence. By the final layer, the [CLS] representation is a compressed summary of the input that has been shaped by the pretraining objective to capture sequence-level semantics.

In practice, the quality of [CLS] representations for downstream tasks depends heavily on whether the pretraining objective required sequence-level understanding. BERT's NSP pretraining task did require this, but later work by Lan et al. (2019) on ALBERT and by Liu et al. (2019) on RoBERTa questioned whether NSP helped. RoBERTa, which removed NSP from pretraining, sometimes matched or exceeded BERT's performance on sentence-pair tasks, suggesting that [CLS] can learn useful sequence-level representations even without an explicit sequence-level pretraining signal, as long as the MLM training corpus is large and diverse enough.

For classification tasks, you typically pass the [CLS] representation through a task-specific linear layer:

For sequence labeling tasks like named entity recognition or part-of-speech tagging, you use the representation at each token position:

For extractive question answering, BERT predicts start and end positions of the answer span. Two linear layers project each token's representation to start and end logits:

Each head produces output with the appropriate shape for its task. The classification head reduces the sequence to a single prediction per sample. The token classification head produces a label prediction for each position. The QA head generates start and end scores for every token, allowing span extraction by finding the highest-scoring start-end pair.

The flexibility of BERT's output representations is key to its success. The same pretrained model can power classification, sequence labeling, question answering, and many other tasks by simply changing the task-specific head.

A common question is whether to fine-tune BERT's weights at all, or to freeze them and only train the task-specific head. The answer is almost always to fine-tune. Frozen BERT features, while powerful, do not adapt to the vocabulary or style of your specific domain. Fine-tuning allows BERT's internal representations to shift slightly toward representations that are most useful for your task. For small datasets (under a few thousand examples), fine-tuning can overfit, in which case freezing lower layers while fine-tuning upper layers provides a useful middle ground.

Let's assemble all components into a complete BERT implementation:

Our implementation produces approximately 85 million parameters, which is lower than the full 110M of BERT-Base because we omit the masked language modeling head and some auxiliary components. The model processes a batch of 2 sequences with 32 tokens each, producing contextualized representations at every position plus a pooled representation for sequence-level tasks.

The architecture we have built mirrors the structure described in the original BERT paper almost exactly. The real BERT model available through Hugging Face adds a few details: the masked LM head is a linear layer mapping each token representation to a vocabulary-sized logit vector (30,522 outputs), and the NSP head is a linear layer mapping the pooled [CLS] representation to a 2-class logit (is-next / not-next). These heads are only used during pretraining and are discarded when you download a pretrained BERT model for fine-tuning. When you call BertModel.from_pretrained('bert-base-uncased'), you get exactly the encoder stack and embeddings we implemented here, with the pretraining heads stripped away and the weights initialized from the pretrained checkpoint.

When deploying BERT, two factors dominate resource requirements: memory consumption and computational cost. Both scale with sequence length, making long documents particularly expensive to process. Understanding these scaling properties is essential for making informed decisions about batch sizes, sequence length truncation, and hardware requirements before you begin a project.

BERT's memory usage during inference scales with sequence length squared due to the attention mechanism. For a batch of sequences, the dominant terms are:

• Embeddings: $O(\text{batch} \times \text{seq} \times d)$
• Attention scores: $O(\text{batch} \times \text{heads} \times \text{seq}^2)$
• Intermediate activations: $O(\text{batch} \times \text{seq} \times 4d)$

For BERT-Base with a sequence length of 512 and batch size of 1, the attention scores alone require approximately 12 × 512 × 512 × 4 bytes (float32) = 12.6 MB per layer, or about 150 MB for all 12 layers.

The computational complexity of BERT is dominated by three operations:

1. Attention: $O(n^2 \cdot d)$ for sequence length $n$ and dimension $d$
2. Feed-forward: $O(n \cdot d^2)$ with the 4x expansion
3. Embeddings: $O(n \cdot d)$

For typical sequence lengths (128-512), attention and feed-forward costs are comparable. The quadratic attention cost becomes prohibitive only for very long sequences, motivating efficient attention variants like Longformer and BigBird.

The crossover point between attention and feed-forward dominance is worth calculating explicitly. Attention cost scales as $O(n^2 d)$ and feed-forward scales as $O(n d^2)$, so attention dominates when $n > d$. For BERT-Base with $d = 768$, attention dominates for sequences longer than 768 tokens, which is already beyond BERT's 512-token limit. This means that for all practical BERT inputs, attention and feed-forward computation are similar in magnitude, and reducing sequence length reduces both approximately linearly.

In practice, most fine-tuning workloads for classification tasks truncate sequences to 128 tokens rather than using the full 512. At 128 tokens, BERT runs roughly 16 times faster than at 512 tokens (due to the quadratic attention term, a 4x reduction in length gives a 16x reduction in attention compute), with only modest performance degradation for tasks whose inputs fit in 128 tokens. This truncation is the single most impactful optimization you can make when deploying BERT in a latency-constrained environment.

BERT's architecture introduced several constraints that its successors have worked to address. The fixed sequence length of 512 tokens limits document-level understanding; longer documents must be chunked and processed separately, losing cross-chunk context. The quadratic attention complexity makes extending this limit computationally expensive. Later models like Longformer use sparse attention patterns to handle 4,096+ tokens efficiently.

The pretrain-then-finetune paradigm, while successful, requires task-specific training data and separate models for each task. Each time you want to add a new task, you must fine-tune a full copy of BERT on that task's training data and store the resulting model separately. For an organization with dozens of NLP tasks, this means maintaining dozens of BERT-sized models. This limitation motivated research into prompt-based methods where a single model handles multiple tasks through careful input formatting, culminating in the GPT-3 few-shot learning paradigm where no task-specific fine-tuning is required at all.

BERT also cannot generate text autoregressively, restricting it to discriminative tasks. This is a direct consequence of the bidirectional attention design: to predict the next token autoregressively, you must not allow the current token to attend to future tokens during training, because those future tokens would not be available at generation time. BERT's architecture has no such restriction, so it cannot generate coherent text. This creates a fundamental division between language-model architectures: encoder-only models like BERT excel at understanding tasks, while decoder-only models like GPT excel at generation tasks. Sequence-to-sequence models like T5, which use both an encoder and decoder, attempt to bridge this gap.

The [CLS] token aggregation approach has known limitations for long or complex texts. A single 768-dimensional vector must summarize an entire 512-token document. For short texts (a sentence or paragraph), this works well. For longer documents, task-relevant information may be lost in the compression. Alternative pooling strategies, such as averaging all token representations or using the maximum activation across positions, sometimes outperform [CLS] pooling on certain tasks, which suggests that the single-vector bottleneck is a real limitation.

Despite these limitations, BERT's architectural choices proved remarkably effective. The bidirectional attention mechanism captures context that autoregressive models miss. The three-embedding input representation elegantly handles sentence pairs. The standardized output format enables easy adaptation to diverse tasks. These design decisions established patterns that influenced nearly every subsequent language model.

BERT's release in late 2018 catalyzed a transformation in NLP. Within months, BERT or BERT-derived models topped leaderboards for question answering (SQuAD), natural language inference (MNLI), sentiment analysis (SST-2), and many other benchmarks. The pretrain-finetune paradigm became standard practice, and "BERT" became shorthand for transformer-based language understanding. The architecture we've examined in this chapter, while not the final word in language model design, remains a foundational reference point for understanding modern NLP. In the next chapter, we examine how later variants such as RoBERTa and ALBERT, as well as DistilBERT, changed BERT's training procedure and its efficiency at different scales while preserving its core architectural insights.

The BERT architecture is defined by a small set of core hyperparameters that determine model capacity, memory usage, and computational cost:

• hidden_size (768 for Base, 1024 for Large): The dimensionality of token representations throughout the model. Larger values increase capacity but quadratically increase attention computation costs.

• num_layers (12 for Base, 24 for Large): The number of stacked transformer encoder blocks. More layers enable more complex feature hierarchies but increase memory and compute linearly.

• num_heads (12 for Base, 16 for Large): The number of parallel attention heads. More heads allow the model to attend to different aspects of the input simultaneously. The head dimension is typically hidden_size / num_heads.

• intermediate_size (3072 for Base, 4096 for Large): The hidden dimension of the feed-forward network, typically 4× hidden_size. This expansion allows the FFN to learn complex transformations.

• max_position (512): The maximum sequence length the model can process. Longer sequences require more memory due to quadratic attention complexity.

• vocab_size (30,522 for BERT): The number of unique tokens in the vocabulary. Larger vocabularies reduce out-of-vocabulary issues but increase embedding parameter count.

• dropout (0.1): Applied to attention weights, feed-forward outputs, and embeddings during training to prevent overfitting. Set to 0 during inference.

When adapting BERT for specific applications, the most impactful parameters are max_position (for document length requirements) and dropout (for controlling overfitting on small datasets).

BERT's architecture combines familiar transformer components in a configuration optimized for language understanding. The key architectural elements include:

• Two model sizes: BERT-Base (110M parameters, 12 layers) and BERT-Large (340M parameters, 24 layers) balancing capability against deployment constraints
• Three embedding types: The input combines token embeddings with positional and segment embeddings, enabling rich representations of sentence pairs
• Bidirectional attention: Unlike autoregressive models, every token attends to every other token, capturing full context for understanding tasks
• Flexible outputs: The [CLS] token representation supports classification, token representations enable sequence labeling, and span predictions handle extractive QA

The architecture's success established the pretrain-finetune paradigm that dominated NLP through the early 2020s. While subsequent models have extended and improved upon BERT's design, understanding this architecture provides essential foundation for comprehending the evolution of language models.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about BERT architecture.