BERT Architecture › Summary
Michael BrenndoerferUpdated July 16, 20251 min read
Language AI HandbookAll chapters Part 24 of 67BERT and Variants
BERT Architecture › SummaryOne section · Sections in this chapter11 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BERT's architecture combines familiar transformer components in a configuration optimized for language understanding. The key architectural elements include:
- Two model sizes: BERT-Base (110M parameters, 12 layers) and BERT-Large (340M parameters, 24 layers) balancing capability against deployment constraints
- Three embedding types: The input combines token embeddings with positional and segment embeddings, enabling rich representations of sentence pairs
- Bidirectional attention: Unlike autoregressive models, every token attends to every other token, capturing full context for understanding tasks
- Flexible outputs: The [CLS] token representation supports classification, token representations enable sequence labeling, and span predictions handle extractive QA
The architecture's success established the pretrain-finetune paradigm that dominated NLP through the early 2020s. While subsequent models have extended and improved upon BERT's design, understanding this architecture provides essential foundation for comprehending the evolution of language models.