BERT (2018) › 2018: BERT

In October 2018, researchers from Google AI Language published "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." The paper introduced BERT (Bidirectional Encoder Representations from Transformers), which achieved state-of-the-art results across eleven natural language understanding tasks, from question answering to sentiment analysis. Its main contribution combined those results with a reusable approach: pretrain a bidirectional transformer encoder on masked language modeling and next sentence prediction, then fine-tune it for many NLP tasks with minimal task-specific architecture changes.

BERT emerged at a critical moment in the evolution of language AI. The transformer architecture, introduced just a year earlier, had shown promise but had primarily been applied to machine translation. Researchers were testing transformers on understanding tasks because self-attention could capture contextual relationships between words. However, existing approaches to pretraining language representations had limitations. Traditional language models processed text left-to-right or right-to-left, never both directions simultaneously. As a result, their word representations omitted part of the available context, limiting their effectiveness on tasks that required information from both directions.

The breakthrough came from recognizing that pretraining for understanding tasks required different objectives than pretraining for generation. Unlike autoregressive language models that predicted the next word sequentially, understanding tasks benefited from bidirectional context. BERT solved this by using masked language modeling: randomly masking some tokens in the input and training the model to predict them from surrounding context in both directions. This simple change enabled the model to learn rich bidirectional representations that proved dramatically more effective than previous approaches.

BERT immediately raised the state of the art on the GLUE benchmark, outperforming previous systems by substantial margins. More significantly, BERT established a new paradigm for NLP: pretrain a large transformer model on massive text corpora, then fine-tune it for specific tasks. This paradigm became the standard approach for virtually all subsequent language models, from GPT-2 to T5 to modern large language models. BERT's success validated the transformer architecture for understanding tasks, not just generation, and showed how bidirectional context improved language understanding.