Masked Language Modeling › MLM for Representation Learning

The primary use of MLM is learning representations that transfer to downstream tasks. After pretraining, the model's hidden states capture rich semantic information that can be fine-tuned for classification, question answering, named entity recognition, and other tasks. The pretrain-then-fine-tune paradigm that BERT established has become the dominant approach for NLP and has since been adopted in computer vision and other domains.

The typical workflow is:

1. Pretrain on large unlabeled corpus with MLM objective
2. Fine-tune on labeled data for specific task
3. Infer using the fine-tuned model

During fine-tuning, the [MASK] token is never used. The model processes normal text and uses its pretrained representations as a starting point. Fine-tuning updates all weights to adapt to the specific task. The MLM head is discarded and replaced with a task-specific head: a linear layer for classification, a span-extraction layer for question answering, or a token-level classifier for named entity recognition.

The reason fine-tuning works so well is that MLM pretraining has already solved the hardest part: learning what language means. By the time fine-tuning begins, the model's encoder encodes syntactic roles and semantic relationships, plus coreference and world knowledge, in its hidden states. Fine-tuning then needs only to learn a relatively simple mapping from these rich representations to task outputs. This is why fine-tuning on small labeled datasets (hundreds to thousands of examples) can still produce high-quality models: the heavy lifting has already been done.

Consider the difference between fine-tuning a BERT model on sentiment classification versus training a sentiment classifier from scratch. From scratch, you must learn simultaneously how to represent words and sentences, as well as discourse, and how those representations map to sentiment. With BERT, you only need to learn the last mapping. The task-specific training signal does not need to reconstruct the entire representation space, only to adjust it for the specific task. This is why BERT-style models dominated NLP benchmarks for several years after their introduction.

The representations produced by pretrained MLM models are also valuable for tasks beyond supervised classification. Sentence embeddings derived from MLM encoders power semantic search systems that find documents matching a query even when no exact keywords match. Similarity scores between embeddings enable document clustering, duplicate detection, and recommendation. The [CLS] token's representation, which aggregates information from the entire input, provides a natural sentence-level embedding for downstream use.