Masked Language Modeling › Understanding vs. Generation

MLM and CLM produce fundamentally different models suited for different tasks. The choice between them defines what the model can do, how it encodes language internally, and what kind of representations emerge from training.

Causal LM excels at generation because it models the natural process of producing text token by token. Each prediction extends the sequence, and the model can generate indefinitely by sampling from its predictions. The architecture enforces a causal constraint: position $i$ cannot attend to positions $j > i$. This means the hidden state at position $i$ encodes everything the model knows about the prefix $x_1, \ldots, x_i$, which is exactly the information you need to predict $x_{i+1}$. GPT and LLaMA, along with most chatbots, use CLM.

Masked LM excels at understanding because it captures relationships in both directions. For classification, the model can integrate information from the entire input before making a decision. For question answering, it can match question words with answer words regardless of their positions. For named entity recognition, it can determine a token's role from both its left and right neighbors. BERT and RoBERTa, along with most embedding models, use MLM.

The key reason MLM representations are better for understanding tasks is that each position's hidden state is conditioned on the full sequence. When you extract the [CLS] token's representation for classification, that vector has already attended to every other token in the input. When you extract a span's representation for question answering, those vectors encode bidirectional context. A CLM model's final hidden states are also conditioned on the entire prefix, but the model was never trained to produce representations that are useful for tasks requiring symmetric, full-sequence understanding.

Table mlm-vs-clm summarizes the key differences:

Neither approach is universally better. They're different tools optimized for different jobs. In practice, many systems combine both: an MLM encoder for understanding input, and a CLM decoder for generating output. T5 and BART take exactly this approach. T5 converts every NLP task into a text-to-text format and trains an encoder-decoder architecture where the encoder uses full bidirectional attention (like MLM) and the decoder uses causal attention (like CLM). BART pre-trains by corrupting documents with various noising functions and training the model to reconstruct the original, giving the encoder bidirectional context while training the decoder autoregressively. These hybrid architectures capture the representational advantages of both paradigms.