Prefix Language Modeling › Summary

Prefix language modeling combines bidirectional context encoding with causal generation. This gives a useful middle ground between MLM and CLM. This chapter covered the key concepts:

• The prefix LM split divides sequences into a bidirectional prefix and a causal target, enabling rich context understanding while maintaining generation capability
• The attention pattern forms an L-shape, with full attention in the prefix region and causal masking in the target region
• Loss computation focuses only on target positions, training the model specifically for conditioned generation
• UniLM-style training combines multiple objectives (bidirectional, causal, prefix) in a single model, creating versatile architectures
• Architectural choices include encoder-decoder and decoder-only implementations, each with distinct trade-offs for different use cases
• Applications include translation and summarization, as well as conditional tasks such as QA and dialogue, where the prefix-target structure matches the task naturally

The next chapter explores replaced token detection, an alternative to masking that uses a discriminative objective for more efficient pretraining.