Masked Language Modeling › The Bidirectional Advantage

The core insight behind MLM is that understanding a word often requires seeing what comes after it, not just what came before. Consider the sentence:

The scientist studied the cell under a microscope.

When predicting "cell," a left-to-right model sees only "The scientist studied the." This provides some signal, but "cell" could still mean a prison cell, a biological cell, or a spreadsheet cell. The word "microscope" appearing later disambiguates completely, pointing to the biological meaning.

MLM allows the model to use this future context. By masking "cell" and asking the model to predict it, we force the model to integrate information from both "scientist studied" and "under a microscope" to make the prediction. The result is representations that encode richer semantic relationships.

The bidirectional advantage matters most for words whose meaning depends heavily on what follows them. In English, verbs often appear before their objects, so knowing the object changes how you interpret the verb. In German and many other languages, the verb comes at the end of the clause, making unidirectional left-to-right models especially unsuited for understanding. In all languages, pronouns and anaphora require looking backward and forward to resolve references. A sentence like "The trophy didn't fit in the suitcase because it was too big" requires understanding whether "it" refers to the trophy or the suitcase, a judgment that demands processing the entire sentence simultaneously.

Consider a few more examples where future context is decisive. In "I need to book a flight" versus "I need to read a book," the word "book" has different meanings that become clear from subsequent context. In "The bank failed" versus "The bank by the river flooded," the ambiguity resolves only when you see the rest of the sentence. In question answering, the model often needs to connect a question word to an answer that appears much later in the passage. All of these tasks benefit from full bidirectional context.

Notice that this advantage comes with a cost. Because the model sees the entire sequence at once, it cannot generate text one token at a time by conditioning on its own previous outputs. MLM is fundamentally a discriminative, not generative, pretraining objective. The model learns to score and fill in text, not to produce it from scratch. This design choice shapes everything downstream: what tasks the model excels at, how you fine-tune it, and what applications you can build.

This bidirectional context is the key advantage of MLM. For classification, entailment, question answering, and other understanding tasks, seeing the full context produces better representations than the partial view available to autoregressive models. When you fine-tune a BERT-style model on a classification task, every token in the sequence has already attended to every other token during the forward pass. By the time you read out the representation from the [CLS] token, it encodes information from the entire input rather than a running summary of what appeared so far.