Whole Word Masking › The Subword Masking Problem

Modern language models use subword tokenization to handle vocabulary efficiently. Algorithms like WordPiece and BPE, along with SentencePiece, break words into smaller units based on frequency or likelihood, balancing vocabulary size against the ability to represent any input without unknown-token fallbacks. Common words like "the", "is", and "run" remain intact as single tokens. Moderate-frequency words decompose into recognizable pieces: "transformation" becomes ["transform", "##ation"]. Rare or highly specific words may split into many fragments: "photosynthetically" might become four or five subword pieces.

This design is essential for handling open vocabularies and morphologically diverse text. A fixed-word vocabulary of 30,000 entries would fail on technical jargon, proper nouns, or even common words in inflected forms. Subword tokenization sidesteps this problem elegantly. But subword tokenization interacts badly with random masking because it creates a fundamental asymmetry between single-token and multi-token words.

Think of it as two different exam questions for the same concept. The first question shows you a word with the last three letters blanked out: "transfor___". The second question hides the entire word: "The _________ was complete." Both questions test knowledge about the same word, but they test very different things. The first tests your knowledge of English morphology, the second tests your understanding of the sentence's meaning. When you train BERT with random subword masking, a significant fraction of the "questions" the model faces are of the first type: morphology lookups rather than semantic reasoning.

This decomposition creates an asymmetry in the masking process. When we randomly select 15% of tokens for masking, we are selecting subword tokens, not words. A multi-token word might have some subwords masked while others remain visible. The visible subwords leak information about the masked ones. The degree of leakage depends on how much information a prefix or suffix carries about the rest of the word, and for many common English morphological patterns, this information is substantial.

Consider how BERT processes the sentence "The transformation was remarkable." Using WordPiece tokenization:

If random masking selects only "##ation", the input becomes:

The model sees "transform" immediately adjacent to the mask. How many English words start with "transform-"? Only a handful: transformation, transformed, transforming, transformer. The prediction task collapses from choosing among 30,000 vocabulary items to distinguishing between 3-4 suffixes. With a bit of context bias (this is probably a noun form given the sentence structure), the model can essentially narrow it down to one answer without consulting the rest of the sentence at all.

This is problematic for several reasons:

• Weak learning signal: The model learns to pattern-match subword combinations rather than understand context deeply. It does not need to reason about meaning, just morphology. Gradient updates during backpropagation reflect this: they encode "##ation follows transform-" rather than "this word fits here because of its semantic role."

• Uneven difficulty: Single-token words like "was" face the full prediction challenge when masked, while multi-token words get easy hints. The model develops uneven representations, with single-token words anchored more firmly in their semantic contexts and multi-token words understood more through their morphological structure.

• Distributional mismatch: During fine-tuning, the model sees complete words. Pre-training on partial words creates a subtle distribution shift. The model has learned to rely on subword co-occurrence statistics that simply do not apply during inference, where every word appears complete.

• Gradient dilution across multi-token words: When only one subword of a word is masked, only that subword's prediction contributes to the loss. The model updates its parameters based on a small, easy prediction, rather than the harder full-word prediction that would produce richer gradient signal.

The problem compounds across training. Early in training, the model might struggle with suffix prediction. But it quickly learns the most common suffix patterns for each root, at which point those examples become nearly loss-free and contribute almost nothing to further learning. The model has overfit to morphological patterns and underlearned semantic patterns, precisely the opposite of what we want for a general-purpose language model.

To understand exactly how much information leaks when sibling subwords are visible, we need a way to measure uncertainty. How hard is the prediction task? Information theory gives us a precise tool: entropy.

The intuition behind entropy

Imagine you are playing a guessing game. If someone picks a number between 1 and 1,000, you have high uncertainty: you need many guesses or much information to identify the answer. If they pick between 1 and 2, you have low uncertainty: one yes-or-no question resolves it. Entropy quantifies this precisely: it measures how many "bits" of information you need to identify the answer. More possible answers with similar probabilities means higher entropy; fewer possibilities or more concentrated probability means lower entropy.

For language model predictions, entropy captures how "spread out" the model's probability distribution is over the vocabulary. When the model is very confident about one token, entropy is low. When it is nearly equally uncertain between many tokens, entropy is high. A good training signal comes from high-entropy situations: the model must work hard to make the right prediction, and the resulting gradient update carries rich information about what the model still needs to learn.

The entropy formula

Given a masked position, the model outputs a probability distribution over all possible tokens. We want to quantify how uncertain that distribution is. The conditional entropy of this prediction is:

$$H(x_{\text{mask}} \mid x_{\text{context}}) = -\sum_{v \in V} P(v \mid x_{\text{context}}) \log P(v \mid x_{\text{context}})$$

where:

• $H(x_{\text{mask}} \mid x_{\text{context}})$: the conditional entropy of the masked token given visible context, measured in bits (if using $\log_2$) or nats (if using natural log)
• $x_{\text{mask}}$: the masked token the model must predict
• $x_{\text{context}}$: the visible tokens surrounding the masked position
• $V$: the vocabulary of all possible tokens
• $P(v \mid x_{\text{context}})$: the probability the model assigns to token $v$ being the correct prediction given the context

The formula works by taking each possible token, weighting its log-probability by the probability itself, and summing. The negative sign ensures we get a positive number (since log probabilities are negative). When probability concentrates on one token, that token's contribution $P \log P$ is large and negative, but all others contribute near zero, so the total sum (before negation) is a large negative number, and after negation the entropy is small. When probability spreads uniformly across many tokens, many moderate contributions sum to a large negative number, and entropy is high.

Why does this formula make sense? Notice that $-\log P(v)$ measures how "surprising" token $v$ would be: common tokens have probabilities near 1, making their log near 0 (unsurprising), while rare tokens have small probabilities and large negative logs (very surprising). Entropy is the expected surprise under the model's own distribution. High entropy means the model expects to be frequently surprised, so it is uncertain and must rely on context to narrow down the answer.

Entropy bounds tell us about task difficulty

Two extreme cases illuminate the formula's behavior:

1. Maximum entropy: When all $|V|$ tokens are equally likely, each has probability $1/|V|$, and entropy reaches $\log |V|$. For a vocabulary of 30,000 tokens, that is about 14.9 bits: a difficult prediction requiring deep contextual reasoning.

2. Minimum entropy: When one token has probability 1.0 and all others have probability 0, entropy equals 0. The prediction is trivial; the model is certain before even reasoning about context.