Denoising Objectives › Text Infilling
Language AI HandbookAll chapters Denoising Objectives › Text InfillingOne section · Sections in this chapter7 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Text infilling combines aspects of span corruption and token deletion. Random spans of text are replaced with a single mask token, and the model must generate the missing content. Unlike span corruption where each span gets a unique sentinel, infilling uses a single mask token for all gaps.
This is more challenging than either simple masking or span corruption with unique sentinels because the model must determine both the content and the length of each missing span. A single [MASK] might represent one word or ten words. The model cannot treat each mask as a fixed-size prediction slot. Instead, it must learn to read the surrounding context and infer which words are missing and how many are missing. This length-prediction requirement is what makes text infilling so effective for training generation models: it forces the model to practice the same kind of variable-length expansion that it will need to perform during summarization, paraphrase generation, and story continuation.
The key insight is that text infilling trains the decoder to make length decisions. When generating text conditioned on a context, the model must decide how many tokens to produce before stopping. Standard autoregressive training on clean text teaches the model to produce a fixed target but doesn't specifically train it to infer the appropriate length from an incomplete input. Text infilling does exactly this, because every [MASK] token forces the model to decide: how many tokens does this gap need?
def apply_text_infilling(
tokens, mask_prob=0.15, mean_span_length=3, mask_token="[MASK]"
):
"""
Replace spans of tokens with a single mask token.
Args:
tokens: List of tokens
mask_prob: Probability of each token being in a masked span
mean_span_length: Average length of masked spans
mask_token: Token to insert for masked spans
Returns:
infilled: Tokens with spans replaced by mask tokens
original: Original tokens
spans: List of (start, end) tuples for masked spans
"""
n = len(tokens)
num_to_mask = int(n * mask_prob)
# Sample span lengths from geometric distribution
p = 1 / mean_span_length
spans = []
total_masked = 0
while total_masked < num_to_mask and len(spans) < n:
span_length = np.random.geometric(p)
span_length = min(span_length, num_to_mask - total_masked)
if span_length > 0:
spans.append(span_length)
total_masked += span_length
# Randomly place spans
mask = np.zeros(n, dtype=bool)
span_positions = []
for span_length in spans:
# Find valid start positions
valid_starts = []
for start in range(n - span_length + 1):
if not any(mask[start : start + span_length]):
valid_starts.append(start)
if not valid_starts:
continue
start = np.random.choice(valid_starts)
mask[start : start + span_length] = True
span_positions.append((start, start + span_length))
# Sort spans by position
span_positions.sort()
# Build infilled sequence
infilled = []
last_end = 0
for start, end in span_positions:
infilled.extend(tokens[last_end:start])
infilled.append(mask_token)
last_end = end
infilled.extend(tokens[last_end:])
return infilled, tokens, span_positionssentence = "The transformer architecture has revolutionized natural language processing".split()
infilled, original, spans = apply_text_infilling(
sentence, mask_prob=0.3, mean_span_length=2
)Original: The transformer architecture has revolutionized natural language processing Infilled: The transformer [MASK] revolutionized natural language processing Masked 2 tokens (25%) in 1 span(s): [2:4] 'architecture has' -> [MASK]
Multiple tokens collapse into single [MASK] placeholders. The 8-token sentence shrinks to fewer tokens because each span, regardless of length, becomes a single mask. The model must learn to generate the correct number of tokens for each mask, a more challenging objective than predicting a fixed number of tokens per placeholder.
Span Length Distribution
The geometric distribution controls how span lengths are sampled. With a mean of 3, most spans are short (1-2 tokens), but the heavy tail allows occasional longer spans that test phrase-level understanding.
The geometric distribution is a natural choice here because it has a memoryless property: the probability of a span having length given that it has length at least is constant. This means every additional token in a span is equally likely to be the last one, creating an exponentially decaying distribution of lengths. The parameter controls the rate of decay. A small produces a heavy-tailed distribution with many long spans, while a large produces mostly length-1 spans similar to standard token masking.
The distribution shows that roughly one-third of spans contain just a single token, similar to standard MLM. However, the remaining two-thirds contain multiple tokens, forcing the model to generate coherent phrases rather than isolated words. This balance between token-level and phrase-level prediction is key to text infilling's effectiveness. It ensures the model receives practice both at predicting individual words and at generating multi-word phrases and clauses, which is exactly what tasks like summarization and paraphrase generation require.