Denoising Objectives › Token Deletion
Language AI HandbookAll chapters Denoising Objectives › Token DeletionOne section · Sections in this chapter3 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Token deletion randomly removes tokens from the input sequence. Unlike masking, which replaces tokens with a placeholder, deletion removes them entirely. The model must determine which tokens are missing and where they belonged.
This corruption is more challenging than masking because position information is lost. With masking, the model knows exactly where each missing token should go because the [MASK] placeholder preserves the sequence length and signals the exact location of each gap. With deletion, the remaining tokens are simply concatenated without any placeholder marking where deletions occurred. The model must infer both the missing content and its correct location from context alone.
In practice, this means the model's decoder must learn to output a sequence longer than its encoder input. For masked language modeling, the input and output are always the same length. For token deletion, the output can be longer than the input by exactly the number of tokens that were removed. This length-expansion capability is particularly useful for tasks like abstractive summarization, where a short compressed input must be expanded into a longer and more detailed output.
def apply_token_deletion(tokens, deletion_prob=0.15):
"""
Delete tokens randomly from the sequence.
Args:
tokens: List of tokens
deletion_prob: Probability of deleting each token
Returns:
corrupted: Tokens with some deleted
original: Original tokens for reconstruction target
"""
corrupted = []
for token in tokens:
if np.random.random() > deletion_prob:
corrupted.append(token)
return corrupted, tokens# Demonstrate token deletion
original_sentence = "The quick brown fox jumps over the lazy dog".split()
deleted, target = apply_token_deletion(original_sentence, deletion_prob=0.2)Original (9 tokens): The quick brown fox jumps over the lazy dog After deletion (6 tokens): The quick brown fox lazy dog Deleted 3 tokens (33% of original)
The corrupted sequence is shorter than the original. With a 20% deletion probability, we expect roughly 2 tokens to be removed from a 9-token sentence. The actual number varies due to random sampling. Notice that the deleted tokens could be anywhere in the sequence, and the remaining tokens are simply concatenated without any placeholder marking where deletions occurred. The model must learn to expand the sequence during reconstruction, inserting the missing tokens in the correct positions.
Token deletion teaches several important skills. First, the model learns less brittle representations that don't depend on specific tokens being present. Second, it learns about syntactic obligatoriness, recognizing when articles, prepositions, or other grammatical elements are missing. Third, it develops sensitivity to semantic completeness, detecting when content words have been removed. A sentence missing only function words ("quick brown fox jumps lazy dog") retains most of its meaning, while a sentence missing key content words ("The over the") loses its meaning almost entirely. Through many examples of both, the model learns which tokens carry the most informational weight.
Deletion Rate Selection
The deletion probability controls the difficulty of reconstruction. Too low (under 5%), and most sequences pass through unchanged. This provides little training signal. Too high (over 30%), and too much information is lost for reliable reconstruction.
A 15% deletion rate, matching BERT's masking rate, provides a reasonable balance. This removes enough tokens for meaningful learning while preserving sufficient context for accurate reconstruction. Note that BART's final configuration uses text infilling (30%) rather than pure token deletion, but 15% remains a common baseline when experimenting with deletion-based corruption. The BART paper found that pure token deletion performed worse than text infilling on generation tasks, because deletion alone doesn't train the model to handle variable-length gaps the same way that span-level masking does. However, deletion can be more effective than masking for tasks that specifically require understanding when information is absent rather than merely obscured.