Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

BART Pre-training › Token Deletion

Michael BrenndoerferOctober 19, 20254 min read

Language AI HandbookAll chapters

Part 25 of 67Encoder-Decoder Models

Chapter 181BART Pre-training › Token Deletion

Reading section 4 of 16 · Token Deletion · 4 minSection 4 of 16 · 4 min

Section 4 of 16 · 4 min read

BART Pre-training › Token DeletionOne section · Sections in this chapter4 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Token deletion takes masking one step further: instead of replacing tokens with a placeholder, we remove them entirely. This seemingly small change has measurable implications for what the model must learn. Where masking leaves behind a marker saying "something was here," deletion removes all evidence of the token's existence, forcing the model to infer the missing content and the locations of the gaps.

How It WorksLink Copied

Random tokens are deleted from the sequence with probability $p$ :

\tilde{x} = \{x_i : r_i > p\}

where:

$\tilde{x}$ : the corrupted sequence (a subset of the original tokens)
$x_i$ : the $i$ -th token in the original sequence
$r_i$ : a random value drawn independently for each token
$r_i \sim \text{Uniform}(0, 1)$ : each $r_i$ is drawn uniformly from the interval $[0, 1]$
$p$ : the deletion probability

The notation $\{x_i : r_i > p\}$ uses set-builder notation to express "keep only those tokens where the random draw exceeds $p$ ." Since each $r_i$ is uniformly distributed between 0 and 1, the probability that $r_i > p$ is exactly $1-p$ , meaning each token survives with probability $1-p$ . The corrupted sequence is shorter than the original (its expected length is $(1-p)$ times the original length), and there's no marker showing where deletions occurred. The tokens that remain simply adjoin each other, closing the gaps left by deleted content.

For example, "The quick brown fox jumps over the lazy dog" might become "The brown fox over lazy dog" with "quick," "jumps," "the," and possibly other tokens deleted. The resulting sequence reads almost grammatically but contains subtle errors.missing descriptors, absent verbs, dropped articles.that reveal the corruption.

What the Model LearnsLink Copied

Token deletion forces the model to solve two problems simultaneously, which creates a richer learning signal than masking alone:

Detection: Where were tokens deleted? Without mask tokens to mark positions, the model must infer deletion points from grammatical or semantic inconsistency. This requires understanding what should be present in well-formed text.
Prediction: What tokens were deleted? Once the model hypothesizes a gap location, it must determine what content is missing based on surrounding context.

This dual challenge is harder than masked language modeling. Consider "The brown fox over lazy dog": the model must recognize that an adjective is missing before "brown," that something is missing between "fox" and "over," and that an article is missing before "lazy."

The detection problem is especially important for generation tasks. A summarization model might receive a document with implicit information gaps (places where the source text assumes background knowledge that isn't stated) and must decide what content to generate and where. Token deletion pre-training provides practice at exactly this skill: identifying where information is missing and filling those gaps appropriately.

In[6]:

Code

def token_deletion(
    tokens: List[str], delete_prob: float = 0.15
) -> Tuple[List[str], List[str]]:
    """Apply token deletion corruption.

    Returns:
        Tuple of (corrupted_tokens, original_tokens)
    """
    corrupted = []
    for token in tokens:
        if random.random() >= delete_prob:  # Keep token if >= threshold
            corrupted.append(token)
    return corrupted, tokens

In[7]:

Code

# Example showing length difference
random.seed(123)
original = "The quick brown fox jumps over the lazy dog".split()
corrupted, _ = token_deletion(original, delete_prob=0.3)

Out[8]:

Console

Original (9 tokens):  The quick brown fox jumps over the lazy dog
Corrupted (5 tokens): brown jumps the lazy dog
Tokens removed: 4

With a 30% deletion probability, approximately 3 tokens are removed from the 9-token sentence. Notice the length mismatch: the decoder must generate a longer sequence than the encoder received. This requires the model to learn when to "expand" during generation, inserting tokens that have no corresponding position in the input. This skill is entirely absent from pure masked language modeling, where input and output lengths always match.