Span Corruption › Sentinel Tokens

Sentinel tokens serve as placeholders for corrupted spans. Unlike BERT's single [MASK] token, span corruption uses multiple distinct sentinels, typically denoted <extra_id_0>, <extra_id_1>, and so on.

Why use different sentinels for each span? Consider reconstructing two spans from the corrupted input. If both used the same [MASK] token, the model couldn't distinguish which output corresponds to which span. Distinct sentinels create a clear mapping between input positions and output targets. The sentinel index directly encodes the span's position in the sequence, giving the decoder an unambiguous signal about which reconstruction task it is working on at each moment.

The target sequence concatenates all corrupted spans, each preceded by its corresponding sentinel:

Original: "The quick brown fox jumps over the lazy dog"

Corrupted Input: "The <extra_id_0> fox <extra_id_1> lazy dog"

Target: "<extra_id_0> quick brown <extra_id_1> jumps over the"

This format enables autoregressive generation of the target. The model sees <extra_id_0> and generates "quick brown" before the next sentinel signals a new span.

The target sequence design reflects a deliberate choice about when to stop generating for each span. When the decoder encounters a new sentinel token (<extra_id_1>) while generating after <extra_id_0>, it knows the current span has ended and a new one is beginning. This boundary signaling is implicit: the model learns through training that sentinel tokens delimit span boundaries, not through any special stopping logic in the architecture itself. The loss computed during training naturally teaches the model to emit a sentinel when the span is complete, because that is what the target sequence shows.

T5's vocabulary reserves 100 sentinel tokens (<extra_id_0> through <extra_id_99>). This typically suffices since even long sequences rarely contain more than 50-60 spans with a 15% corruption rate and average span length of 3. The 100-sentinel budget provides headroom for unusual cases: very long sequences, higher corruption rates during ablations, or fine-tuning settings where practitioners experiment with different hyperparameters. Reserving these tokens in the vocabulary does carry a small cost: 100 vocabulary slots that could otherwise encode additional subword units are dedicated to sentinels. In practice, given vocabularies of 32,000 tokens, this overhead is negligible.

It is worth understanding what happens if a sequence somehow requires more spans than available sentinels. The implementation simply caps the number of corrupted spans at the sentinel count. In practice with T5's standard hyperparameters, a 512-token sequence with 15% corruption and mean span length 3 produces around 26 spans, far below the 100-sentinel limit. The limit only becomes active if practitioners push corruption rates very high or apply span corruption to unusually long sequences.