Whisper Training › Multitask Training Format

Following the text-to-text framework established by T5 (which we covered in Part XXV), Whisper frames all speech processing tasks as sequence-to-sequence translation problems. This unification is elegant in its simplicity: regardless of whether the task is transcription, translation, or timestamp prediction, the model receives audio features as input and generates text tokens as output. The decoder generates text tokens conditioned on audio features, with special tokens showing the desired task.

This approach eliminates the need for task-specific architectures. Traditional ASR systems might use different model structures for transcription versus translation, or require external language models for timestamp alignment. Whisper handles all these variations within a single framework, reducing engineering complexity and letting emergent capabilities through shared representations. The unified format also means that during inference, changing between tasks requires only changing the prompt tokens, not switching models or loading different weights.

Whisper uses a set of special tokens to specify the task, language, and format:

• <|startoftranscript|>: Begins the decoding sequence, signaling to the model that it should start generating output
• <|langid|>: Language identifier tokens (<|en|>, <|de|>, <|ja|>, etc. for 99 languages), conditioning the decoder on the target language
• <|transcribe|> and <|translate|>: Task specification tokens that modify the decoding behavior
• <|notimestamps|>: Instructs the model to produce clean text without temporal markers
• <|endoftext|>: End-of-sequence marker, signaling completion

For example, to transcribe English audio without timestamps, the decoder target begins with:

To translate German audio to English, the target becomes:

This unified format allows a single model to handle multiple tasks without architectural changes. The model learns that when it sees <|translate|> in the prefix, it should generate text in a different language from the source audio, effectively performing speech-to-text translation. During training, examples from different tasks (transcription, translation, language identification) are interleaved in the same batch, similar to the data mixing strategies used in instruction tuning. This interleaving prevents catastrophic forgetting and ensures balanced capability across all tasks.

The design of the special token vocabulary also encodes a deliberate information hierarchy. The language token always comes before the task token, which comes before the format token. This ordering teaches the model that language identity is a property of the audio source, while task and format are properties of the desired output. A model that understands this hierarchy can generalize to novel combinations: if it learns to transcribe audio from 99 languages and to translate from 40 of them, it develops an implicit understanding of what transcription and translation mean as operations independent of language, letting better generalization than a model where language and task are entangled.

Whisper can predict timestamps at the word or segment level by including time tokens in the vocabulary. The model learns tokens representing discrete time intervals (e.g., <|0.00|>, <|0.02|>, <|0.04|>, and so on up to 30 seconds). This allows it to align transcriptions with audio segments. This emerges naturally from training on subtitle data, which includes temporal alignment information. Subtitles inherently contain timestamps showing when text should appear and disappear on screen. This provides supervision for temporal alignment without requiring manual annotation.

When timestamp tokens are requested (by omitting <|notimestamps|> from the prefix), the decoder interleaves time tokens with text tokens:

The model learns to predict these time tokens based on the encoder's processing of the audio signal, effectively estimating the duration of spoken phrases and the boundaries between utterances. This capability is particularly valuable for video subtitling applications, where accurate timing is as important as transcription accuracy. The temporal resolution of 0.02 seconds (one time token every 20 milliseconds) is sufficient for subtitle alignment but not for phoneme-level alignment, which reflects the intended application domain.

The timestamp prediction task is harder than transcription because it requires the model to learn the relationship between acoustic events and absolute time positions. A word's duration varies based on speaking rate and emphasis as well as the acoustic context. The model learns these statistical relationships from subtitle timing across millions of examples, developing an implicit model of speech timing that generalizes across speakers and languages.