Whisper Architecture › The Multitask Format

Whisper's most innovative aspect is its treatment of diverse speech tasks as conditional text generation. Rather than training separate models for English transcription, Spanish-to-English translation, and language identification, Whisper uses special tokens to specify the desired operation within a unified decoding framework. This approach treats the decoder as a universal interface for speech understanding, where the same autoregressive mechanism can produce transcripts, translations, or language labels simply by varying the initial prompt tokens.

The idea draws direct inspiration from T5's text-to-text formulation, where every NLP task was recast as a sequence-to-sequence problem by prepending task descriptors to the input. Whisper applies the same principle to speech: since the decoder's output is text, all tasks that produce text from audio fit naturally within the autoregressive generation framework. The innovation is extending this to timestamp prediction, where the model interleaves temporal markers with text tokens to produce aligned transcripts.

The decoder's vocabulary extends beyond standard text tokens to include control tokens that structure the prediction task. These tokens function as instructions to the model, directing its behavior without requiring architectural changes or separate output heads. The full set includes:

• <|startoftranscript|>: Signals the beginning of generation and initializes the decoder state
• Language tokens: <|en|>, <|es|>, <|fr|>, etc. (99 tokens covering languages from Afrikaans to Welsh)
• Task tokens: <|transcribe|> or <|translate|>
• Timestamp tokens: <|0.00|> through <|30.00|> in 0.02-second increments (1500 tokens total)
• <|notimestamps|>: Disables timestamp prediction for that sample
• <|endoftext|>: Terminates generation

These tokens are part of the vocabulary in exactly the same way as ordinary text tokens. The model does not have a separate classification head for language detection or a separate regression head for timestamps. Instead, it predicts these control tokens through the same softmax output layer as all other tokens. This unification has a important training advantage: the model shares parameters between tasks, letting knowledge from one task to transfer to others. A model that has learned to identify Spanish phonemes for transcription has also learned features useful for Spanish-to-English translation.

This design mirrors the text-to-text framework we encountered in T5, where every task is cast as generating target text from source text. In Whisper's case, the "source" is the acoustic encoding, and the "target" is structured text prefixed with control tokens. By including language and task information in the target sequence itself, the model learns to condition its predictions on these explicit cues, letting zero-shot task transfer. A model trained on Spanish transcription and French translation can perform Spanish translation without ever seeing that specific task-language combination during training, provided it understands the individual components.

A typical decoding sequence for English transcription with timestamps appears as:

For translation from Spanish to English:

Note the order: language identification first (what language is being spoken), then task specification (transcribe vs translate), then potentially target language for translation, followed by the content. This hierarchical structuring allows the model to narrow its internal representation space progressively. First, it selects the appropriate acoustic and linguistic model for the source language. Second, it determines the output modality and target language. Finally, it generates the content under these constraints. This explicit conditioning prevents mode collapse, where the model might otherwise default to its most common training configuration.

The language token is particularly interesting from a multitask learning perspective. During inference, you can either force the language token to specify which language to expect, or allow the model to predict it freely from the acoustic content. The latter case performs automatic language identification as a byproduct of the generative process, using the same acoustic representations that will guide transcription. This means the model cannot be "tricked" into language misidentification the way a separate classifier might be: if the model predicts French because the audio sounds French, the subsequent transcription will use French linguistic patterns. The tasks are coupled by design.

During training and inference, the initial tokens are "forced" or prepended to the decoder input to condition the model:

1. <|startoftranscript|> always comes first to initialize the generation process
2. The language token is predicted by the model or specified by you based on the acoustic content
3. The task token specifies the operation to perform on that audio
4. If timestamps are enabled, timestamp tokens interleave with text tokens at appropriate boundaries

This conditioning allows a single model to serve multiple functions without architectural changes. The model learns to associate specific acoustic patterns with language identities and to map the same Spanish audio to either Spanish text (transcription) or English text (translation) based on the task token. During training, the model sees all these variations, learning that the same acoustic encoding can yield different text outputs depending on the decoder prompt. This multitask learning creates positive transfer between tasks: learning to translate improves the model's phonetic understanding, which in turn improves transcription accuracy.

The forcing mechanism also lets a useful inference strategy called "language forcing": instead of relying on the model's automatic language identification, you can force the language token to a specific value. This is useful when you already know the source language and want to prevent the model from misidentifying accented speech as a different language. Similarly, you can force the task token to override the default behavior. This keeps transcription rather than translation even when the model might otherwise prefer the latter.