Whisper Training › Worked Example: Understanding the Training Data Flow

To solidify how Whisper training works, let's trace a single training example through the pipeline. Understanding this flow clarifies how raw audio and noisy text become a training signal for the model.

Consider a 30-second clip of Spanish audio from an interview, paired with YouTube's auto-generated Spanish captions. The captions contain minor errors and omit some filler words, representing the weak supervision paradigm.

Step 1: Audio Processing. The raw audio is converted to a log-mel spectrogram with 80 channels, resulting in a tensor of shape $(3000, 80)$ representing 30 seconds at 100 frames per second. This conversion extracts frequency features from the raw waveform, stressing perceptually relevant frequencies (the mel scale approximates human hearing sensitivity) while reducing dimensionality. The log compression helps normalize the dynamic range of audio signals. This makes the representation less sensitive to the absolute volume of the recording.

Step 2: Text Preparation. The caption text is tokenized using the multilingual BPE tokenizer. The target sequence becomes:

Note that special task tokens are prepended to indicate Spanish transcription without timestamps. These tokens are not present in the original YouTube captions; they are added during preprocessing to structure the training target. The <|es|> token tells the model to expect Spanish text, while <|transcribe|> indicates that the output should match the source language (as opposed to translation).

Step 3: Forward Pass. The encoder processes the spectrogram into contextualized representations through multiple Transformer layers. The decoder attends to these representations while predicting each token autoregressively using causal masking (as detailed in Part XIII, Chapter 4). At each step, the decoder can attend to the entire encoded audio sequence but only to previous text tokens in the target sequence. This keeps it learns to generate text left-to-right.

Step 4: Loss Computation. Cross-entropy loss is computed between the predicted token distribution and the ground-truth tokens. Despite errors in the "ground truth" captions, the gradient descent process averages over millions of examples, learning the statistical regularities of correct transcription. If the caption incorrectly says "programa" when the audio says "programma" (with a geminate consonant), this single error is drowned out by the thousands of correct examples of "programa" the model sees elsewhere. Over time, the model learns the dominant patterns of the language while filtering out noise through statistical averaging.

Step 5: Multitask Interleaving. In the next batch, the model might see English translation data, then Mandarin transcription, then timestamp prediction. This prevents the model from overfitting to any single language or task. The interleaving ensures that the model maintains all capabilities simultaneously, rather than suffering from catastrophic forgetting where learning one task degrades performance on another. The optimizer steps update the shared parameters based on the gradient from whichever task is currently being processed, creating a true multitask learning system where the shared encoder and decoder parameters must serve all tasks simultaneously.