Whisper Training › Training Objective and Optimization

Given audio features $\mathbf{x}$ and target text tokens $y = (y_1, \ldots, y_T)$, the model maximizes the log-likelihood. This objective function penalizes the model when it assigns low probability to the correct target tokens, encouraging it to learn the conditional distribution of text given audio:

$$\mathcal{L}(\theta) = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, \mathbf{x}; \theta)$$

where:

• $\mathcal{L}(\theta)$: the negative log-likelihood loss (cross-entropy) for the target sequence, summed over all positions
• $y_t$: the target token at position $t$ (the specific token the model should predict at this timestep)
• $y_{<t}$: all target tokens preceding position $t$ (the autoregressive context available to the decoder)
• $\mathbf{x}$: the input audio features (log-mel spectrogram encoded by the encoder into a sequence of context vectors)
• $T$: the total number of tokens in the target sequence
• $\theta$: the model parameters (weights and biases of the Transformer encoder-decoder)

The negative sign converts log-probabilities, which are negative values, into positive loss values that decrease as the model becomes more confident in the correct tokens. Summing across all positions ensures the model learns to predict accurately at every timestep, while the autoregressive conditioning on $y_{<t}$ captures the sequential dependencies needed for coherent text generation.

Note that this loss is computed over the entire output sequence, including the special task tokens at the beginning (such as <|startoftranscript|><|en|><|transcribe|>). This is intentional: the model must also learn to predict these task tokens correctly, which reinforces the task specification behavior. In practice, the loss contribution from the first few task tokens is small relative to the full transcript, but their inclusion in the objective ensures the model treats them as outputs rather than just conditioning signals.

This is computed using teacher forcing (as discussed in Part XII, Chapter 2), where the decoder receives the ground-truth previous tokens during training, not its own predictions. While teacher forcing can create exposure bias (a mismatch between training and inference conditions where the model is never trained to recover from its own mistakes), the massive scale of Whisper's training data mitigates this effect. The decoder learns reliable conditional distributions that generalize well to autoregressive generation because it has seen so many different acoustic and textual contexts that it can recover from minor prediction errors during inference.

The Whisper architecture is a standard encoder-decoder Transformer with modifications we detailed in the previous chapter (Part LI, Chapter 2). Training proceeds with several engineering choices that make large-scale training practical.

Data parallelism distributes the large dataset across multiple GPUs, with each GPU processing different batches simultaneously and gradients aggregated across devices via all-reduce operations. This is the same approach used in training the large language models we discussed in the scaling laws chapters, and it allows training time to scale nearly linearly with the number of GPUs for large enough batch sizes.

Mixed precision training uses FP16 or BF16 arithmetic to accelerate training and reduce memory usage (similar to modern LLM training discussed in Part XXXI, Chapter 9). The encoder's attention computations and feed-forward operations run in reduced precision, while a copy of the model weights is maintained in FP32 for the optimizer state. This allows larger batch sizes and faster matrix multiplications on modern GPU hardware.

Gradient accumulation achieves large effective batch sizes without requiring prohibitive amounts of GPU memory. Gradients are accumulated over multiple forward passes before performing a parameter update, simulating the effect of training with a larger batch without materializing all the activations simultaneously. For Whisper, large effective batch sizes are important because the training examples (30-second audio clips) are large, and the signal from weak labels benefits from averaging over many examples per update.

Learning rate scheduling uses linear warmup followed by cosine decay. The warmup phase gradually increases the learning rate from near zero to its peak value over the first 10% of training steps. This is important because early in training, the model weights are random and gradients can be large and inconsistent; a warm-up period prevents the optimizer from taking destructively large steps before the loss landscape becomes more well-behaved. The subsequent cosine decay smoothly reduces the learning rate to settle the model into a local minimum without oscillating.

The training objective is purely supervised; unlike modern LLMs that might use RLHF (which we covered in Part XXXVII), Whisper relies entirely on maximum likelihood estimation on the weakly supervised corpus. This supervised approach is feasible because the weak labels provide sufficient signal, and the diversity of the data serves the role that alignment procedures typically play in so robustness and generalization.

To improve robustness, Whisper applies several data augmentation techniques during training. These augmentations expand the effective training distribution beyond what the raw web data provides, helping the model generalize to acoustic conditions it may encounter less frequently in the training set.

SpecAugment applies time- and frequency-domain masking of mel spectrograms, similar to dropout but in the input space. Time masking removes contiguous blocks of time frames from the spectrogram, simulating brief audio dropouts, pauses, or recording glitches. Frequency masking removes specific frequency bands entirely, simulating telephone bandwidth limitations (which cut off high frequencies), audio compression artifacts (which affect specific frequency ranges), or the spectral effects of speaking with a mask or muffled microphone. SpecAugment forces the model to learn reliable representations that do not rely on any single time window or frequency region, encouraging it to use the full spectral-temporal context when making predictions.

Gain augmentation applies random volume changes to simulate different recording levels. This keeps the model recognizes speech whether it is shouted into a microphone or whispered from across a room. Volume normalization at inference time handles most real-world cases, but training with variable gain teaches the model to be invariant to absolute amplitude and instead respond to relative amplitude patterns (such as the ratio of voiced to unvoiced speech) that are more consistent indicators of speech content.

Speed perturbation applies minor changes to playback speed (0.9$\times$ to 1.1$\times$) without changing pitch, simulating natural variations in speaking rate and preventing overfitting to specific temporal patterns. Different speakers speak at dramatically different rates, and even the same speaker varies their rate across contexts, so training with speed perturbation explicitly teaches the model to handle this variability. Note that because the perturbation range is small (only 10% in either direction), the pitch remains perceptually similar, which is important: pitch-shifting would change the phoneme content and create false training examples.

These augmentations are particularly important given that Whisper is intended as a general-purpose model that will encounter audio from diverse sources: professional podcasts recorded in studios, amateur videos shot on phones, archival recordings from the mid-20th century, phone calls captured at 8 kHz, and everything in between.

A less discussed but practically important aspect of Whisper's training is how batches are constructed from the heterogeneous multilingual and multitask dataset. Naive uniform sampling would cause the model to see roughly the same distribution of languages as the training data, meaning it would see orders of magnitude more English examples than Swahili or Amharic examples. While this mirrors the real-world distribution, it risks underfitting the low-resource languages.

The Whisper team used a temperature-based sampling strategy for language selection within each batch. Rather than sampling uniformly proportional to dataset size, they sample languages with a probability proportional to $n_l^{\alpha}$, where $n_l$ is the number of training hours for language $l$ and $\alpha$ is a temperature parameter less than 1.0. When $\alpha = 1$, sampling is proportional to dataset size. When $\alpha < 1$, the sampling distribution flattens, giving low-resource languages more representation than their raw data count would suggest. This temperature sampling is the same technique used in multilingual text models like mBERT and XLM-R, and it provides a principled way to trade off between high-resource performance (favored by $\alpha \approx 1$) and low-resource performance (favored by $\alpha \approx 0$).

Task mixing follows a similar logic. Transcription data is far more abundant than translation data in the weakly supervised corpus, so without upweighting, the model would see many more transcription examples than translation examples per training step. The training pipeline applies task-level reweighting to ensure the model maintains translation capabilities even as transcription data dominates the corpus. This is analogous to the data mixing strategies we discussed in the context of instruction tuning, where balancing task types in the training mixture was found to be necessary for maintaining broad capabilities.