Whisper Training › The Weak Supervision Approach

Traditional ASR systems depend on expensive, time-consuming human transcription. Creating 10,000 hours of verified transcripts might cost millions of dollars and take years. Professional transcriptionists must listen to audio repeatedly to ensure accuracy, and quality control requires additional reviewers to catch errors. This cost structure made it economically unfeasible to build datasets larger than a few thousand hours, effectively capping the performance of supervised ASR systems regardless of architectural advances.

Whisper sidestepped this bottleneck by using weak supervision: training on audio paired with existing text transcripts that were never intended for machine learning, such as subtitles and captions or other transcripts found alongside audio on the internet. These weak labels contain errors: subtitles might paraphrase rather than transcribe verbatim, auto-generated captions contain hallucinations, and translations might be loose approximations. However, they provide a important signal: the text is temporally aligned with the audio, creating a statistical relationship that neural networks can exploit.

The term "weak" in weak supervision refers specifically to label quality, not quantity or alignment. The labels are weak because they were generated without the strictness of human transcription: automated captioning systems make mistakes, human subtitle writers paraphrase, and translation systems introduce semantic drift. But each label is still correlated with the acoustic content it accompanies, which is what matters for training an ASR model.

Whisper was trained on 680,000 hours of audio, divided into several categories:

• 438,000 hours of English-only audio with weakly supervised labels
• 117,000 hours of multilingual audio covering 96 other languages
• 125,000 hours of $X \to \text{En}$ translation data (non-English speech with English translations)

This is approximately 100 times the size of the next largest publicly available dataset at the time of Whisper's release. To appreciate the magnitude: 680,000 hours is equivalent to approximately 77 years of continuous audio. If a human were to listen to this dataset working 40 hours per week, it would take over 3,000 years to hear it all. The important question is: how can a model learn from such noisy labels?

Weak supervision succeeds in Whisper for three key reasons, all relating to the nature of speech and language.

First, audio-visual alignment provides natural filtering. Subtitles and captions are temporally aligned with audio, even if imperfectly. When someone uploads a video with subtitles, the text generally corresponds to the speech, even if there are errors. This temporal correlation creates a strong prior: the text describes what happens during that time window, even if it is not a verbatim transcript. This is fundamentally different from random text paired with random audio, which would provide no learning signal. The alignment ensures that the model learns acoustic-semantic mappings rather than spurious correlations.

Second, the sequence-to-sequence architecture (which we explored in Part XII) provides inherent error correction. Unlike CTC-based models that make independent frame-level predictions, Whisper's encoder-decoder structure generates coherent sequences. The decoder's language modeling capabilities help correct errors in the weak labels during training, similar to how denoising objectives work in BART, which we covered in Part XXV, Chapter 5. When the weak label contains "the cat sat" but the audio clearly says "the cats sat," the decoder's understanding of subject-verb agreement and acoustic cues can override the training label error, learning the correct statistical pattern across millions of examples.

Third, the sheer diversity of the data provides robustness. While individual transcripts might contain errors, the aggregate distribution of 680,000 hours captures the full variability of human speech: different accents, background noises, speaking styles, and recording qualities. A model trained on this distribution learns to generalize across acoustic conditions that smaller, cleaner datasets cannot represent. The model learns that "hello" sounds different when whispered in a library versus shouted at a concert, because both variations appear in the training data with their approximate text labels.

The Whisper authors conducted ablation studies comparing models trained on different data regimes. They found that a model trained on the full 680,000 hours of weakly supervised data significantly outperformed models trained on smaller, higher-quality datasets like LibriSpeech (960 hours of clean, read speech). This confirms a pattern we observed in Part XXIII on Scaling Laws: for neural networks, quantity often beats quality when the quantity is sufficiently large. The model effectively performs its own quality control through statistical averaging: rare errors in labels are drowned out by the overwhelming number of correct examples, while the model absorbs the rich acoustic diversity that clean datasets lack.

The errors in weak supervision fall into distinct categories:

• Deletion errors: Subtitles skip sections of speech (common in live captions), omitting filler words, false starts, or sections the transcriber deemed unimportant
• Insertion errors: Subtitles include speaker names or sound effects (such as [APPLAUSE] or John:), or contain annotations not present in the audio
• Substitution errors: Words are mistranscribed or simplified, particularly proper nouns, technical terms, or accented speech
• Translation errors: Non-English audio paired with English subtitles (useful for translation training but noisy for transcription), where the text conveys the meaning but not the exact words spoken

A common misconception about weak supervision is that the model simply ingests raw internet data without any preprocessing. In reality, Whisper's training required a substantial data curation pipeline to filter out the lowest-quality material and organize the raw web data into usable training examples. The goal is not perfect labels but a floor of quality below which training examples are discarded entirely.

The Whisper team implemented several filtering stages. First, they detected the language of the audio using a preliminary speech recognition model and discarded examples where the detected language did not match the subtitle language. This catches common problems like a video in English accompanied by auto-translated German subtitles incorrectly labeled as German speech. Second, they applied heuristic filters to remove transcripts containing machine-generated artifacts such as auto-generated caption watermarks, excessive punctuation errors, or text with very low character coverage relative to audio duration. Third, they filtered out examples where the text was too short or too long relative to audio length, catching cases where subtitles were grossly misaligned.

Another necessary filtering step was detecting and removing machine-generated transcripts that were auto-generated by speech recognition systems rather than written by humans. If a model trained on ASR-generated transcripts, it would risk inheriting the systematic errors and biases of whatever system generated those transcripts, essentially learning from a degraded teacher. To avoid this, the team used transcript perplexity and formatting heuristics to identify and exclude auto-generated captions, preferring human-written subtitles even when they were less precise.