Whisper Training › Whisper Training: Weak Supervision and Multilingual ASR

OpenAI's Whisper transformed automatic speech recognition (ASR). Before Whisper, state-of-the-art speech models relied on carefully curated datasets containing a few thousand hours of high-quality, human-transcribed audio. This traditional approach created a basic bottleneck: collecting and transcribing audio data is extraordinarily expensive, time-consuming, and difficult to scale. Research teams would spend years assembling datasets like LibriSpeech (960 hours of clean audiobook recordings) or Switchboard (2,000 hours of telephone conversations), investing millions of dollars in professional transcription services and quality verification. These datasets, while pristine, captured only a narrow slice of human speech: typically read speech in quiet environments from specific demographic groups.

Whisper broke from this tradition by embracing weak supervision at internet scale, training on 680,000 hours of multilingual and multitask data scraped from the web. This approach, inspired by the success of large language models like GPT-3 that we discussed in Part XXVIII, demonstrated that speech recognition follows similar scaling laws to text: more data, even noisy data, yields more reliable and generalizable models. The revolutionary insight was that the internet already contains billions of hours of audio paired with text, in the form of subtitles and captions or other transcripts, created for human consumption rather than machine learning. By tapping into this reservoir of weakly aligned data, Whisper bypassed the data scarcity that had constrained ASR research for decades.

The key insight behind Whisper is that the diversity and quantity of training data matter more than perfect label quality. Traditional machine learning wisdom emphasized the garbage-in, garbage-out principle, assuming that noisy labels would corrupt model performance. Whisper challenged this assumption, showing that when the volume of data reaches internet scale, the noise averages out while the signal compounds. The statistical law at work is simple: if a label is wrong in 10% of examples but correct in 90%, and the model sees millions of instances, it will learn the correct underlying pattern. Incorrect labels add variance but not systematic bias, so gradient descent finds the right solution as long as the majority signal is present.

By training simultaneously on nearly 100 languages and multiple speech processing tasks (transcription, translation, language identification), Whisper learns representations that transfer remarkably well across domains and accents under varied acoustic conditions without fine-tuning. This chapter explores how Whisper was trained, the weak supervision methodology that made this scale possible, the data curation pipeline that filtered raw internet audio, the multilingual training strategy, and the capabilities that emerged from this approach.