Whisper (2022) › The Solution

Whisper converted each task into conditional text generation. Audio entered the encoder, while control tokens told the decoder which language and task to use. The shared model could therefore switch behavior without loading a separate network for each request.

Audio is resampled to 16 kHz and divided into 30-second windows. Whisper converts each window to an 80-channel log-Mel spectrogram. Two convolutional layers process the spectrogram before a transformer encoder produces the audio representation. A transformer decoder then predicts text tokens.

The decoder prompt contains control tokens. A language token records the detected or supplied language. A task token selects transcription or translation into English, and another token selects whether timestamps should be emitted. This framing lets one set of weights learn the related tasks together.

The 680,000-hour dataset consisted of audio-transcript pairs gathered from the internet. About 438,000 hours were English transcription data. The paper reported 117,000 hours of transcription in other languages and 125,000 hours in which non-English speech was paired with an English translation.

Web data brought varied recording conditions at a scale unavailable in hand-labeled benchmarks, but it also brought unreliable labels. The authors applied automated filters for language agreement, transcript quality, and duplicate content. They also tried to remove machine-generated transcripts. These filters reduced obvious errors without making the corpus equivalent to manually verified speech data.

Training examples used the same decoder vocabulary but different token prefixes. An English transcription example and a French-to-English translation example therefore exercised the same network with different instructions. Timestamp tokens taught the decoder to align spans of text with positions in the 30-second audio window.

OpenAI trained model sizes from Tiny through Large. The family exposed a practical tradeoff: smaller checkpoints need less memory and run faster, while larger checkpoints generally produce lower error rates. English-only variants were also released for several sizes.