Whisper Architecture › Worked Example: Decoding a Transcript

To see all these pieces work together, let's trace how Whisper processes a 10-second English audio clip from raw audio to structured text output.

Before tracing through the complete pipeline, note that this example uses a 10-second clip to keep the numbers concrete. In practice, Whisper is most commonly applied to clips that are already segmented into roughly sentence-length utterances by a preprocessing step, or to full recordings that are chunked with overlap to handle the 30-second limit. The tracing below focuses on the single-chunk case, which is the atomic unit of Whisper's processing regardless of how the audio was segmented.

Step 1: Input Processing. The audio is sampled at 16kHz, giving 160,000 raw samples for 10 seconds. The STFT computes frequency content at each 10-millisecond frame, creating a spectrogram with roughly 1000 time frames. Mapping to 80 mel bins gives a $(80, 1000)$ array. This is padded with silence to $(80, 3000)$ to fill the required 30-second window, since Whisper always expects fixed-size input.

Step 2: Convolutional Stem. The two-layer convolutional network processes the $(80, 3000)$ spectrogram. The first layer with stride 2 in time reduces the temporal dimension to 1500 frames while projecting to $d_{model}$ feature dimensions. The second layer with stride 1 applies further nonlinear transformations, creating a $(1500, d_{model})$ feature sequence.

Step 3: Positional Encoding. Sinusoidal positional embeddings are added to each of the 1500 positions, encoding temporal location information. Each frame now carries both its acoustic content and its temporal position.

Step 4: Encoder Processing. All 1500 positions pass through the encoder's transformer blocks. Because self-attention is bidirectional, every frame can attend to every other frame. The encoder builds contextual representations where each position encodes local acoustic features and information about the entire 30-second context, including what came before and after. After all encoder layers, we have a $(1500, d_{model})$ matrix of contextualized acoustic representations.

Step 5: Decoder Initialization. The decoder begins with the <|startoftranscript|> token. This single token is embedded and processed through the first decoder layer's self-attention (trivially, since there is only one token), followed by cross-attention that queries all 1500 encoder positions to determine the most relevant acoustic features.

Step 6: Language Identification. The decoder predicts the next token, which should be a language identifier. The cross-attention mechanism focuses on gross acoustic features like pitch range, vowel quality, and consonant patterns that distinguish languages. The model predicts <|en|> for English, conditioning all subsequent generation on English linguistic patterns.

Step 7: Task Specification. With the language established, the decoder predicts the task token. For this example, the model predicts <|transcribe|>, showing verbatim transcription.

Step 8: Content Generation. The decoder now enters the main generation loop:

• Predicts <|0.00|>, anchoring the start of speech at time zero
• Predicts "Hello" as the first word, using cross-attention to the early acoustic frames
• Predicts <|1.25|>, showing the timestamp after "Hello"
• Predicts "and" attending to the next acoustic segment
• Continues until speech ends

Step 9: Termination. The decoder predicts <|endoftext|> when the cross-attention weights indicate no more speech content remains in the encoder representations.

Each prediction attends to the encoder output via cross-attention and all previously generated tokens via causal self-attention, exactly as described in Part XIII: Cross-Attention. The decoder cannot attend to future tokens. This keeps the model respects the temporal order of speech and maintains causality in its predictions.