Whisper Architecture › Encoder-Decoder Architecture

Whisper's architecture follows the standard transformer blueprint established in Part XVI: Transformer Architectures, with specific modifications for acoustic input and multitask output generation. The architecture must bridge the modality gap between continuous acoustic signals and discrete text tokens, requiring careful handling of sequence lengths and positional information. Understanding why each component exists and how it solves specific challenges in speech processing reveals the thoughtfulness of the design.

The encoder-decoder split is not arbitrary. In speech recognition, the encoder's job is to build rich contextual representations of the acoustic input, while the decoder's job is to convert those representations into text. These are fundamentally different tasks: the encoder needs to attend globally across the audio to resolve ambiguities (such as determining whether a phoneme belongs to one word or another based on context), while the decoder needs to generate tokens causally, conditioning each word on previous words and the entire audio representation. Separating these concerns into distinct modules allows each to be optimized for its specific role.

An alternative architecture would use a decoder-only model that autoregressively processes audio frames interleaved with text tokens, similar to how GPT-4o handles audio. This approach has appeal for streaming applications, since the model can generate text tokens as audio frames arrive without waiting for the complete audio. However, Whisper's encoder-decoder design has key advantages for batch offline processing: the encoder can attend bidirectionally to the complete audio before any text is generated, letting it to resolve ambiguities that only become clear in retrospect. A speaker who begins a sentence with an ambiguous word might disambiguate it five seconds later; the bidirectional encoder sees both parts simultaneously, while a causal decoder-only model would have already committed to an interpretation based on the early frames alone.

The overall information flow through Whisper's architecture proceeds in three phases. First, the encoder turns the raw spectrogram into a sequence of contextualized acoustic representations, one per 20-millisecond frame, integrating information from the full audio context. Second, these representations are frozen and stored as the encoder memory. Third, the decoder generates text token by token, at each step using cross-attention to query the encoder memory for the most relevant acoustic information. This separation means the encoder runs only once per audio clip (regardless of output length), while the decoder runs once per output token. For long transcripts with many words, this amortizes the encoder's computational cost effectively.

Before transformer layers process the spectrogram, Whisper applies a two-layer convolutional stem that downsamples the temporal resolution and projects the feature dimension. This preprocessing step is needed for computational efficiency and effective feature extraction. Without this downsampling, the transformer would face sequences of 3000 time steps, creating prohibitive computational costs for standard self-attention, which scales quadratically with sequence length as we discussed in Part XVII: Efficient Attention. A sequence of length 3000 would require attention matrices with 9 million entries per head per layer, rapidly exhausting memory resources during training and inference.

The first convolutional layer uses 3x3 kernels with stride (1, 2) along the time dimension, processing across frequency and time. This initial layer captures local spectral patterns, such as formant transitions and harmonic structures, while beginning the compression of the temporal dimension. The second layer uses stride (1, 1) with GELU activation. This gives nonlinear gating mechanisms that help the model select relevant acoustic features. This architecture reduces the sequence length from 3000 to 1500 while increasing the feature dimension to the model's hidden size (typically 768 for the "small" model or 512 for the "base" model).

This convolutional preprocessing serves multiple purposes beyond mere downsampling. Consider the analogy with computer vision: convolutional layers in image classifiers learn to detect edges, corners, and textures before higher-level features like faces and objects. Similarly, Whisper's convolutional stem learns to detect local spectral patterns like formant transitions, harmonic structures, and fricative noise characteristics before the transformer layers process higher-level linguistic patterns. The CNN layers pool information across adjacent time frames, effectively giving each position a wider temporal receptive field before the attention layers begin processing. This means the self-attention layers can focus on relationships between acoustic units (phonemes and syllables) rather than between individual spectrogram frames, which would be noisy and redundant.

The choice of a two-layer convolutional stem with stride 2 is a deliberate balance. A single strided layer would give less feature extraction depth, potentially leaving the transformer to handle too much low-level processing. A deeper stem or larger stride would lose temporal precision needed for accurate timestamp prediction. Two layers with total stride 2 is the sweet spot: sufficient local feature extraction without sacrificing temporal resolution. This design mirrors successful approaches in other audio processing models, including wav2vec 2.0, which also uses convolutional feature extractors before transformer layers.

Following the convolutional stem, Whisper adds fixed sinusoidal position embeddings to the sequence. Since the transformer architecture processes all positions in parallel without inherent sequential information, we must explicitly encode where each element occurs in the sequence. This is particularly important for speech because temporal order determines linguistic meaning; reversing the order of phonemes changes the word entirely, unlike in certain text processing scenarios where bag-of-words approaches might suffice.

As we explored in Part XIV: Positional Encoding, these embeddings map each position index to a unique vector using sine and cosine functions of varying wavelengths:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$ $$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

where:

• $PE_{(pos, 2i)}$: the positional encoding value at position $pos$ for even dimension index $2i$, computed using the sine function
• $PE_{(pos, 2i+1)}$: the positional encoding value at position $pos$ for odd dimension index $2i+1$, computed using the cosine function
• $pos$: the position of the token in the input sequence (from 0 to maximum sequence length)
• $i$: the dimension index within the embedding vector (from 0 to $d_{model}/2 - 1$)
• $d_{model}$: the total dimensionality of the model's hidden representations (e.g., 512 for the base model)
• $10000^{2i/d_{model}}$: the denominator scaling factor, ranging from 1 to 10000, which produces sinusoids with geometrically increasing wavelengths from $2\pi$ to $10000 \cdot 2\pi$

This geometric progression of wavelengths allows the model to attend to relative positions. For any fixed offset $k$, the relationship between the embeddings at positions $pos$ and $pos+k$ depends only on the offset $k$, not the absolute position $pos$. This property lets the model to learn position-invariant attention patterns and generalize to sequence lengths not seen during training. In the acoustic domain, this means the model can recognize that a certain phoneme pattern indicates a word boundary, regardless of whether that pattern occurs at the beginning or end of the 30-second window.

An important subtlety: sinusoidal positional encodings are added to the representations, not concatenated. This means the model must learn to disentangle content information from positional information through the attention mechanism. The addition is possible because the embeddings are designed so that different dimensions encode different wavelengths, letting the network to attend to specific frequency components of the positional signal. In practice, this works remarkably well: transformer models learn to extract positional information from the embedding dimensions that encode the relevant wavelengths while treating other dimensions as content features.

The choice of sinusoidal rather than learned positional embeddings for the encoder connects to an important practical concern: generalization to audio lengths not seen during training. Learned position embeddings are tied to specific integer positions, so a model trained on sequences up to length 1500 cannot gracefully handle longer sequences at inference time. Sinusoidal embeddings, by contrast, produce well-defined values for any position index, and the relative position property means the model's attention patterns can generalize from trained positions to untrained ones. For Whisper specifically, this matters less than for language models (since the encoder always sees exactly 1500 positions), but the choice shows the established best practice for encoder models designed to process fixed-length inputs with precise temporal structure.

It is also worth contrasting Whisper's approach to temporal encoding with rotary positional embeddings (RoPE), which we explored in Part XIV: Positional Encoding. RoPE encodes relative positions by rotating the query and key vectors before computing attention, which theoretically gives cleaner relative position information than additive sinusoidal embeddings. More recent speech models have adopted RoPE, and it is plausible that Whisper's architecture would benefit from it. However, additive sinusoidal embeddings were the standard when Whisper was designed, and they have proven sufficient for high-quality transcription, suggesting that positional encoding quality is not the limiting factor in Whisper's performance. The primary bottleneck is training data scale and acoustic diversity rather than positional encoding precision.

Whisper uses these fixed embeddings for the encoder only. The decoder, which generates text tokens, uses learned position embeddings instead. This asymmetry shows different requirements: the encoder must handle fixed-length acoustic sequences where relative position matters absolutely, while the decoder benefits from the flexibility of learned embeddings for variable-length text generation. The fixed sinusoidal embeddings ensure that the encoder maintains consistent geometric relationships between time steps, which is important for accurate timestamp prediction and temporal alignment. Learned embeddings are constrained to the sequence lengths seen during training, but since text output is always short relative to the 30-second audio window, this constraint does not cause issues in practice.

Both encoder and decoder consist of standard transformer blocks. The encoder uses full bidirectional self-attention, where every position can attend to every other position, capturing global acoustic context. The decoder uses causal (masked) self-attention, where each token can only attend to previous tokens. This keeps the model cannot "cheat" by looking at future text. The decoder additionally contains cross-attention layers that allow text generation to be conditioned on the encoder's acoustic representations.

Each block in both encoder and decoder includes:

• Multi-head self-attention to capture dependencies within the sequence (Part XIII)
• Cross-attention in the decoder only, connecting text generation to encoder outputs (Part XVI)
• Feed-forward networks with GELU activation (Part XV)
• Pre-normalization with LayerNorm before each sub-layer (Part XV)
• Residual connections around each sub-layer to preserve information flow

The attention mechanism follows the scaled dot-product formulation from Scaled Dot-Product Attention, computing attention weights between all positions in the sequence. For the encoder, this means every time frame can attend to every other frame, capturing long-range acoustic dependencies like prosody and speaker characteristics across the full 30-second context. This global receptive field allows the encoder to resolve ambiguities using distant context, such as determining whether a muffled syllable is part of a word based on the speaker's intonation pattern established seconds earlier.

The choice of pre-normalization (applying LayerNorm before the attention and FFN sub-layers rather than after) is important for training stability. As we discussed in Part XV: Transformer Blocks, pre-norm architectures train more reliably at large scale without learning rate warmup, because the gradients flowing through the residual path remain well-scaled even in early training. Whisper benefits from this property given the scale of its training data and the diversity of acoustic conditions it must handle.

The feed-forward networks in each transformer block use GELU (Gaussian Error Linear Unit) activation rather than the ReLU common in earlier models. GELU gives smooth, differentiable nonlinearity near zero, which has been shown empirically to improve training convergence for language models. In the speech domain, this smoother activation may help the model learn gradual spectral transitions more accurately than ReLU, which introduces a hard zero at negative inputs.

The decoder's self-attention uses causal masking to enforce left-to-right generation. Each token can only attend to previous tokens in the sequence, preventing the decoder from "seeing the future" during training and so that the generation process at inference time is consistent with how the model was trained. This is implemented by adding a mask matrix to the attention logits before the softmax, setting future positions to negative infinity so that their attention weights become zero.

Causal masking is necessary for technical correctness and for proper language modeling. If the decoder could attend to future tokens during training, it could trivially copy output text from future positions rather than learning to predict it from acoustic evidence and previous text. The causal constraint forces the model to develop real predictive capabilities, which is exactly what lets autoregressive generation at inference time.

The interaction between causal self-attention and cross-attention in each decoder layer deserves careful consideration. In each decoder block, the causal self-attention processes the current text sequence, building representations that integrate the history of previously generated tokens. These representations then attend to the encoder output through cross-attention, pulling in acoustic evidence. The combination means that the model conditions each new token on two sources of information: what has been said (via self-attention) and what was heard (via cross-attention). This dual conditioning is precisely what makes encoder-decoder models powerful for sequence-to-sequence tasks.

Cross-attention in the decoder is the mechanism that connects acoustic representations to text generation. When the decoder generates each text token, it uses cross-attention to look back at all 1500 encoder positions and determine which acoustic features are most relevant for the current token. This is where the model performs the basic mapping from sound to symbol.

The cross-attention computation follows the standard formulation. Given decoder hidden states $\mathbf{Q}$ (queries) and encoder outputs $\mathbf{K}$, $\mathbf{V}$ (keys and values):

$$\text{CrossAttn}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}\right)\mathbf{V}$$

where:

• $\mathbf{Q} \in \mathbb{R}^{t_{dec} \times d_k}$: query matrix from decoder hidden states, where $t_{dec}$ is the current decoder sequence length
• $\mathbf{K} \in \mathbb{R}^{1500 \times d_k}$: key matrix derived from encoder outputs, covering all 1500 acoustic positions
• $\mathbf{V} \in \mathbb{R}^{1500 \times d_v}$: value matrix derived from encoder outputs, containing the acoustic content to be retrieved
• $d_k$: the key dimension, used as a scaling factor to prevent the dot products from becoming too large

The scaling by $\sqrt{d_k}$ prevents the dot products from growing so large that the softmax saturates, which would cause vanishing gradients. As $d_k$ increases, the dot products grow proportionally to $d_k$ in expectation, so dividing by $\sqrt{d_k}$ keeps them at a manageable scale regardless of model size.

Critically, cross-attention uses no masking on the encoder side: the decoder can attend to any position in the encoder output at any step of generation. This is what makes encoder-decoder architecture powerful for speech: the model can use information from the end of an utterance to resolve ambiguity encountered at the beginning. If the speaker says something muffled early in the audio but uses a clarifying phrase later, the decoder can access that later acoustic information when generating the ambiguous word's transcription.

An important efficiency consideration is that the keys and values for cross-attention, computed from the encoder output, are the same at every decoder step. Since the encoder output does not change during generation, these key-value matrices can be computed once and cached, avoiding redundant computation. This key-value caching is standard practice in transformer inference and is one reason that long-form generation with Whisper is more efficient than it might initially appear: the expensive bidirectional encoder computation happens only once, and subsequent decoder steps are relatively cheap.

The number of cross-attention heads matches the number of self-attention heads in each model variant. This means the cross-attention mechanism can specialize across heads just as self-attention does, with different heads learning to focus on different aspects of the acoustic-linguistic alignment. Interpretability research on similar encoder-decoder models has found that some attention heads learn to track word boundaries, while others focus on prosodic features like pitch and stress. These specializations emerge from training without any explicit supervision. Multi-head attention can therefore learn several interacting aspects of the relationship between audio and text.