Whisper Architecture › Architecture Variants

Whisper comes in several sizes, from Tiny (39M parameters) to Large-v3 (1550M parameters), letting deployment across different computational constraints:

The table encodes an important scaling strategy. Each step roughly doubles the number of layers and increases the width by a factor of roughly 1.3 to 1.5. This joint scaling of depth and width mirrors the principles from Part XXIII: Scaling Laws, where optimal compute allocation allocates parameters to both dimensions rather than scaling only one. A model with 32 layers but narrow width would struggle to stand for complex acoustic patterns in its limited hidden state, while a very wide but shallow model would lack the compositional depth to build hierarchical representations of speech.

The number of attention heads also scales with model size, following the convention that each head operates on a $d_{model}/n_{heads}$ dimensional subspace. For the base model, each of the 8 heads operates on 64-dimensional queries and keys; for the large model, each of the 20 heads operates on 64 dimensions as well. This consistency in per-head dimension is deliberate: each head learns to specialize in a different aspect of the acoustic or linguistic representation, and the dimension needs to be large enough to encode real patterns but consistent enough that learned attention patterns transfer between model sizes when distilling or fine-tuning.

All variants share the same architecture blueprint, differing only in depth, width, and attention heads. This scaling follows the patterns we explored in Part XXIII: Scaling Laws, where increased model capacity improves recognition accuracy, particularly for low-resource languages and noisy audio conditions. The smaller variants let deployment on edge devices and real-time applications where latency is necessary, while the Large model gives state-of-the-art accuracy for offline transcription and challenging acoustic environments. The consistent architecture across scales also means that techniques and fine-tuning strategies developed for one size typically transfer to others, letting practitioners to prototype with Tiny and deploy with Large without changing their processing pipeline.

The version history of Whisper's large models illustrates an important point about continued improvement without architectural change. The original Large model, Large-v2, and Large-v3 all share the same architecture but differ in training data and data augmentation strategies. Large-v2 introduced improved training procedures with better regularization and data filtering, reducing word error rates substantially on several benchmarks. Large-v3 further improved by training on a larger and more carefully filtered dataset with better multilingual coverage. This progression shows that for speech models at this scale, training data quality and quantity remain the primary drivers of performance improvement, consistent with the scaling laws we studied earlier.

A practical consideration when choosing between variants is the speed-accuracy tradeoff. The Tiny model runs in real-time on a modern CPU, which makes it suitable for applications where immediate transcription is more important than perfect accuracy, such as live captioning or voice command recognition. The Base and Small models offer improved accuracy at the cost of modestly higher latency, fitting well on GPU-equipped edge devices. The Medium model approaches Large-quality accuracy while running on consumer GPUs with 8-16GB of VRAM. The Large model requires at least 10GB of GPU memory just for the weights, which makes it practical primarily for server-side deployment or high-end workstations. Quantization techniques can reduce the memory footprint of all models by roughly 2 to 4 times without major accuracy degradation, making the Medium model usable on 4GB GPUs and the Large model on 8GB GPUs when precision requirements allow for INT8 inference.