Whisper Architecture › From Audio to Spectrograms

While we assume familiarity with speech representations from the previous chapter, Whisper's specific input format warrants detailed review. The model consumes raw audio converted into log-mel spectrograms, which capture frequency content over time in a format suitable for neural processing. This representation bridges the gap between raw acoustic waveforms and the structured patterns that transformer architectures can effectively process.

The choice of log-mel spectrograms shows decades of speech processing research. The mel scale approximates human auditory perception, compressing high frequencies where the human ear is less discriminative and expanding low frequencies where fine distinctions matter for speech intelligibility. The logarithmic compression of amplitude mimics the human ear's nonlinear response to loudness, while also stabilizing the variance across different recording conditions. This preprocessing gives the model input aligned with human hearing rather than raw physical measurements.

To understand why these choices matter, consider the alternative of feeding raw waveforms directly. A 30-second audio clip at 16kHz contains 480,000 samples. Processing this sequence with a transformer's quadratic-complexity self-attention would be computationally prohibitive, and the raw waveform representation contains enormous redundancy since most of the information relevant for speech lies in the frequency domain rather than the time domain. The Short-Time Fourier Transform (STFT) projects the time-domain signal into a time-frequency representation, revealing which frequencies are active at each moment. By compressing these frequencies onto the mel scale, we reduce the frequency axis to 80 bins while stressing the frequency ranges most relevant to human speech (roughly 0 to 8kHz).

The logarithm of the power spectrum serves two purposes. First, it compresses the dynamic range: a whisper and a shout might differ by six orders of magnitude in raw power, but the log compression brings them into a more similar numerical range that neural networks can handle without extreme weight scaling. Second, it aligns with Weber's law from psychoacoustics, which states that the just-noticeable difference in loudness is roughly proportional to the current loudness level. The log transformation encodes this perceptual property directly into the feature representation.

Whisper processes audio in 30-second chunks sampled at 16kHz. The raw waveform undergoes a Short-Time Fourier Transform (STFT) to produce a spectrogram, which is then mapped to 80 mel-frequency bins. The logarithm of these values yields the input features. Unlike image-based approaches that might use raw waveforms or complex feature engineering, Whisper follows the tradition of ASR systems while keeping the preprocessing minimal and fixed. This standardization means that the model learns to be reliable to variations in recording quality, background noise, and speaker characteristics directly from the data, rather than relying on hand-engineered feature extraction that might fail in novel acoustic environments.

The resulting input tensor has shape $(80, 3000)$, representing 80 mel channels across 3000 time frames (since 30 seconds at 16kHz with a hop length of 160 samples yields 3000 frames). This two-dimensional representation is the "image" that Whisper's encoder processes. The fixed 30-second window gives enough context for most utterances while keeping computational requirements predictable and batch processing efficient. For audio shorter than 30 seconds, the sequence is padded with silence; for longer content, the audio is segmented into overlapping windows, though this introduces the boundary challenges we will discuss in the limitations section.

The specific parameter choices deserve attention. A hop length of 160 samples at 16kHz corresponds to 10 milliseconds between successive frames. This temporal resolution is finer than a typical phoneme duration (roughly 50 to 150 milliseconds). This keeps phoneme transitions appear as smooth trajectories in the spectrogram rather than abrupt discontinuities. The window length of 400 samples (25 milliseconds) is long enough to estimate frequency content accurately while short enough to track rapid changes. These values stand for the standard choices from decades of speech processing research, and Whisper inherits their acoustic desirability while discarding the complex feature processing stages that traditionally followed.