Speech Representations › Worked Example: Computing a Mel Spectrogram

Let's trace through a concrete example to solidify these concepts. Consider a 1-second audio clip sampled at 16,000 Hz containing the spoken word "hello." This word contains a variety of phonetic elements: the aspiration of the "h," the diphthong quality of the "e," the lateral approximant "l," and the rounded vowel "o."

With a frame size of 25ms (400 samples) and hop length of 10ms (160 samples), we extract frames starting at samples 0, 160, 320, up to 15,840. This yields 100 frames for the 1-second clip. The total frame count depends on boundary handling: with zero-padding at the end, we get exactly 100 frames; without padding, slightly fewer.

For each frame, the processing proceeds as follows. First, we apply the Hamming window: multiply the 400 samples by the window function, tapering the edges to zero. The center 200 samples receive weights close to 1.0, while the first and last samples receive weights close to 0. This smoothing prevents the FFT from seeing artificial discontinuities at frame boundaries.

Second, we compute the 512-point FFT (padded with 112 zeros at the end to reach 512 samples) to get smooth spectral interpolation. The zero-padding does not add information; it simply fills in additional points on the continuous spectral curve, making the discrete frequency bins more closely spaced without changing the basic frequency resolution. Third, we take the magnitude and square it to get the power spectrum (257 bins from 0 to 8 kHz), discarding the complex phase information.

Fourth, we apply 80 mel filters, each spanning a range of FFT bins. The low-frequency filters, covering 0-500 Hz, span very few FFT bins each (perhaps 3-5 bins) because the mel scale allocates many filters to this low-frequency region. The high-frequency filters, covering 4-8 kHz, span many FFT bins each (perhaps 20-40 bins) because the mel scale compresses this region into fewer filters. Fifth, for each filter, we compute the weighted sum of power spectrum values, creating one mel-band energy value. Sixth, we take the natural logarithm to compress the dynamic range.

The result is a $100 \times 80$ matrix representing the time-frequency evolution of the utterance. The first few rows show low energy across all frequencies (silence before speech begins), followed by the broad-spectrum noise of the "h" aspirate. The "e" vowel appears as concentrated energy bands at specific frequencies (the first formant near 400 Hz and the second formant near 2000 Hz for this vowel). The lateral "l" sounds show a specific resonance pattern, and the rounded "o" shifts the formant pattern to lower frequencies (first formant near 500 Hz, second formant near 1000 Hz for a rounded back vowel). The final frames return to near-silence as the utterance ends.

This spatial structure in the mel spectrogram, where phonemes appear as distinctive horizontal patterns that shift vertically over time, is what transformer encoders learn to read. The self-attention mechanism can relate any two time positions, discovering that the formant pattern at frame 30 during the "e" corresponds to a particular phoneme while the shifted pattern at frame 70 during the "o" corresponds to another, even if the patterns are visually similar but physically at different frequency positions.