Speech Representations › The Mel Scale and Filterbanks

The mel scale addresses the mismatch between linear frequency and human perception by warping the frequency axis to approximate human pitch perception. Developed through psychoacoustic experiments in the 1930s and 1940s, where listeners judged subjective pitch distances between pure tones, the mel scale spaces frequencies according to perceived equality rather than physical equality. On the mel scale, equal distances correspond to perceptually equal steps, compressing high frequencies and expanding low frequencies to match the non-linear frequency resolution of the human ear.

The name "mel" comes from "melody," which reflects its origins in musical pitch perception research. The psychoacoustic experiments involved subjects adjusting the frequency of one tone until it sounded exactly halfway between two reference tones in pitch, building up a perceptual scale of equal pitch steps. This differs from the physical frequency scale in important ways: the mel scale is nearly linear below 1000 Hz and approximately logarithmic above, following the pattern of the basilar membrane's mechanical frequency analysis.

The mel scale conversion from frequency $f$ (in Hz) to mel $m$ follows:

$$m = 2595 \cdot \log_{10}\!\left(1 + \frac{f}{700}\right)$$

where:

• $m$: the perceived pitch in mel units, spaced according to human perception
• $f$: the physical frequency in Hertz (Hz)
• $2595$: the scaling constant that maps 1000 Hz to approximately 1000 mel, anchoring the scale at a convenient reference point
• $\log_{10}$: the base-10 logarithm that converts frequency ratios to perceived pitch differences, with approximately linear response below 1000 Hz and logarithmic response above

Alternatively, some implementations use the natural logarithm variant:

$$m = 1127 \cdot \ln\!\left(1 + \frac{f}{700}\right)$$

where:

• $m$: the perceived pitch in mel units
• $f$: the physical frequency in Hertz (Hz)
• $1127$: the scaling constant for the natural logarithm variant, mathematically equivalent to $2595 / \ln(10)$ to maintain the same mapping
• $\ln$: the natural logarithm (base $e$), giving the same perceptual warping as the base-10 variant

The two formulas are mathematically equivalent and produce the same frequency warping; they differ only in the numerical value of the mel units. The shape remains: approximately linear below 1000 Hz (where human hearing is roughly linear), transitioning to logarithmic above 1000 Hz. This reflects the structure of the cochlea, where the basal end (high frequencies) has logarithmic spacing of hair cells, while the apical end (low frequencies) has more linear organization. The constant 700 Hz represents the corner frequency where the transition occurs, determined empirically from psychoacoustic studies.

The inverse transformation from mel to Hz is also needed when constructing filterbanks:

$$f = 700 \cdot \left(10^{m/2595} - 1\right)$$

This inverse allows us to convert equally spaced points in mel space back to their corresponding Hz values for constructing the actual filters.

@fig-mel-scale-mapping illustrates this nonlinear relationship, showing how the mel scale compresses high frequencies and expands low frequencies to match human perceptual resolution.

To create a mel spectrogram, we need filters that collect energy according to this perceptual scale rather than the linear bins of the FFT, aggregating fine frequency details where they matter most for phonetic discrimination.

A mel filterbank consists of overlapping triangular filters spaced uniformly in the mel domain. The key insight is that "uniformly spaced in mel" means "logarithmically spaced in Hz," giving us dense coverage at low frequencies and sparse coverage at high frequencies, exactly matching the perceptual resolution of the ear.

The construction process involves five steps:

1. Select frequency range: Typically 0 Hz to the Nyquist frequency ($f_s/2$), or a narrower range like 0 to 8000 Hz for speech
2. Convert to mel scale: Map the frequency bounds to mel values using the mel formula
3. Create equally spaced points: Divide the mel range into $M+2$ points (where $M$ is the number of filters), creating boundaries for the triangular filters
4. Convert back to Hz: Map these mel points back to the linear frequency scale to determine the center frequencies of each filter in physical frequency units
5. Construct triangular filters: For each filter $m$, create a triangle with its peak at mel point $m$ and zero crossings at mel points $m-1$ and $m+1$

The unequal spacing in Hz means that low-frequency filters are narrow (covering perhaps 50-100 Hz each) while high-frequency filters are wide (covering 500-1000 Hz each). This is not a limitation but a feature: it matches the frequency resolution of the cochlea, which has more hair cells per octave at low frequencies than at high frequencies.

Mathematically, the $m$-th filter $H_m[k]$ applied to FFT bin $k$ (corresponding to frequency $f_k$) is defined as:

$$H_m[k] = \begin{cases} 0 & \text{for } f_k < f_{m-1} \\[4pt] \dfrac{f_k - f_{m-1}}{f_m - f_{m-1}} & \text{for } f_{m-1} \leq f_k \leq f_m \\[8pt] \dfrac{f_{m+1} - f_k}{f_{m+1} - f_m} & \text{for } f_m \leq f_k \leq f_{m+1} \\[4pt] 0 & \text{for } f_k > f_{m+1} \end{cases}$$

where:

• $H_m[k]$: the weight of the $m$-th mel filter applied to FFT bin $k$, ranging from 0 to 1
• $f_k$: the center frequency of FFT bin $k$ in the linear frequency domain (Hz)
• $f_m$: the center frequency of the $m$-th mel filter in Hz (the peak of the triangle)
• $f_{m-1}$ and $f_{m+1}$: the center frequencies of adjacent mel filters, defining the lower and upper bounds of the triangular passband
• The first case rejects frequencies below the filter's bandwidth
• The second case linearly interpolates from 0 to 1 as frequency increases from $f_{m-1}$ to $f_m$
• The third case linearly decreases from 1 to 0 as frequency increases from $f_m$ to $f_{m+1}$
• The fourth case rejects frequencies above the filter's bandwidth

The triangular shape has an elegant property: each pair of adjacent filters partitions their shared frequency band without gaps and without double-counting, since whenever filter $m$'s weight increases from 0 to 1, filter $m+1$'s weight correspondingly decreases from 1 to 0. This partition of unity property ensures that the total energy is conserved (no frequency component is lost or double-weighted) while giving smooth interpolation between filters.

The filters overlap, with each frequency bin adding to multiple mel bins. This smoothing reduces dimensionality while preserving perceptually relevant information, and provides a form of spectral averaging that makes the representation reliable to small frequency shifts. Typical ASR systems use 40, 80, or 128 mel filters, balancing resolution against computational cost. More filters provide finer frequency discrimination but increase the input dimensionality to the neural network. Whisper, the speech recognition model we will examine in subsequent chapters, uses 80 mel bins, a choice that provides good phonetic resolution while keeping the input compact.

To compute the mel spectrogram, we apply these filters to the power spectrum of each frame, effectively asking: "How much energy is present in each perceptual frequency band?"

$$S[m, t] = \sum_{k=0}^{N/2} |X[t, k]|^2 \cdot H_m[k]$$

where:

• $S[m, t]$: the mel spectrogram energy at mel filter $m$ and time frame $t$, representing the total power in that mel band
• $|X[t, k]|^2$: the power spectrum at time frame $t$ and FFT bin $k$, computed as the squared magnitude of the STFT
• $H_m[k]$: the weight of the $m$-th mel filter for FFT bin $k$, determining how much that frequency bin contributes to the mel band
• $t$: the time frame index, corresponding to discrete time steps (typically every 10 ms)
• $m$: the mel filter index, ranging from 0 to $M-1$ where $M$ is the total number of mel filters (typically 80)
• $k$: the FFT bin index, ranging from 0 to $N/2$ (the number of unique frequency bins for a real signal)
• $N$: the FFT size (number of points in the Fast Fourier Transform, typically 512 or 1024)

This summation aggregates energy across FFT bins according to the mel filter weights, creating a vector of $M$ values for each time frame. The result is a compressed representation where each mel bin represents the total energy in a perceptual frequency band rather than a single frequency component. The compression ratio is substantial: a 512-point FFT produces 257 unique frequency bins, but we condense these into 80 mel bins, reducing the frequency dimension by more than threefold while preserving the perceptually important structure.

Human loudness perception follows a logarithmic scale known as the Weber-Fechner law, which states that the just-noticeable difference in stimulus intensity is proportional to the current intensity level. A doubling of acoustic energy does not produce a doubling of perceived loudness; instead, we perceive loudness on a roughly logarithmic scale. This property has a practical consequence for speech processing: the raw energy values in mel spectrogram bins can span many orders of magnitude between a quiet fricative and a loud vowel. A softmax vowel might produce 1000 times the energy of a quiet "sh," meaning these two phonemes appear very different in linear scale but comparably different in log scale.

To match this perceptual non-linearity and to compress the dynamic range of the signal, we apply a logarithm to the mel energies:

$$\text{log-mel}[m, t] = \log(S[m, t] + \epsilon)$$

where:

• $\text{log-mel}[m, t]$: the log-compressed mel spectrogram value at mel filter $m$ and time $t$
• $S[m, t]$: the mel spectrogram energy at mel filter $m$ and time $t$ (linear scale)
• $\epsilon$: a small constant (typically $10^{-10}$ or $10^{-12}$) added for numerical stability
• $\log$: the logarithm function (natural log or base-10), compressing the dynamic range to match human loudness perception

The small constant $\epsilon$ prevents numerical instability when $S[m, t]$ approaches zero, avoiding negative infinity values in logarithmic space. The result is often expressed in decibels relative to some reference, though for machine learning purposes, the natural log or base-10 log suffices without explicit dB conversion.

Log compression also aligns with the properties that make mel spectrograms easy for neural networks to process. After log compression, the numerical range of the features is typically -100 to 0 dB (or an equivalent log scale), which is much more tractable for gradient-based optimization than raw energies spanning many orders of magnitude. Gradient updates are more stable when input features have comparable magnitudes across all dimensions, and log compression contributes substantially to this stability.

The log-mel spectrogram has become the standard input representation for neural speech recognition systems, giving a good tradeoff between:

• Temporal resolution: Capturing rapid phonetic transitions and consonant bursts
• Frequency resolution: Respecting perceptual frequency spacing where low frequencies require fine discrimination
• Dimensionality: Typically 80 dimensions per frame versus 256+ for linear spectrograms, reducing computational load
• Invariance: Robust to minor pitch variations and phase shifts that do not affect phonetic identity