Speech Representations › Audio Preprocessing Pipeline

Before computing mel spectrograms, several preprocessing steps condition the signal for analysis. These steps standardize the input, emphasize speech-relevant frequencies, and prepare the signal for spectral analysis. Each step addresses specific physical or perceptual properties of speech production and hearing, forming a principled pipeline that has evolved through decades of speech processing research.

Speech signals have more energy at low frequencies (below 1 kHz) than at high frequencies, due to the physics of vocal fold vibration and vocal tract resonance. As the vocal folds open and close, they create a glottal pulse rich in harmonics, but the spectral envelope naturally rolls off at approximately -6 dB per octave due to the radiation characteristic of the lips and the physical properties of the glottal source. This spectral tilt causes high-frequency consonants (like fricatives) to be relatively weak compared to vowels. If we compute the mel spectrogram without pre-emphasis, the low-frequency vowel energy will dominate and high-frequency consonant information may be too faint for the network to learn from reliably.

Pre-emphasis applies a first-order high-pass filter to compensate:

$$y[n] = x[n] - \alpha \cdot x[n-1]$$

where:

• $y[n]$: the pre-emphasized signal at sample $n$
• $x[n]$: the input signal at sample $n$
• $\alpha$: the pre-emphasis coefficient (typically 0.97), controlling the degree of high-frequency boost
• $x[n-1]$: the input signal at the previous sample (delay of one sample)

With $\alpha = 0.97$, this filter boosts high frequencies by approximately 6 dB per octave, flattening the spectrum and compensating for the natural roll-off of speech. To understand why $\alpha = 0.97$ rather than some other value, consider the z-domain transfer function of this filter: $H(z) = 1 - \alpha z^{-1}$. Its zero is at $z = \alpha = 0.97$, very close to the unit circle at low frequencies, meaning the filter has weak attenuation at DC and increasingly strong attenuation as frequency decreases toward zero. The result is a gentle high-pass filter that amplifies high frequencies without introducing audible artifacts.

Pre-emphasis also compensates for the radiation characteristic of the lips, which attenuates low frequencies relative to high frequencies. By equalizing the spectrum, pre-emphasis ensures that high-frequency consonant information (necessary for distinguishing "s" from "sh" or "f" from "v") receives adequate representation in the subsequent spectral analysis.

The effect is reversible through de-emphasis, though in practice recognition systems work entirely in the pre-emphasized domain since the neural network learns to interpret the equalized spectral patterns directly.

Speech analysis proceeds frame-by-frame to respect the quasi-stationarity assumption discussed earlier. Standard parameters include:

• Frame size: 25 milliseconds (400 samples at 16 kHz)
• Frame shift (hop length): 10 milliseconds (160 samples at 16 kHz)
• Overlap: 60% (15 ms of each frame overlaps with neighbors)

This overlap ensures temporal continuity and prevents information loss at frame boundaries. A phoneme transition falling between frame centers will still influence adjacent frames, maintaining acoustic context. The 25ms window provides sufficient duration to resolve frequency details (frequency resolution is inversely proportional to window length) while being short enough to capture rapid phonetic changes.

The relationship between window length and frequency resolution is quantitative. For a window of $N$ samples at sampling rate $f_s$, the frequency resolution is $\Delta f = f_s / N$. At 16 kHz with $N = 400$ samples (25ms), this gives $\Delta f = 16000 / 400 = 40$ Hz per frequency bin. Formant frequencies in typical male speech are spaced roughly 1000-1200 Hz apart, meaning our 40 Hz resolution is more than sufficient to separate them. For higher-pitched voices with formants spaced closer together, the 25ms window remains adequate; the basic frequency (100-300 Hz for most speakers) determines the harmonic spacing, and the formant envelope is typically much smoother than individual harmonics.

After extracting each frame, we apply a window function (Hamming or Hann) to taper the signal to zero at the edges. Without windowing, the sharp discontinuities at frame boundaries would introduce spectral artifacts called sidelobes, spreading energy from strong frequencies into other frequency bins and obscuring the true spectral content.

Each windowed frame undergoes Fast Fourier Transform (typically with $N = 512$ or $N = 1024$ points, zero-padded if necessary). Zero-padding interpolates the spectrum. This provides smoother frequency bins without adding actual frequency resolution (which is determined by the actual data length, not the FFT size). We compute the power spectrum by squaring the magnitude of the complex FFT output:

$$P[k] = \frac{1}{N} |X[k]|^2$$

where:

• $P[k]$: the power spectral density at frequency bin $k$
• $N$: the FFT size (number of points in the Fast Fourier Transform)
• $|X[k]|^2$: the squared magnitude of the complex FFT output at bin $k$
• $k$: the frequency bin index, ranging from $0$ to $N/2$

For $k = 0$ to $N/2$, the remaining bins are redundant for real signals due to conjugate symmetry. This yields $N/2 + 1$ unique frequency bins spanning 0 to $f_s/2$ Hz. The power spectrum represents the distribution of energy across frequencies for that brief moment in time.

The FFT is computationally efficient due to the Fast Fourier Transform algorithm, which reduces the complexity of computing the Discrete Fourier Transform from $O(N^2)$ to $O(N \log N)$. For typical frame sizes and batch processing, this makes spectrogram computation a small fraction of the total processing time compared to the neural network forward pass.

The full preprocessing pipeline turns raw audio into model-ready features through a sequence of operations that progressively extract and compress perceptually relevant information:

1. Pre-emphasis: $x'[n] = x[n] - 0.97x[n-1]$: Equalizes the spectrum by boosting high frequencies
2. Framing: Extract overlapping 25ms frames every 10ms: Segments the signal into quasi-stationary chunks
3. Windowing: Apply Hamming window to each frame: Tapers edges to prevent spectral leakage
4. FFT: Compute 512-point FFT, extract magnitude: Transforms to frequency domain
5. Power spectrum: Square magnitudes: Converts to energy representation
6. Mel filtering: Apply 80 triangular mel filters: Aggregates linear frequencies into perceptual bands
7. Log compression: Apply natural log with $\epsilon$: Matches human loudness perception and compresses dynamic range

The output is a 2D tensor of shape $(T, M)$ where $T$ is the number of time frames (100 per second for 10ms shift) and $M$ is the number of mel bins (typically 80). This represents a compression from 16,000 samples per second to 8,000 values per second (100 frames $\times$ 80 mel bins), a 50-fold reduction in dimensionality while preserving phonetically necessary information.