Speech Representations › Feature Normalization

Raw mel spectrograms vary between speakers and utterances, especially when recording conditions change. Differences in microphone distance, room acoustics, and speaking volume create variations that obscure the linguistic content. Feature normalization stabilizes training and improves generalization by removing non-linguistic variation, letting the model to focus on phonetic patterns rather than recording artifacts.

To understand why normalization matters so much for speech, consider what happens without it. A speaker 30 cm from the microphone produces far more energy than one at 1 meter, shifting the entire spectrogram up or down uniformly. A phone conversation passes through bandwidth-limiting filters that attenuate both low and high frequencies. Background noise raises the energy floor in all frequency bins. These effects are consistent within a recording but vary dramatically between recordings, creating a distribution mismatch that degrades model performance on new recordings that differ in acoustic conditions from the training data.

Cepstral Mean and Variance Normalization (CMVN) standardizes features to zero mean and unit variance, effectively removing channel characteristics and volume differences. For a sequence of feature vectors $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_T$, we compute:

$$\boldsymbol{\mu} = \frac{1}{T} \sum_{t=1}^{T} \mathbf{x}_t$$ $$\boldsymbol{\sigma}^2 = \frac{1}{T} \sum_{t=1}^{T} (\mathbf{x}_t - \boldsymbol{\mu})^2$$ $$\hat{\mathbf{x}}_t = \frac{\mathbf{x}_t - \boldsymbol{\mu}}{\sqrt{\boldsymbol{\sigma}^2 + \epsilon}}$$

where:

• $\boldsymbol{\mu}$: the mean feature vector computed across all $T$ time frames, representing the average spectral characteristics
• $\boldsymbol{\sigma}^2$: the variance vector (element-wise variance across frequency bins), measuring the spread of feature values
• $\hat{\mathbf{x}}_t$: the normalized feature vector at time $t$, with zero mean and unit variance (z-score normalization)
• $\mathbf{x}_t$: the original input feature vector (e.g., log-mel spectrogram frame) at time $t$
• $T$: the total number of time frames in the utterance or normalization window
• $\epsilon$: a small constant added to the variance for numerical stability, preventing division by zero when features have near-zero variance
• The square root in the denominator converts variance to standard deviation for proper normalization scaling

The mean subtraction step removes additive channel effects. If a microphone consistently boosts a frequency band by a fixed amount, that offset will appear in the mean and be subtracted away. The variance normalization step addresses multiplicative effects: if one speaker consistently has higher-energy speech than another, dividing by the standard deviation brings all frequency bands to comparable scale.

CMVN can be applied per utterance (using statistics from the current speech segment) or globally (using statistics from the training corpus). Utterance-level CMVN handles channel and speaker variation effectively but requires the entire utterance before processing, introducing latency unsuitable for streaming recognition where decisions must be made with minimal delay. Global CMVN enables frame-by-frame processing but is less adaptive to specific recording conditions or speakers not represented in the training data.

For online recognition systems that process live audio, running estimates of mean and variance can be maintained and updated incrementally using exponential moving averages, approximating the full-sequence statistics without requiring future context. This approach effectively estimates the "current channel conditions" as audio streams in, adapting to changes in acoustic environment within a single session.

While static mel spectrograms capture spectral content at each frame, they do not explicitly represent how spectra change over time. Temporal dynamics are important for phonetic discrimination: vowels have formant transitions (moving resonances), and consonants have rapid spectral changes (bursts, friction noise). A recording of the vowel "ee" and a recording of "oo" differ primarily in their static spectral patterns. But the difference between the word "bad" and "bid" depends partly on the formant trajectory during the vowel, including the direction and rate of formant movement rather than only its static endpoint.

Delta features approximate the first derivative (velocity) of the feature trajectory, showing how quickly the spectrum is changing:

$$d_t = \frac{\sum_{n=1}^{N} n \cdot (c_{t+n} - c_{t-n})}{2\sum_{n=1}^{N} n^2}$$

where:

• $d_t$: the delta feature (first derivative approximation) at time $t$, representing the rate of change of the spectral envelope
• $c_t$: the static feature value (e.g., a single mel coefficient) at time $t$
• $c_{t+n}$: the static feature value $n$ frames ahead of time $t$ (future context)
• $c_{t-n}$: the static feature value $n$ frames before time $t$ (past context)
• $N$: the regression window size (typically 2), meaning the computation uses a 5-frame window ($t-2$ to $t+2$)
• The numerator computes a weighted sum of differences, giving more weight to frames further from the center (linear weighting by $n$)
• The denominator normalizes by the sum of squared weights ($2\sum n^2 = 2(1^2 + 2^2) = 10$ for $N=2$) to produce an unbiased estimate of the derivative

Delta-delta features (acceleration) are computed by applying the same formula to the delta features, capturing how the rate of change itself is changing. This second derivative is sensitive to the inflection points in spectral trajectories, useful for detecting the onset of formant transitions or the release of stop consonants.

Concatenating static, delta, and delta-delta features triples the feature dimension but provides explicit temporal context. Modern deep learning systems with large receptive fields (convolutional layers or long attention windows) often learn these temporal patterns implicitly from static features, making explicit delta features less common than in classical GMM-HMM systems. However, they remain useful for resource-constrained models or as auxiliary inputs that explicitly guide the model toward dynamic patterns.

In noisy environments, simple normalization may not suffice. Additive background noise raises the energy floor across frequencies, masking weak speech components and distorting spectral patterns. A classic example is the cocktail party problem: background conversations at comparable volume to the target speaker create spectral interference that appears across all frequency bands, making CMVN ineffective since the noise shifts the mean uniformly.

Spectral subtraction estimates the noise spectrum during non-speech segments (silence periods detected by voice activity detection) and subtracts it from the signal:

$$|\hat{X}[k]|^2 = |Y[k]|^2 - \alpha \cdot |N[k]|^2$$

where:

• $|\hat{X}[k]|^2$: the estimated clean signal power at frequency bin $k$ after noise removal
• $|Y[k]|^2$: the observed noisy signal power at frequency bin $k$, containing both speech and noise components
• $|N[k]|^2$: the estimated noise power at frequency bin $k$, typically measured during non-speech segments (silence)
• $\alpha$: the over-subtraction factor (typically greater than 1, such as 1.5 to 3.0), which scales the noise estimate to account for noise variability and prevent musical noise artifacts
• $k$: the frequency bin index across the spectrum

The over-subtraction factor $\alpha > 1$ accounts for the fact that noise varies over time; simply subtracting the average noise estimate leaves residual noise during high-noise frames. However, aggressive subtraction can introduce "musical noise," artifactual tones resulting from isolated spectral peaks that appear when most frequency bins are suppressed but a few retain significant energy. This musical noise is perceptually distracting and can confuse recognition systems. More sophisticated techniques like Wiener filtering or minimum mean-square error spectral estimation handle this more gracefully by adapting the subtraction amount per-bin based on local signal-to-noise estimates.

Vocal Tract Length Normalization (VTLN) addresses a different form of variability: anatomical differences between speakers. A child's shorter vocal tract produces formants at higher frequencies than an adult's longer tract. VTLN warps the frequency axis of the spectrogram by a speaker-specific factor $\alpha$, effectively normalizing for vocal tract length differences. Classical systems estimated this warp factor from the speech itself; modern deep learning systems often learn speaker normalization implicitly through multi-speaker training or explicit speaker embedding conditioning.

Deep learning has largely absorbed these preprocessing functions into learned representations that are reliable to such variations, but understanding the signal processing foundations helps you diagnose failures when models struggle with specific acoustic conditions, design targeted augmentation strategies, and adapt pretrained models to challenging environments.