Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Speech Representations › Code Implementation

Michael BrenndoerferFebruary 15, 202614 min read

Language AI HandbookAll chapters

Part 51 of 67Speech and Audio

Chapter 393Speech Representations › Code Implementation

Reading section 7 of 11 · Code Implementation · 14 minSection 7 of 11 · 14 min

Section 7 of 11 · 14 min read

Speech Representations › Code ImplementationOne section · Sections in this chapter7 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's implement the complete speech preprocessing pipeline using NumPy, SciPy, and Matplotlib. We will create a speech-like sample, compute mel spectrograms, and visualize the representations.

In[6]:

Code

import numpy as np

# Set random seed for reproducibility
np.random.seed(42)

# Generate a synthetic speech-like signal
# In practice, you would load audio from disk and resample it to this rate.
sr = 16000  # Sample rate
duration = 2.0  # seconds
t = np.linspace(0, duration, int(sr * duration))

# Create a synthetic signal with formant-like structure
# Fundamental frequency (pitch) around 150 Hz with harmonics
f0 = 150
signal = np.zeros_like(t)
for harmonic in range(1, 6):
    # Add harmonics with decreasing amplitude (simulating glottal source)
    signal += (0.5**harmonic) * np.sin(2 * np.pi * f0 * harmonic * t)

# Add formant structure (resonances) by filtering
# Simulate two formants at 500 Hz and 1500 Hz
from scipy.signal import butter, lfilter


def add_formant(signal, sr, freq, bandwidth=100):
    """Add a formant resonance using a bandpass filter"""
    nyquist = sr / 2
    low = (freq - bandwidth / 2) / nyquist
    high = (freq + bandwidth / 2) / nyquist
    b, a = butter(2, [low, high], btype="band")
    return lfilter(b, a, signal)


signal = add_formant(signal, sr, 500, 200)
signal = add_formant(signal, sr, 1500, 300)
signal = signal * 0.5  # Normalize amplitude

# Add some noise for realism
signal += 0.01 * np.random.randn(len(signal))

We have created a synthetic speech signal with harmonic structure and formant resonances. In real applications, you would load recorded speech from an audio file and resample it to the target rate, such as 16 kHz for many ASR systems. Now let's examine the raw waveform and compute its spectrographic representations.

Out[7]:

Visualization

Time-domain waveform of a synthetic speech signal over 500 milliseconds showing oscillating amplitude variations with a slowly changing envelope which reflects syllabic structure. — Synthetic speech waveform showing the first 500 milliseconds of a signal constructed with harmonic structure and two formant resonances at 500 Hz and 1500 Hz. The dense oscillations at the fine scale reflect the 150 Hz basic frequency and its harmonics, while the slowly varying amplitude envelope reflects the combined effect of the bandpass formant filters. This structure mimics the voiced speech produced by the human vocal tract.

Sample rate: 16000 Hz
Duration: 2.0 seconds
Total samples: 32000

The waveform shows the oscillating pressure variations characteristic of voiced speech. The dense oscillations represent the high-frequency components, while the amplitude envelope varies slowly according to the syllabic structure. Next, we compute the Short-Time Fourier Transform to examine the frequency content over time.

In[8]:

Code

import numpy as np

# Pre-emphasis
pre_emphasis = 0.97
emphasized_signal = np.append(
    signal[0], signal[1:] - pre_emphasis * signal[:-1]
)

# STFT parameters
n_fft = 512
hop_length = 160  # 10ms at 16kHz
win_length = 400  # 25ms at 16kHz

# Compute STFT using scipy
from scipy import signal as sp_signal

# Create Hamming window
hamming_window = 0.54 - 0.46 * np.cos(
    2 * np.pi * np.arange(win_length) / (win_length - 1)
)

# Compute STFT
frequencies, times, stft_matrix = sp_signal.stft(
    emphasized_signal,
    fs=16000,
    nperseg=win_length,
    noverlap=win_length - hop_length,
    nfft=n_fft,
    window=hamming_window,
    boundary="zeros",
)

magnitude = np.abs(stft_matrix)
power_spectrogram = magnitude**2

# Convert to dB scale for visualization
db_spectrogram = 20 * np.log10(magnitude + 1e-10)
db_spectrogram = db_spectrogram - np.max(
    db_spectrogram
)  # Normalize to max = 0 dB

@fig-preemphasis-effect demonstrates the spectral effect of pre-emphasis on our synthetic speech signal. The filter boosts high-frequency components, flattening the spectral envelope and compensating for the natural roll-off of the glottal source.

Out[9]:

Visualization

Overlay of original and pre-emphasized speech spectra in dB from 0 to 4000 Hz showing the high-frequency boost applied by the alpha=0.97 pre-emphasis filter that compensates for the natural spectral tilt. — Spectral effect of pre-emphasis on the synthetic speech signal. The original spectrum (blue) shows a characteristic roll-off above the basic frequency, with low-frequency components dominating. After applying the first-order pre-emphasis filter with alpha=0.97 (orange), high-frequency energy is boosted by approximately 6 dB per octave, flattening the spectral envelope so that consonant information at 2000-4000 Hz receives comparable representation to the dominant low-frequency vowel energy.

Out[10]:

Visualization

Heatmap spectrogram with time on the x-axis and linear frequency from 0 to 4000 Hz on the y-axis, colored by decibel energy, showing harmonic bands with equal spacing that compress the low-frequency region. — Linear-frequency spectrogram of the synthetic speech signal, with energy shown in decibels across the 0-4000 Hz range over 2 seconds. The horizontal bands represent harmonic energy from the 150 Hz basic and its overtones, equally spaced in Hz. The low-frequency region (below 800 Hz) contains the first formant energy but occupies little vertical space, illustrating why the linear frequency axis is perceptually inefficient for speech analysis.

Spectrogram shape: (257, 201)
Frequency bins: 257
Time frames: 201

The linear spectrogram shows frequency on the vertical axis with equal spacing per Hz. Notice how the harmonic structure appears as horizontal bands, but the low-frequency region (below 2 kHz) appears compressed while high frequencies are stretched. The fine vertical striations indicate the periodic glottal pulses. Now let's compute the mel spectrogram with perceptual frequency scaling.

In[11]:

Code

import numpy as np

# Mel spectrogram parameters
n_mels = 80
f_min = 0
f_max = 8000  # 8 kHz


def hz_to_mel(freq_hz):
    return 2595 * np.log10(1 + freq_hz / 700)


def mel_to_hz(mel):
    return 700 * (10 ** (mel / 2595) - 1)


def create_mel_filterbank(sr, n_fft, n_mels, fmin, fmax):
    mel_points = np.linspace(hz_to_mel(fmin), hz_to_mel(fmax), n_mels + 2)
    hz_points = mel_to_hz(mel_points)
    freqs = np.linspace(0, sr / 2, 1 + n_fft // 2)
    filters = np.zeros((n_mels, len(freqs)))

    for i in range(n_mels):
        left, center, right = hz_points[i : i + 3]
        left_slope = (freqs - left) / (center - left)
        right_slope = (right - freqs) / (right - center)
        filters[i] = np.maximum(0, np.minimum(left_slope, right_slope))

    return filters, freqs


mel_filters, mel_filter_freqs = create_mel_filterbank(
    sr=sr, n_fft=n_fft, n_mels=n_mels, fmin=f_min, fmax=f_max
)
mel_spectrogram = mel_filters @ power_spectrogram

# Convert to log scale (dB)
log_mel_spectrogram = 10 * np.log10(
    np.maximum(mel_spectrogram, 1e-10) / np.max(mel_spectrogram)
)

@fig-pipeline-stages visualizes the transformation of a single speech frame through the preprocessing stages: from the raw waveform through windowing, Fourier analysis, and mel filtering.

Out[12]:

Visualization

Line plot of a raw 25ms speech frame showing oscillating amplitude over time in milliseconds. — Raw 25ms speech frame extracted from around the midpoint of the synthetic signal, showing the oscillating pressure variations before any processing. The signal captures approximately 3.75 complete cycles of the 150 Hz basic frequency within this 25ms window.

Line plot of the Hamming-windowed speech frame overlaid with the window envelope showing tapered edges. — The same speech frame after Hamming windowing. The green curve shows the windowed signal, while the red dashed line shows the window function itself. The edges taper smoothly to zero, eliminating the sharp discontinuities that would otherwise cause spectral leakage artifacts.

Magnitude spectrum in dB of the windowed frame from 0 to about 3000 Hz revealing harmonic peaks. — Linear magnitude spectrum of the windowed frame computed via 512-point FFT, showing energy in dB from 0 to about 3000 Hz. The peaks correspond to the 150 Hz basic and its harmonics (300, 450, 600 Hz), modulated by the two formant resonances at 500 Hz and 1500 Hz.

Bar chart of mel filterbank energies across 80 mel bins showing energy distribution in the perceptual frequency space. — Mel filterbank energies across all 80 mel bins for the same frame, showing the energy distribution in perceptual frequency space. The concentrated energy in the lower mel bins reflects the dominant low-frequency formant structure of the synthetic vowel-like signal.

Out[13]:

Visualization

Heatmap spectrogram with linear frequency axis from 0 to 4000 Hz showing harmonic bands with equal spacing that compress the important low-frequency formant region. — Linear-frequency spectrogram of the synthetic speech signal showing harmonic structure as equally spaced horizontal bands. The vertical axis allocates equal space to each 31 Hz bin, meaning that the phonetically necessary low-frequency region (below 1 kHz, containing the first formant) occupies only a small fraction of the plot while the largely featureless high-frequency region dominates.

Heatmap mel spectrogram with mel frequency axis up to 8000 Hz showing expanded low-frequency resolution and compressed high-frequency region. — Mel-frequency spectrogram of the same signal with 80 perceptually spaced bins. The warped frequency axis expands the low-frequency region, giving the model substantially more representational capacity for the first and second formants while compressing the high-frequency region where fine frequency discrimination matters less for phonetic identity.

The comparison reveals how the mel scale redistributes frequency resolution. In the linear spectrogram, the region below 1 kHz occupies a small portion of the vertical space, yet this is where important phonetic information resides, including the first two formants that distinguish most vowels. The mel spectrogram expands this region, giving the model more parameters to distinguish low-frequency content while compressing the high-frequency region where fine discrimination is less perceptually important.

Now let's implement feature normalization and examine the effect of CMVN.

In[14]:

Code

# Compute statistics for normalization
# Mean and variance across time frames (axis=1)
mean = np.mean(log_mel_spectrogram, axis=1, keepdims=True)
std = np.std(log_mel_spectrogram, axis=1, keepdims=True)

# Apply CMVN
normalized_mel = (log_mel_spectrogram - mean) / (std + 1e-8)

# Alternative: Global normalization (using full spectrogram statistics)
global_mean = np.mean(log_mel_spectrogram)
global_std = np.std(log_mel_spectrogram)
global_normalized = (log_mel_spectrogram - global_mean) / (global_std + 1e-8)

Out[15]:

Visualization

Heatmap of the original log-mel spectrogram across 80 mel bins over 2 seconds showing higher energy in low-frequency bins. — Original log-mel spectrogram before normalization, showing raw spectral energy across 80 mel bins over 2 seconds. Low mel bins (low frequencies) consistently show higher energy than high mel bins, which reflects the dominance of low-frequency energy in the synthetic signal. This imbalance, if fed directly to a neural network, would cause low-frequency features to disproportionately influence learned representations.

Heatmap of the CMVN-normalized mel spectrogram with standardized energy distribution showing enhanced contrast across frequency bands. — Mel spectrogram after utterance-level CMVN, standardizing each mel bin to zero mean and unit variance across the utterance. The normalization removes the systematic energy imbalance between frequency bands, enhancing contrast in mid-frequency regions and so all mel bins contribute comparably to the model's learning. The temporal structure is preserved while amplitude offsets are removed.

Original range: [-55.49, 0.00]
Normalized range: [-5.60, 3.04]
Normalized mean: -0.0000, std: 1.0000

Normalization centers the features and scales them to comparable ranges across frequency bands. This prevents high-energy low-frequency bands from dominating the learning process and makes the model more reliable to variations in recording volume and microphone characteristics. The normalized spectrogram shows enhanced contrast in mid-frequency regions that might have been overshadowed by strong low-frequency energy in the original.

Finally, let's examine the mel filterbank itself to understand how frequencies are aggregated.

In[16]:

Code

# Frequency axis for plotting
freqs = mel_filter_freqs

Out[17]:

Visualization

Overlay of every fifth triangular mel filter plotted as filter weight versus frequency from 0 to 8000 Hz, showing narrow filters densely packed at low frequencies and wide filters sparsely spaced at high frequencies. — Mel filterbank visualization showing every fifth filter from a bank of 80 overlapping triangular filters. At low frequencies (left side), the filters are narrow and closely spaced in Hz, giving fine frequency resolution for the first and second formants necessary for vowel discrimination. At high frequencies (right side), the filters are wide and sparsely spaced, aggregating energy across large frequency ranges to match the reduced perceptual sensitivity of the human ear above 2 kHz. Each filter color represents a different filter index, cycling through the default color scheme.

Filterbank shape: (80, 257)
Number of filters: 80
FFT bins: 257
First filter center: ~31 Hz
Last filter center: ~7719 Hz

The filterbank visualization demonstrates the non-linear spacing. The leftmost filters are narrow and steep. This provides fine resolution in the low-frequency range where formant information is necessary. The rightmost filters are broad and gentle, aggregating energy across wide swaths of high frequencies where precise discrimination is less perceptually important. This allocation reflects the frequency resolution of the human ear, optimizing the representation for speech perception.