Text-to-Speech › Code Implementation: Building a TTS Pipeline
Language AI HandbookAll chapters Text-to-Speech › Code Implementation: Building a TTS PipelineOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement a complete neural TTS pipeline using modern libraries. We will use a pre-trained model from the transformers library to generate speech, then examine the intermediate representations to understand the acoustic modeling and vocoding stages.
# Install required packages
# !uv pip install transformers torch torchaudio soundfile librosa matplotlib numpy textalloc
import warnings
import torch
from transformers import set_seed
warnings.filterwarnings("ignore")
# Set seed for reproducibility
set_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"Using device: cpu
We use the VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) model, which combines acoustic modeling and vocoding into a single end-to-end architecture. Unlike the two-stage pipeline discussed earlier, VITS uses a variational autoencoder with adversarial training to map text directly to waveform, though internally it still operates on latent representations analogous to spectrograms.
## Load pre-trained VITS model and tokenizer
## VITS is an end-to-end model that combines acoustic features and vocoding
from transformers import VitsModel, VitsTokenizer
model_name = "facebook/mms-tts-eng"
tokenizer = VitsTokenizer.from_pretrained(model_name)
model = VitsModel.from_pretrained(model_name).to(device)
model.eval()Model loaded: facebook/mms-tts-eng Vocabulary size: 38
Now we process text through the pipeline. The tokenizer converts text to phoneme IDs using a G2P (grapheme-to-phoneme) model internally, handling normalization and pronunciation automatically.
# Input text for synthesis
text = "Text to speech synthesis bridges the gap between written language and spoken communication."
# Tokenize input
inputs = tokenizer(text, return_tensors="pt")
input_ids = inputs["input_ids"].to(device)Input text: 'Text to speech synthesis bridges the gap between written language and spoken communication.' Token IDs shape: torch.Size([1, 181]) Token IDs: [0, 33, 0, 7, 0, 31, 0, 33, 0, 19, 0, 33, 0, 22, 0, 19, 0, 8, 0, 13]...
With tokenized inputs, we generate speech. The model outputs a waveform directly, but we can also extract intermediate features to visualize the acoustic modeling process.
# Generate speech
with torch.no_grad():
outputs = model(input_ids)
waveform = outputs.waveform[0].cpu().numpy()
# Extract intermediate spectrogram if available (model-dependent)
# VITS internally uses latent representations, but we can compute mel-spec from outputGenerated waveform shape: (102656,) Duration: 6.42 seconds Sampling rate: 16000 Hz
The generated waveform spans the calculated duration of audio at the model's sampling rate. This gives enough temporal resolution for clear speech intelligibility while maintaining reasonable computational efficiency.
The waveform displays the characteristic amplitude modulation of speech, with higher amplitude regions corresponding to voiced segments (vowels) and lower amplitude regions to consonants and pauses. Unlike the square waves of synthetic beeps, natural speech exhibits complex harmonic structures with gradual transitions between sounds.
Now let's compute and visualize the mel-spectrogram to examine the frequency content over time. This is what an acoustic model would output in a two-stage system.
def hz_to_mel(freq_hz):
return 2595 * np.log10(1 + freq_hz / 700)
def mel_to_hz(mel):
return 700 * (10 ** (mel / 2595) - 1)
def mel_filter_bank(sr, n_fft, n_mels, fmin=0, fmax=8000):
"""Create triangular mel filters without an external audio package."""
mel_points = np.linspace(hz_to_mel(fmin), hz_to_mel(fmax), n_mels + 2)
hz_points = mel_to_hz(mel_points)
bins = np.floor((n_fft + 1) * hz_points / sr).astype(int)
filters = np.zeros((n_mels, n_fft // 2 + 1))
for m in range(1, n_mels + 1):
left, center, right = bins[m - 1], bins[m], bins[m + 1]
if center > left:
filters[m - 1, left:center] = (np.arange(left, center) - left) / (
center - left
)
if right > center:
filters[m - 1, center:right] = (
right - np.arange(center, right)
) / (right - center)
return filters
def stft_power(signal, n_fft=1024, hop_length=256):
"""Compute a power spectrogram with NumPy."""
window = np.hanning(n_fft)
frames = []
for start in range(0, max(len(signal) - n_fft + 1, 1), hop_length):
frame = np.zeros(n_fft)
chunk = signal[start : start + n_fft]
frame[: len(chunk)] = chunk
spectrum = np.fft.rfft(frame * window)
frames.append(np.abs(spectrum) ** 2)
return np.array(frames).T
## Compute mel-spectrogram from the generated waveform
## This simulates the intermediate representation in a two-stage TTS system
n_fft = 1024
hop_length = 256
n_mels = 80
power_spec = stft_power(waveform, n_fft=n_fft, hop_length=hop_length)
mel_filters = mel_filter_bank(
model.config.sampling_rate, n_fft=n_fft, n_mels=n_mels, fmax=8000
)
mel_spec = np.maximum(mel_filters @ power_spec, 1e-10)
## Convert to log scale (dB)
mel_spec_db = 10 * np.log10(mel_spec / np.max(mel_spec))Mel-spectrogram shape: (80, 398) Time frames: 398 Frequency bins: 80
This compression to 80 frequency bins and the computed number of time frames reaches approximately a 200:1 reduction compared to raw audio samples, retaining the needed spectral envelope while discarding phase information that the vocoder will reconstruct.
The mel-spectrogram reveals the spectral envelope of the speech. The horizontal dark bands stand for formants: resonant frequencies of the vocal tract that distinguish different vowel sounds. The vertical striations indicate pitch periods: the basic frequency of vocal cord vibration. The smoothness and continuity of these patterns indicate high-quality synthesis without the buzziness or metallic artifacts characteristic of older vocoders.
Let's save the audio and examine the attention alignment that the model learned between text and audio (if available in this architecture).
import wave
# Save the generated audio
output_path = "tts_output.wav"
audio_int16 = np.int16(np.clip(waveform, -1.0, 1.0) * 32767)
with wave.open(output_path, "wb") as wav_file:
wav_file.setnchannels(1)
wav_file.setsampwidth(2)
wav_file.setframerate(model.config.sampling_rate)
wav_file.writeframes(audio_int16.tobytes())
# Note: VITS uses stochastic duration prediction internally rather than explicit attention
# For visualization purposes, we'll simulate an alignment matrix showing how text
# positions map to time steps (conceptually similar to Tacotron attention)Audio saved to: tts_output.wav
The synthesized WAV file is now ready for subjective listening evaluation or objective quality assessment using metrics such as MOS or PESQ discussed in the evaluation section.
# Create a simulated alignment matrix to illustrate the concept
# In real Tacotron models, this would be the attention weights
text_length = min(len(text), 50) # Cap for visualization
time_steps = mel_spec_db.shape[1]
# Create monotonic alignment with varying durations (simulating different phoneme lengths)
alignment = np.zeros((text_length, time_steps))
duration_weights = 1.0 + 0.45 * np.sin(np.arange(text_length) * 0.7) ** 2
frame_edges = np.rint(
np.concatenate(([0], np.cumsum(duration_weights)))
/ duration_weights.sum()
* time_steps
).astype(int)
for i, (start, end) in enumerate(zip(frame_edges[:-1], frame_edges[1:])):
if start < end:
alignment[i, start:end] = 1.0
if end < time_steps:
alignment[i, end : min(end + 2, time_steps)] = np.linspace(
0.45, 0.0, min(2, time_steps - end)
)
The alignment visualization shows the monotonic left-to-right attention required for TTS. Unlike machine translation where attention might jump between distant words, TTS attention must progress steadily through the text. The varying widths of the attention bands indicate different phoneme durations: consonants typically occupy fewer frames than vowels.
Finally, let's compare the spectral characteristics of our synthesized speech against what would be expected in natural speech by examining the basic frequency (pitch) contour.
# Estimate pitch with frame-wise autocorrelation in the human speech range.
n_fft = 1024
hop_length = 256
sampling_rate = model.config.sampling_rate
frame_starts = range(0, max(len(waveform) - n_fft + 1, 1), hop_length)
frames = []
for start in frame_starts:
frame = np.zeros(n_fft)
chunk = waveform[start : start + n_fft]
frame[: len(chunk)] = chunk
frames.append(frame)
frame_rms = np.array([np.sqrt(np.mean(frame**2)) for frame in frames])
energy_threshold = 0.08 * frame_rms.max()
min_lag = int(sampling_rate / 300)
max_lag = int(sampling_rate / 75)
pitch_values = np.full(len(frames), np.nan)
for t, frame in enumerate(frames):
if frame_rms[t] < energy_threshold:
continue
centered = (frame - frame.mean()) * np.hanning(n_fft)
autocorr = np.correlate(centered, centered, mode="full")[n_fft - 1 :]
if autocorr[0] <= 0:
continue
lag = min_lag + np.argmax(autocorr[min_lag : max_lag + 1])
if autocorr[lag] / autocorr[0] >= 0.25:
pitch_values[t] = sampling_rate / lag
# Suppress isolated octave errors without filling unvoiced gaps.
smoothed_pitch = pitch_values.copy()
for t in range(len(pitch_values)):
if np.isfinite(pitch_values[t]):
neighborhood = pitch_values[max(0, t - 2) : t + 3]
smoothed_pitch[t] = np.nanmedian(neighborhood)
# A single-speaker utterance has a stable pitch register. Fold estimates that
# land well above that register down one octave, a common autocorrelation error.
pitch_values = smoothed_pitch.copy()
speaker_register = np.nanmedian(pitch_values)
octave_doubled = pitch_values > 1.55 * speaker_register
pitch_values[octave_doubled] /= 2
# Smooth within voiced segments once more, without bridging silent gaps.
smoothed_pitch = pitch_values.copy()
for t in range(len(pitch_values)):
if np.isfinite(pitch_values[t]):
neighborhood = pitch_values[max(0, t - 2) : t + 3]
smoothed_pitch[t] = np.nanmedian(neighborhood)
pitch_values = smoothed_pitch
time_axis = (
np.arange(len(pitch_values)) * hop_length / model.config.sampling_rate
)
The pitch contour reveals the prosodic patterns of the synthesized utterance. Natural speech exhibits continuous variation in basic frequency (F0) to convey emphasis and emotion as well as syntactic structure. The smoothness of this contour indicates that the model has learned appropriate prosody rather than generating robotic, monotone speech.