Text-to-Speech › Evaluation Metrics for TTS

Evaluating speech synthesis presents unique challenges. Unlike text generation where we can compare against reference strings using BLEU or ROUGE (covered in Part LIII), audio quality involves perceptual dimensions that resist simple mathematical characterization. Naturalness depends on subtle cues like breathiness, vocal effort, and prosodic appropriateness that are difficult to quantify with simple distance metrics.

The gold standard for TTS evaluation remains human judgment, typically quantified through Mean Opinion Score (MOS) studies. In these studies, listeners rate synthesized speech on a 5-point scale:

1. Bad (unintelligible, unnatural)
2. Poor (intelligible but annoying)
3. Fair (slightly annoying)
4. Good (perceptible distortion, not annoying)
5. Excellent (imperceptible distortion)

CMOS (Comparative Mean Opinion Score) presents listeners with paired samples (A and B) and asks which is better. This gives relative quality assessments useful for comparing system variants without requiring absolute calibration.

While MOS gives the most reliable quality assessment, it is expensive, time-consuming, and suffers from variability across listener populations and testing conditions. Different demographic groups may have different expectations for synthetic speech, and laboratory conditions may not reflect real-world usage scenarios. Despite these limitations, MOS remains needed for final quality validation, particularly for commercial systems.

Objective metrics attempt to approximate human perception algorithmically, giving faster and cheaper evaluation during development:

Mel Cepstral Distortion (MCD) measures the Euclidean distance between mel-frequency cepstral coefficients (MFCCs) of synthesized and reference audio. MFCCs capture the spectral envelope in a compact representation inspired by human auditory processing. Lower MCD indicates better spectral envelope matching:

$$\text{MCD} = \frac{10}{\ln 10} \sqrt{2 \sum_{d=1}^{D} (c_d - \hat{c}_d)^2}$$

where:

• $\text{MCD}$: the Mel Cepstral Distortion measuring spectral difference between synthesized and reference audio
• $D$: the total number of cepstral coefficients (dimensions) used in the comparison
• $d$: the index iterating over cepstral coefficients from $1$ to $D$
• $c_d$: the $d$-th MFCC coefficient of the reference (ground truth) audio
• $\hat{c}_d$: the $d$-th MFCC coefficient of the synthesized audio
• $\frac{10}{\ln 10}$: a scaling constant (approximately 4.3429) converting natural logarithm units to decibels

However, MCD correlates poorly with human perception of naturalness as it penalizes all spectral deviations equally regardless of perceptual salience. Some spectral differences are inaudible to humans, while others are highly noticeable, but MCD treats them identically.

PESQ (Perceptual Evaluation of Speech Quality) and STOI (Short-Time Objective Intelligibility) predict subjective quality and intelligibility respectively. PESQ models human auditory perception by comparing internal representations of reference and degraded speech, accounting for frequency masking and loudness perception. STOI focuses specifically on intelligibility, correlating highly with word recognition rates in noisy conditions, which makes it useful for applications where clarity is the primary requirement.

Fundamental Frequency (F0) metrics assess prosody quality by comparing pitch contours between synthesized and reference speech. Metrics like F0 Frame Error (FFE) measure the percentage of frames where the predicted pitch differs materially from the reference, helping to quantify whether the model has learned appropriate intonation patterns.

As we discussed in Embedding Evaluation, TTS evaluation can be intrinsic (measuring acoustic quality directly) or extrinsic (measuring performance on downstream tasks). Word Error Rate (WER) from speech recognition systems gives an extrinsic metric: if a speech recognizer transcribes synthesized speech accurately, the TTS system likely produces intelligible, natural-sounding output. This "TTS-for-ASR" evaluation proves particularly useful for evaluating low-resource languages where human MOS studies are impractical or for rapidly iterating on system designs without organizing listening tests.