Speech-Language Integration › Training Strategies for Speech LLMs

The first stage freezes both the speech encoder and the LLM, training only the connector (projection layer, perceiver, or Q-former). Using paired speech-text data, typically ASR training sets, the objective teaches the connector to map acoustic sequences to their textual equivalents.

For a speech encoder creating features $\mathbf{A}$ and text transcript with token embeddings $\mathbf{T}$, the alignment loss is:

$$\mathcal{L}_{\text{align}} = -\sum_{i} \log P(t_i \mid \mathbf{A}, t_{<i}; \theta_{\text{connector}})$$

where:

• $\mathcal{L}_{\text{align}}$: the negative log-likelihood loss measuring alignment quality
• $t_i$: the $i$-th text token in the target transcript
• $\mathbf{A}$: the audio features from the frozen speech encoder
• $t_{<i}$: the sequence of previous tokens $(t_1, t_2, \ldots, t_{i-1})$ used for autoregressive prediction
• $\theta_{\text{connector}}$: the trainable parameters of the connector (projection layer)
• $P(t_i \mid \cdot)$: the probability assigned by the LLM to token $t_i$ given audio context and previous tokens

This objective teaches the connector to map acoustic sequences such that the frozen LLM assigns high probability to the correct transcript tokens, effectively translating speech into the LLM's semantic space without updating the base models. The LLM parameters remain frozen; only $\theta_{\text{connector}}$ updates. This stage assumes the LLM already understands text; the connector merely translates speech into "LLM-readable" form.

Data requirements at this stage are substantial: thousands to tens of thousands of hours of speech with transcripts. However, since the base models are frozen, computation remains manageable compared to full fine-tuning. The quality of this alignment determines the ceiling for downstream performance: if the connector fails to map the word "weather" to the acoustic pattern for "weather", no amount of subsequent training will fix this basic misalignment.

An important subtlety of Stage 1 is the choice of training data. Simply maximizing transcription accuracy on a single ASR dataset risks over-fitting the connector to that particular recording style, microphone quality, and speaker distribution. Systems that perform well across diverse conditions typically use multi-condition ASR data spanning telephone speech, meeting recordings, broadcast audio, and spontaneous conversation. This diversity forces the connector to learn representations that are invariant to acoustic conditions rather than representations that merely match a specific studio recording signature.

After alignment, the model undergoes large-scale multimodal pre-training where both the connector and the LLM (or LoRA adapters on the LLM) update. The data mixture expands beyond ASR to include:

• Speech continuation (audio to future audio tokens)
• Speech-to-text translation
• Text-to-speech synthesis
• Speech-based question answering

This stage requires high-quality interleaved speech-text data, often sourced from audiobooks (aligned text and speech), podcasts with transcripts, and synthetic data generated by high-quality TTS systems. The mixture ratio matters: too much ASR biases the model toward transcription; too much generation degrades comprehension.

The curriculum during this stage often progresses from simpler to harder tasks. Early in this phase, the model might focus on speech-to-text tasks where the mapping is relatively direct. As training progresses, more weight is given to open-ended generation and question answering tasks that require the model to interpret audio content and formulate novel responses rather than simply transcribe. This staged curriculum prevents the model from overfitting to the most abundant but least cognitively demanding ASR data.

Another necessary consideration is catastrophic forgetting, a phenomenon we examined in Part XXXIV: Fine-tuning Fundamentals. When the LLM parameters update during Stage 2, the model may gradually lose its text-only capabilities as it adapts to the speech domain. Practitioners address this by including text-only examples in the Stage 2 mixture. This keeps the model continues to see and predict pure text sequences. This "replay" strategy maintains text performance while expanding to multimodal capabilities.

The final stage adapts the model to follow instructions and engage in dialogue, mirroring the instruction tuning process for text-only LLMs discussed in Part XXXVI: Instruction Tuning. Speech instruction datasets contain tuples of (audio instruction, audio/text response) or (text instruction, audio response).

Critically, instruction tuning for speech LLMs must handle turn-taking in conversations. Unlike text where <USER> and <ASSISTANT> tokens suffice, speech conversations involve acoustic cues: pauses, interruptions, backchannels ("uh-huh", "mm-hmm"). Advanced models learn to generate these paralinguistic behaviors, requiring training data that captures natural conversational flow.

The instruction templates for speech models must also account for the continuous nature of audio. While text instructions have clear boundaries, speech instructions might trail off, overlap with other sounds, or contain disfluencies. The model must learn to detect when a user has finished speaking and when to begin its response, a timing problem that does not exist in text-based chat. Special tokens or learned end-of-utterance detectors help manage these turn-taking transitions.

Instruction data quality matters more at this stage than at earlier stages. Whereas Stage 1 tolerates noisy ASR data (the objective is just to align modalities), Stage 3 data shapes the model's conversational behavior, safety responses, and helpfulness. The same principles that govern RLHF and preference learning for text LLMs apply here: diverse, high-quality human demonstrations produce better conversational agents than large quantities of automatically generated examples.