Speech-Language Integration: Connecting Audio to LLMs

Speech and text represent the same underlying linguistic content through fundamentally different physical manifestations. Text exists as discrete symbolic sequences, naturally compatible with the token-based processing of large language models. Speech, by contrast, is a continuous acoustic waveform, a high-dimensional signal varying over time, with frequency and amplitude as its other dimensions. Bridging this gap requires architectures that can translate between the continuous acoustic domain and the discrete linguistic domain without losing prosody and other nuanced paralinguistic information that speech conveys.

In prior chapters, we explored how models like Whisper convert speech to text through encoder-decoder architectures, effectively collapsing the acoustic signal into a sequence of text tokens. While powerful, this approach discards information. The timbre of a voice, the hesitation in a pause, and the emotion carried in intonation vanish when audio becomes text. Speech-language integration seeks to create unified models that understand and generate speech directly, treating acoustic information as a first-class modality alongside text.

Think about what happens when a friend tells you "sure, that's fine" in a flat, resigned voice versus an enthusiastic one. The words are identical, but any human immediately grasps the difference in meaning. A cascaded system that transcribes speech to text loses this distinction entirely before the language model ever sees it. This chapter is about building systems that keep the acoustic signal alive through the reasoning process, rather than discarding it at the first opportunity.

This chapter examines the architectures that bind speech encoders to large language models, the methods for representing audio as discrete tokens, and the spectrum of approaches from cascaded pipelines to fully end-to-end speech LLMs. We will see how the field has evolved from connecting separate systems to training unified models that think in both sound and text.

The basic challenge in speech-language integration stems from the representational chasm between audio and text. Text tokenizers, as we explored in Part V: Subword Tokenization, map vocabulary to discrete integer indices. A word or subword becomes a single token ID that embeds into a high-dimensional vector space. Audio, however, is sampled at rates between 16,000 and 48,000 samples per second. Even a single second of speech contains thousands of continuous values.

This disparity creates a basic mismatch in how these modalities represent information. Where text provides a compressed, symbolic encoding of semantic content, audio provides a dense, physical measurement of pressure waves. The transformation between them goes beyond format conversion and requires interpreting physical vibrations as linguistic meaning, a process that involves multiple levels of abstraction from acoustic phonetics to semantic understanding.

This density difference creates architectural pressure. If we feed raw audio features directly into a transformer, the sequence length becomes prohibitively long. The quadratic complexity of self-attention, discussed in Part XVII: Efficient Attention, means that processing even a minute of audio as individual frames would require computing attention over 3,000+ tokens, which makes it computationally expensive and memory intensive. The computational burden scales with the square of sequence length, meaning that doubling the audio duration quadruples the attention computation cost.

Beyond length, the semantic gap poses a deeper problem. Text tokens carry linguistic meaning directly; the embedding for "king" relates to monarchy and power. Audio features initially carry only acoustic information: frequency bands, harmonic structures, and temporal dynamics. The meaning emerges only after layers of processing. Connecting these modalities requires mechanisms that align acoustic patterns with linguistic concepts, effectively teaching the LLM to "hear" rather than merely "read transcripts."

This alignment challenge is complicated by the many-to-one nature of the speech-to-text mapping. The same sentence spoken by different voices, in different accents, or in different acoustic environments produces wildly different waveform patterns, yet maps to identical text. A reliable integration must therefore be invariant to speaker characteristics and noise while remaining sensitive to paralinguistic cues such as emotion and emphasis that text alone cannot capture.

The inverse problem is equally important: speech generation from text involves the one-to-many mapping. Any text string can be spoken with countless combinations of pitch, rate, rhythm, and timbre. When a system generates the phrase "I understand," it must choose the words and the entire acoustic realization. That choice carries meaning. Generated speech that ignores this degrades to a monotone, robotic quality that users find alienating even when the words are correct.

You might wonder why we cannot simply feed mel-spectrogram values directly into a transformer as if they were text tokens. The answer involves both practical and conceptual obstacles.

Practically, a mel-spectrogram for one second of speech at 16 kHz, processed with a standard hop length of 160 samples, produces 100 frames, each containing 80 frequency values. The resulting 8,000 floating-point numbers per second dwarf the roughly 3-4 tokens per second of natural speech. A 30-second utterance would produce 240,000 raw spectrogram values, far beyond the context window of any current LLM and an enormous waste of capacity on representing frequency bands that carry no semantic information.

Beyond the practical length issue lies a conceptual mismatch. The transformer's self-attention mechanism learns to relate tokens based on semantic similarity and co-occurrence patterns in training data. A text LLM learned that "cat" and "feline" appear in similar contexts, that questions tend to precede answers, and that proper nouns often follow definite articles. None of this statistical structure exists in raw spectrogram frames. Frame 47 of a mel-spectrogram has no inherent relationship to frame 48 that a pre-trained text LLM can use. The model would need to learn the entire structure of acoustic phonetics from scratch during fine-tuning, essentially relearning what dedicated speech encoders already know.

This is why pre-trained speech encoders are so valuable. Encoders like Whisper or HuBERT have already learned the acoustic structure of language. They know that certain frequency patterns correspond to vowels, that certain temporal transitions correspond to consonant releases, and that certain energy envelopes correspond to stressed syllables. By the time their representations reach a projection layer, the information has been compressed into a format where semantic similarity corresponds more closely to geometric similarity in the embedding space. The bridge between these representations and a text LLM's embedding space is far shorter than the bridge from raw spectrograms.

Speech-language models occupy a spectrum of integration depth. At one extreme, completely separate systems handle speech recognition, language understanding, and speech synthesis. At the other, a single neural network processes audio embeddings directly and generates audio tokens autoregressively. Understanding this spectrum clarifies the tradeoffs between implementation complexity, data requirements, and capability.

The choice of where to sit on this spectrum depends on the specific application requirements. Systems requiring high reliability may prefer the debuggability of cascaded approaches, while applications demanding natural conversation may require the unified reasoning of end-to-end models. Intermediate approaches attempt to balance these competing needs, maintaining some separation of concerns while letting for richer information flow between components.

The simplest approach chains three independent models: an automatic speech recognition (ASR) system converts audio to text, a text-based LLM processes the transcript and generates a response, and a text-to-speech (TTS) system vocalizes the output. This architecture, which we might call ASR-LLM-TTS, has significant practical advantages. Each component can be optimized independently, replaced or upgraded without retraining others, and debugged in isolation. When the system makes an error, you can examine the ASR output to determine whether the mistake occurred during transcription or during language understanding.

The modular design also means that you can take a state-of-the-art text LLM and immediately give it voice capabilities by wrapping it with ASR and TTS. This is why cascaded systems dominated commercial voice assistants for years: they allow organizations to swap in better ASR models as they become available, or to use the same LLM backend for both text chat and voice interaction without any architectural change.

However, the cascaded approach suffers from error propagation and information loss. ASR errors, such as misrecognizing "write" as "right" or failing on accented speech, propagate irreversibly into the LLM. Once speech becomes text, the uncertainty about what was said disappears; the LLM receives a definitive transcript even when the ASR was only 60% confident. Prosodic information, emotion, speaker characteristics, and background sounds disappear during transcription. The LLM receives only the words, not how they were spoken. For many applications, such as emotional support agents, musical analysis, or sound event understanding, this loss is unacceptable.

The latency of cascaded systems is additive: the total response time includes the full ASR processing, LLM generation, and TTS synthesis. Each stage must complete before the next begins, creating pauses in conversation that feel unnatural to human interlocutors. The modular independence that makes cascaded systems easy to maintain becomes a liability when trying to optimize for real-time interaction. Research has repeatedly shown that pauses longer than around 500 milliseconds create noticeable conversational awkwardness, and cascaded systems rarely achieve this threshold for non-trivial responses.

Moving toward deeper integration, we encounter architectures that maintain the ASR-LLM boundary but blur the interface. Instead of forcing the speech encoder to output discrete text tokens through a full CTC or seq2seq decoder, these models extract continuous representations from a speech encoder and project them into the LLM's embedding space.

In this paradigm, a pre-trained speech encoder (such as Whisper's encoder, wav2vec 2.0, or HuBERT) processes the audio into a sequence of hidden states. A projection layer, often a simple linear transformation or a small multi-layer perceptron, maps these acoustic features into the word embedding space of a frozen LLM. The LLM receives these "soft tokens" through its input layer, treating them similarly to word embeddings.

This approach, exemplified by models like Qwen-Audio and early versions of SpeechGPT, uses the powerful representations learned by dedicated speech encoders while tapping into the linguistic capabilities of LLMs. The speech encoder handles the acoustic complexity, compressing high-dimensional audio into meaningful features, while the LLM provides world knowledge, reasoning, and generation capabilities. By bypassing the forced discretization into text tokens, these models preserve more of the acoustic information present in the original signal, letting the LLM to potentially access paralinguistic cues embedded in the continuous representations.

The necessary component is the projection mechanism. Simply adding a linear layer often proves insufficient because speech encoders and LLMs were trained on vastly different objectives. The speech encoder optimized for phonetic discrimination may produce embeddings that live in a very different manifold than the semantic embeddings of an LLM. Effective projection requires alignment training: teaching the projection layer to map acoustic concepts (phonemes, words, speaker characteristics) to their corresponding linguistic representations.

This alignment process is akin to teaching the LLM a new language where the "vocabulary" consists of acoustic patterns rather than written words. The projection layer acts as a bilingual dictionary, translating from the dialect of sound into the dialect of text that the LLM already understands. When properly trained, the LLM can attend to speech features as if they were word embeddings, drawing connections between acoustic patterns and the semantic concepts it learned during text pre-training.

The deepest integration treats audio as a discrete sequence analogous to text tokens, letting a single transformer to process and generate both modalities natively. These systems, including AudioPaLM, VoxtLM, and SpeechGPT in its generative mode, rely on neural audio codecs to quantize continuous waveforms into discrete vocabulary items.

Neural codecs like EnCodec, SoundStream, or SpeechTokenizer compress audio into discrete codes through vector quantization. A VQ-VAE (Vector Quantized Variational AutoEncoder) architecture learns a codebook of audio embeddings. The encoder compresses waveform segments into latent vectors, and each vector gets replaced by its nearest neighbor in the codebook, creating an integer index. The decoder reconstructs the waveform from these indices.

This quantization process creates a vocabulary of sound, analogous to how subword tokenizers create a vocabulary of text fragments. Each codebook entry represents a prototypical acoustic pattern, much like how a text token represents a prototypical semantic unit. By flattening the multi-frame, multi-codebook output into a linear sequence, we obtain a representation that standard transformer architectures can process without modification.

When quantized at appropriate rates (typically 50-100 frames per second with multiple codebooks), audio becomes a sequence of integers that a standard transformer can process. An end-to-end speech LLM extends its vocabulary to include these "audio tokens" alongside text tokens. During training, the model learns to predict the next token, whether that token represents a text subword or an audio code.

This unification enables remarkable capabilities. The model can perform speech-to-text (audio tokens to text tokens), text-to-speech (text tokens to audio tokens), and speech-to-speech translation (audio tokens in language A to text or audio tokens in language B) within a single autoregressive framework. More importantly, it can reason about acoustic content: answering questions about background noise, identifying speaker characteristics, or generating speech with specific emotional qualities described in the prompt.

The autoregressive nature of these models means they generate audio token by token, just as they generate text word by word. This creates a unified reasoning process where the model can plan its acoustic output while considering both linguistic content and prosodic requirements. For example, when generating a question, the model can automatically produce the rising intonation pattern characteristic of interrogatives in many languages, without requiring a separate prosody prediction module.

The feasibility of end-to-end speech LLMs hinges on effective audio tokenization. The discrete representation must balance compression efficiency (minimizing sequence length) with reconstruction quality (preserving speaker identity, prosody, and content). Three major approaches dominate current research, each giving different tradeoffs between linguistic fidelity and acoustic richness.

The choice of tokenization strategy fundamentally shapes what the LLM can learn. Semantic-focused representations may enable better language understanding but produce robotic speech synthesis. Acoustically rich representations enable high-fidelity generation but may burden the LLM with modeling irrelevant details like background noise or recording artifacts. Hybrid approaches attempt to separate these concerns, letting the model to attend to semantic content for understanding while preserving acoustic detail for generation.

Models like HuBERT and wav2vec 2.0, mentioned in Part LI Chapter 1, learn discrete speech representations through self-supervision. The standard approach extracts k-means cluster indices from intermediate transformer layers, typically creating one discrete unit per 20 milliseconds of audio (50 Hz).

These units capture linguistic content well (clustering often aligns with phonemes) but discard speaker information and prosody. When used as the sole audio representation for an LLM, they enable content preservation but produce "robotic" reconstruction unless paired with a separate vocoder or speaker model.

The extraction process involves:

1. Passing audio through the self-supervised model
2. Extracting features from an intermediate layer (typically the 6th or 9th layer of a 12-layer model)
3. Applying k-means clustering with a predetermined number of clusters (typically 50, 100, or 200)
4. Assigning each frame its cluster ID as the discrete unit

The selection of intermediate layers is important and reflects a tradeoff between phonetic and lexical content. Lower layers capture fine-grained acoustic and phonetic details, while higher layers capture more abstract, word-level information. Mid-level layers (around the middle of the network) often provide the best balance, having processed enough context to resolve phonetic ambiguity while maintaining temporal precision. Researchers often evaluate different layers by measuring how well the resulting clusters align with phoneme boundaries or word boundaries, choosing the layer that optimizes the desired level of granularity.

Why does k-means work so well here? The self-supervised training objective encourages the model to produce features that are predictive of masked input regions. This forces the representations to capture structure that is both consistent across time (so that repeated instances of the same phoneme cluster together) and discriminative (so that different phonemes produce different clusters). The result is a representation space where Euclidean distance correlates well with phonetic similarity, making k-means an effective quantization strategy.

Neural codecs provide higher fidelity by learning data-driven compression rather than relying on clustering fixed features. EnCodec, developed by Meta, and Google's SoundStream use encoder-decoder architectures with residual vector quantization (RVQ).

The encoder produces a latent vector for each audio frame. Instead of a single quantization step, RVQ applies multiple codebooks sequentially. The first codebook quantizes the latent, creating a reconstruction error. The second codebook quantizes this error, refining the approximation. This continues for $N$ codebooks (typically 4-8), yielding $N$ discrete codes per frame.

This hierarchical quantization strategy is analogous to successive approximation in analog-to-digital conversion. The first codebook captures the coarse structure of the sound, while subsequent codebooks add finer details. This allows the codec to allocate bits efficiently: simple sounds might be well-represented by the first codebook alone, while complex sounds require the full stack of residual corrections.

The resulting representation captures both content and acoustic details. At 24 kHz sampling with 75 frames per second and 8 codebooks, the bitrate reaches 6 kbps, comparable to high-quality speech codecs, while maintaining speaker identity and prosody.

For LLM integration, these codes flatten into a sequence. With frame rate $r$ (frames/second) and $N$ codebooks, a $T$-second utterance produces $T \times r \times N$ tokens. At 50 fps with 4 codebooks, 10 seconds of speech generates 2,000 tokens, which is long but manageable with modern long-context LLMs discussed in Part XVIII: Long Context.

The flattening process typically interleaves codebooks to maintain temporal locality: instead of emitting all codes for frame 1 then all codes for frame 2, the sequence might proceed as codebook1_frame1, codebook2_frame1, codebook1_frame2, codebook2_frame2, and so on. This ensures that adjacent tokens in the sequence represent acoustically similar content, helping the autoregressive model learn smooth transitions between sounds. Some systems instead emit all frames for codebook 1 first, then all frames for codebook 2, separating the coarse acoustic structure from the fine detail. Each ordering creates different locality properties, and different research groups have found different orderings to work better depending on the downstream task.

The VQ-VAE training objective simultaneously minimizes reconstruction loss (so the decoder can recover the original audio from the codes) and commitment loss (so the encoder learns to commit confidently to codebook entries rather than hovering between two nearby entries). This dual objective is needed: without the commitment loss, the encoder might produce latent vectors that don't map cleanly to any codebook entry, making the discretization noisy.

Recent work recognizes a basic tension: linguistic content requires less information than acoustic detail. SpeechTokenizer and similar models explicitly factorize audio into semantic tokens (content, phonetics) and acoustic tokens (timbre, prosody, recording conditions).

This factorization mirrors the distinction between what is said and how it is said, or between message and style in natural language. Semantic tokens capture the linguistic message in a speaker-independent, environment-invariant format, while acoustic tokens capture the specific realization of that message by a particular voice in a particular setting.

The semantic tokens, extracted at lower bitrates (roughly 300-600 bps), feed into the LLM for understanding. The acoustic tokens remain latent or feed a separate reconstruction pathway. This factorization enables voice conversion (swap acoustic tokens while preserving semantic tokens) and content editing (modify semantic tokens while preserving speaker characteristics).

For speech LLMs, this suggests a hybrid architecture: the model processes semantic audio tokens for comprehension, generates text responses, and optionally produces acoustic tokens for speech synthesis. The modality alignment happens at the semantic level, while acoustic generation remains a separate decoding problem.

This separation of concerns offers significant practical advantages. The LLM can focus its capacity on language understanding and reasoning using the compact semantic stream, while a specialized decoder handles the computationally intensive task of rendering high-fidelity audio from acoustic tokens. By conditioning the acoustic decoder on speaker embeddings, the system can synthesize speech in various voices without requiring the LLM itself to model speaker characteristics.

The training procedure for factorized representations typically involves distilling semantic content from a pre-trained model like HuBERT. The first RVQ codebook is trained to match HuBERT cluster assignments, anchoring it to semantic content. Subsequent codebooks are then free to model the residual, which contains speaker identity, prosody, and recording conditions. This supervision ensures a clean factorization rather than leaving it to the model to discover independently.

When using continuous speech representations rather than discrete tokens, the interface between speech encoder and LLM becomes the necessary design decision. Several architectural patterns have emerged, each with distinct computational and representational tradeoffs.

The interface must solve two problems simultaneously: dimensionality alignment (so speech features match the LLM's expected input size) and semantic alignment (so acoustic concepts map to linguistic concepts). Different architectures approach these challenges with varying degrees of complexity and computational cost.

The simplest interface applies a learned linear projection $W \in \mathbb{R}^{d_{\text{llm}} \times d_{\text{audio}}}$ to map audio features into the LLM's dimensionality:

where:

• $\mathbf{h}_{\text{projected}}$: the audio features projected into the LLM's embedding dimension
• $W \in \mathbb{R}^{d_{\text{llm}} \times d_{\text{audio}}}$: the learned projection matrix mapping from audio to text dimensions
• $\mathbf{h}_{\text{audio}}$: the continuous audio feature vector from the speech encoder
• $\mathbf{b}$: the bias vector

This linear transformation aligns the geometric structure of acoustic embeddings with the semantic manifold of the LLM's word embeddings, letting the model to process speech features as if they were text tokens.

However, speech encoders often output high temporal resolution (e.g., 50 features per second). Feeding every frame to the LLM creates long sequences. Perceiver resamplers, introduced in the Flamingo architecture for vision-language models and adapted for audio, solve this by learning a fixed number of latent queries that attend to the variable-length audio sequence.

Given audio features $\mathbf{A} \in \mathbb{R}^{T \times d_{\text{audio}}}$ and $N$ learned latent queries $\mathbf{Q} \in \mathbb{R}^{N \times d_{\text{llm}}}$, a cross-attention layer computes:

where:

• $\mathbf{Q} \in \mathbb{R}^{N \times d_{\text{llm}}}$: the $N$ learned latent queries that compress the audio sequence
• $\mathbf{A} \in \mathbb{R}^{T \times d_{\text{audio}}}$: the audio features from the encoder with $T$ time frames
• $\mathbf{W}_Q \in \mathbb{R}^{d_{\text{llm}} \times d_k}$: projection matrix for queries
• $\mathbf{W}_K \in \mathbb{R}^{d_{\text{audio}} \times d_k}$: projection matrix for keys
• $\mathbf{W}_V \in \mathbb{R}^{d_{\text{audio}} \times d_{\text{llm}}}$: projection matrix for values
• $d_k$: the dimensionality of the key vectors (typically $d_{\text{audio}}$ or a compressed dimension)

The softmax operation computes a weighted average over audio positions, where each query attends to relevant acoustic features. The resulting $N$ output vectors summarize the entire audio sequence, drastically reducing sequence length from $T$ (potentially thousands) to $N$ (typically 32-64) tokens.

This compresses $T$ audio frames into $N$ fixed-size representations, typically 32-64 tokens regardless of audio duration. The LLM then processes these condensed "audio summaries" alongside text tokens. This approach sacrifices fine-grained temporal alignment for computational efficiency, which makes it suitable for understanding tasks rather than precise phonetic manipulation.

The key insight behind the Perceiver design is that for most understanding tasks, the LLM does not need to inspect every 20-millisecond frame of audio. It needs a small number of high-level summaries, similar to how a reader skimming a document focuses on topic sentences rather than every word. The learned queries develop functional roles during training: some specialize in extracting speaker identity, others in capturing intonation contours, and still others in distilling lexical content. This emergent specialization arises naturally from the training objective without explicit supervision.

The learned queries in the Perceiver act as information bottlenecks, forcing the model to compress the rich acoustic signal into a fixed number of summary vectors. During training, these queries learn to specialize, with different queries attending to different aspects of the audio such as speaker identity, emotional tone, or linguistic content. This specialization emerges naturally from the attention mechanism as the model discovers efficient ways to summarize the input for downstream tasks.

Building on BLIP-2's Q-Former, some speech LLMs employ a transformer-based query network that bridges frozen speech encoders and frozen LLMs. Unlike the simple Perceiver resampler, the Q-Former uses multiple transformer layers with cross-attention to audio features, letting deeper interaction before compression.

The Q-Former learns a set of "query tokens" that interact with audio features through cross-attention layers, while also attending to each other through self-attention. This bidirectional processing extracts richer information than single-layer resampling. The output query representations then project into the LLM's input space.

The architecture typically consists of several transformer blocks where each block contains a self-attention layer among the queries, followed by a cross-attention layer between queries and audio features, followed by a feed-forward network. This allows the queries to iteratively refine their understanding of the audio, with each layer building upon the abstractions learned by the previous layer.

The Q-Former is trained with a two-stage strategy: first aligning the audio representations to the LLM's space using contrastive learning, then fine-tuning for specific generative tasks. This staged approach prevents the powerful LLM from dominating the training dynamics early on, letting the connector to learn meaningful mappings before the LLM adapts to it.

In the first stage, contrastive learning encourages the query outputs to align with text representations of the audio content. This keeps audio of someone saying "hello" maps near the text embedding of the word "hello". In the second stage, generative training teaches the queries to extract the specific information needed for the LLM to perform tasks like transcription or question answering. This curriculum prevents the connector from simply passing through unprocessed audio features that the LLM cannot interpret.

The Q-Former approach is particularly valuable when both the speech encoder and the LLM are large, pre-trained, and expensive to fine-tune. By keeping both frozen and training only the lightweight connector, the method preserves the generalization capabilities of both foundation models while building a bridge between them. The trade-off is that the connector must accomplish all the cross-modal alignment in a relatively small number of parameters, which can limit performance on tasks requiring fine-grained acoustic understanding.

Some architectures, particularly those building on Whisper's encoder, simply feed the encoder outputs directly into early layers of the LLM, bypassing the input embeddings entirely. In this "early fusion" approach, the speech encoder's final hidden states undergo layer normalization and potentially dimensionality projection, then serve as the initial hidden states for the first $k$ layers of a deep LLM.

Mathematically, if the speech encoder produces $\mathbf{H}_{\text{audio}} \in \mathbb{R}^{T \times d}$ and the LLM has $L$ layers, the first $m$ layers of the LLM might process only audio:

where:

• $\mathbf{H}^{(l)}$: the hidden states at layer $l$
• $\mathbf{H}^{(0)}$: the projected audio features, computed as $\text{LayerNorm}(\mathbf{H}_{\text{audio}} \mathbf{W}_{\text{proj}})$
• $\text{TransformerLayer}^{(l)}$: the $l$-th transformer layer of the LLM (including self-attention and feed-forward networks)
• $m$: the number of initial layers dedicated to audio processing before text tokens are introduced
• $L$: the total number of layers in the LLM (with $m < L$)
• $T$: the number of time frames in the audio sequence
• $d$: the dimensionality of the audio features from the encoder
• $\mathbf{W}_{\text{proj}}$: the learned projection matrix mapping audio features to the LLM's hidden dimension

This progressive processing allows the lower layers to transform acoustic representations into linguistically meaningful features before the model handles multimodal interactions between speech and text in deeper layers. At layer $m$, text token embeddings join the sequence, and the remaining layers process the multimodal mixture.

This approach treats the lower LLM layers as learnable adapters, transforming speech representations into a format compatible with the upper layers' linguistic processing. It requires training the LLM from scratch or extensively fine-tuning, which makes it less suitable for integrating off-the-shelf frozen LLMs.

The intuition behind this approach is that the lower layers of a transformer learn more general, syntactic patterns while higher layers handle more abstract, semantic reasoning. By injecting audio directly into the lower layers, we allow the model to learn acoustic-to-phonetic and phonetic-to-lexical mappings using the same architectural components that normally learn character-to-subword and subword-to-word mappings in text-only training. This deep integration can capture subtle acoustic cues that might be lost in compression-based approaches, but at the cost of requiring full model training rather than lightweight adapter tuning.

Training a speech-language model involves more than architecture design. The optimization process must handle the chicken-and-egg problem: the speech encoder produces meaningful features only when the LLM can interpret them, but the LLM learns to interpret only when given meaningful features. Standard practice employs staged training with carefully curated data mixtures.

This bootstrapping challenge is characteristic of multimodal learning generally. Unlike text pre-training where the model learns language and world knowledge simultaneously from a single modality, speech-language models must coordinate two distinct representational systems. The training stages are designed to first establish basic communication between modalities before attempting complex reasoning tasks.

The first stage freezes both the speech encoder and the LLM, training only the connector (projection layer, perceiver, or Q-former). Using paired speech-text data, typically ASR training sets, the objective teaches the connector to map acoustic sequences to their textual equivalents.

For a speech encoder creating features $\mathbf{A}$ and text transcript with token embeddings $\mathbf{T}$, the alignment loss is:

where:

• $\mathcal{L}_{\text{align}}$: the negative log-likelihood loss measuring alignment quality
• $t_i$: the $i$-th text token in the target transcript
• $\mathbf{A}$: the audio features from the frozen speech encoder
• $t_{<i}$: the sequence of previous tokens $(t_1, t_2, \ldots, t_{i-1})$ used for autoregressive prediction
• $\theta_{\text{connector}}$: the trainable parameters of the connector (projection layer)
• $P(t_i \mid \cdot)$: the probability assigned by the LLM to token $t_i$ given audio context and previous tokens

This objective teaches the connector to map acoustic sequences such that the frozen LLM assigns high probability to the correct transcript tokens, effectively translating speech into the LLM's semantic space without updating the base models. The LLM parameters remain frozen; only $\theta_{\text{connector}}$ updates. This stage assumes the LLM already understands text; the connector merely translates speech into "LLM-readable" form.

Data requirements at this stage are substantial: thousands to tens of thousands of hours of speech with transcripts. However, since the base models are frozen, computation remains manageable compared to full fine-tuning. The quality of this alignment determines the ceiling for downstream performance: if the connector fails to map the word "weather" to the acoustic pattern for "weather", no amount of subsequent training will fix this basic misalignment.

An important subtlety of Stage 1 is the choice of training data. Simply maximizing transcription accuracy on a single ASR dataset risks over-fitting the connector to that particular recording style, microphone quality, and speaker distribution. Systems that perform well across diverse conditions typically use multi-condition ASR data spanning telephone speech, meeting recordings, broadcast audio, and spontaneous conversation. This diversity forces the connector to learn representations that are invariant to acoustic conditions rather than representations that merely match a specific studio recording signature.

After alignment, the model undergoes large-scale multimodal pre-training where both the connector and the LLM (or LoRA adapters on the LLM) update. The data mixture expands beyond ASR to include:

• Speech continuation (audio to future audio tokens)
• Speech-to-text translation
• Text-to-speech synthesis
• Speech-based question answering

This stage requires high-quality interleaved speech-text data, often sourced from audiobooks (aligned text and speech), podcasts with transcripts, and synthetic data generated by high-quality TTS systems. The mixture ratio matters: too much ASR biases the model toward transcription; too much generation degrades comprehension.

The curriculum during this stage often progresses from simpler to harder tasks. Early in this phase, the model might focus on speech-to-text tasks where the mapping is relatively direct. As training progresses, more weight is given to open-ended generation and question answering tasks that require the model to interpret audio content and formulate novel responses rather than simply transcribe. This staged curriculum prevents the model from overfitting to the most abundant but least cognitively demanding ASR data.

Another necessary consideration is catastrophic forgetting, a phenomenon we examined in Part XXXIV: Fine-tuning Fundamentals. When the LLM parameters update during Stage 2, the model may gradually lose its text-only capabilities as it adapts to the speech domain. Practitioners address this by including text-only examples in the Stage 2 mixture. This keeps the model continues to see and predict pure text sequences. This "replay" strategy maintains text performance while expanding to multimodal capabilities.

The final stage adapts the model to follow instructions and engage in dialogue, mirroring the instruction tuning process for text-only LLMs discussed in Part XXXVI: Instruction Tuning. Speech instruction datasets contain tuples of (audio instruction, audio/text response) or (text instruction, audio response).

Critically, instruction tuning for speech LLMs must handle turn-taking in conversations. Unlike text where <USER> and <ASSISTANT> tokens suffice, speech conversations involve acoustic cues: pauses, interruptions, backchannels ("uh-huh", "mm-hmm"). Advanced models learn to generate these paralinguistic behaviors, requiring training data that captures natural conversational flow.

The instruction templates for speech models must also account for the continuous nature of audio. While text instructions have clear boundaries, speech instructions might trail off, overlap with other sounds, or contain disfluencies. The model must learn to detect when a user has finished speaking and when to begin its response, a timing problem that does not exist in text-based chat. Special tokens or learned end-of-utterance detectors help manage these turn-taking transitions.

Instruction data quality matters more at this stage than at earlier stages. Whereas Stage 1 tolerates noisy ASR data (the objective is just to align modalities), Stage 3 data shapes the model's conversational behavior, safety responses, and helpfulness. The same principles that govern RLHF and preference learning for text LLMs apply here: diverse, high-quality human demonstrations produce better conversational agents than large quantities of automatically generated examples.

These architectural choices become clearer when we examine how actual deployed systems embody these choices. Several landmark models have defined the trajectory of the field, each making different bets about how tightly to integrate the acoustic and linguistic modalities.

Before considering integration, it is worth understanding why Whisper became the dominant pre-trained speech encoder for downstream systems. OpenAI's Whisper is not a speech LLM; it is a sequence-to-sequence model trained to transcribe and translate audio. However, its encoder produces exceptionally strong representations because it was trained on 680,000 hours of weakly supervised audio spanning 96 languages and 99 source languages for translation.

The main point of Whisper is that scale and diversity matter more than carefully curated labels. By training on audio scraped from the internet with associated text (subtitles, transcriptions), Whisper learned a reliable acoustic model without requiring expensive expert transcription. The encoder's final hidden states capture phonetic content, speaker characteristics, acoustic conditions, and linguistic context. This makes them excellent starting points for downstream integration with LLMs.

When researchers building speech LLMs need a frozen encoder, Whisper is often the first choice precisely because of this breadth. The encoder generalizes well to new domains without fine-tuning, and it has been studied extensively enough that its failure modes are well understood.

Alibaba's Qwen-Audio exemplifies the tight integration paradigm at scale. The system connects a large audio encoder (based on the Whisper large architecture) to Qwen-7B, a 7-billion parameter language model. Rather than a simple linear projection, Qwen-Audio uses a position-aware audio adapter that introduces relative positional information from the audio into the LLM's cross-attention mechanism.

What makes Qwen-Audio notable is the breadth of its training. The model was trained on a mixture that includes ASR data (speech recognition), AAC data (audio captioning, where the model describes non-speech sounds), speech translation, music description, and sound event detection. This diverse training produced a model that can answer questions like "What instrument is playing in the background?" or "How many speakers are present in this recording?", tasks that go well beyond simple transcription.

The practical takeaway is that tight integration architectures scale well. By keeping the core LLM frozen and training only the audio adapter and a limited set of LoRA parameters, Qwen-Audio achieves strong multimodal performance while retaining the text capabilities of the base Qwen model.

SpeechGPT from Fudan University takes the discrete token approach to its logical conclusion. The system uses HuBERT-derived speech units as the audio vocabulary, extending the LLaMA language model's vocabulary with these acoustic tokens. SpeechGPT then trains the unified model on a combination of speech-text cross-modal tasks.

The core capability that SpeechGPT demonstrates is speech-to-speech generation: the model takes a spoken question as input and produces a spoken answer as output, with the language model doing all the reasoning in a unified token space. The model does not explicitly transcribe the input or plan text before generating audio; it reasons directly from acoustic tokens to acoustic tokens, with learned associations linking acoustic patterns to concepts and responses.

SpeechGPT revealed several important phenomena. First, the LLM can learn to associate acoustic tokens with semantic content given sufficient training data, but this learning is slow and requires many more examples than text-only learning. Second, the acoustic token sequences are long enough that memory and compute requirements are significantly higher than for text-only models of the same capacity. Third, the model sometimes "code-switches" between text and speech internal representations during generation, suggesting that the boundary between modalities in the learned representations is less sharp than the token vocabulary implies.

Google's AudioPaLM represents perhaps the most ambitious approach: initializing a speech LLM from a text-only PaLM model and continuing pre-training on a mixture of speech and text data. This avoids the cold-start problem of learning speech representations from scratch and instead builds on PaLM's extensive world knowledge.

The AudioPaLM training procedure begins with a PaLM model that already knows about weather, geography, cooking, medicine, and countless other domains through text pre-training. The speech tokens introduced during continued pre-training must align with this existing knowledge. The model learns to associate the acoustic tokens for the word "Paris" with everything PaLM already knows about the city, rather than having to relearn that association from scratch.

This approach produces a model with stronger knowledge grounding than models trained from scratch on speech data, but it also requires careful management of the training mixture. Too much speech data and the model forgets text capabilities; too little and it never fully bridges the acoustic-linguistic gap. AudioPaLM's researchers found that maintaining roughly equal proportions of speech and text in the continued pre-training mixture worked well, with the speech data giving acoustic grounding and the text data preserving world knowledge.

Across these systems, several design patterns emerge consistently:

The frozen encoder, trainable connector pattern (Qwen-Audio, LLaSA) minimizes compute and preserves pre-trained capabilities in both the speech encoder and the LLM. It is the right choice when you have limited compute, need to preserve an existing LLM's text capabilities, and your application primarily requires speech understanding rather than generation.

The discrete token unification pattern (SpeechGPT, VoxtLM) achieves the deepest integration and enables audio generation, but requires far more data and compute. It is the right choice when you need speech-to-speech capabilities or want the model to reason about acoustic content rather than just understanding transcribed text.

The continued pre-training pattern (AudioPaLM) uses text pre-training to ground the speech representations in world knowledge but requires a large compute budget for the continued pre-training phase. It tends to produce the best knowledge-grounded speech understanding.

Choosing between these patterns requires honest assessment of your data budget, compute budget, and target capabilities. A startup building a voice assistant will likely start with the frozen connector pattern using an off-the-shelf speech encoder, while a research lab with large-scale resources might pursue end-to-end discrete token training.

To concretize these concepts, consider a 5-second audio clip containing the question "What's the weather like?" processed by a speech LLM using discrete audio tokens.

Step 1: Audio Encoding. The raw waveform (80,000 samples at 16 kHz) passes through an EnCodec encoder with 4 codebooks at 50 fps. This produces:

• 250 frames (5 seconds $\times$ 50 fps)
• 4 codebooks per frame
• Total: 1,000 discrete tokens
• Sequence: $[a_{1,1}, a_{1,2}, a_{1,3}, a_{1,4}, a_{2,1}, \ldots, a_{250,4}]$

The encoder compresses the raw audio by a factor of roughly 320:1 (80,000 samples to 1,000 tokens), discarding imperceptible details while preserving the needed acoustic structure needed for reconstruction. The first codebook captures coarse pitch and energy contours; the remaining three capture the fine spectral texture that distinguishes individual voices and recording conditions.

Step 2: Token Stream Construction. Special tokens delimit the modalities:

These delimiters act like XML tags, signaling to the model when it is processing acoustic versus linguistic content. The model learns to treat tokens between <AUDIO> and </AUDIO> as compressed sound requiring acoustic interpretation, while tokens following <TEXT> require semantic interpretation.

Step 3: LLM Processing. The transformer processes the 1,000+ token sequence through its self-attention layers. Rotary Position Embeddings (RoPE) from Part XIV: Positional Encoding apply across the unified sequence, letting the model to relate acoustic events at specific times to generated words.

As the model processes this sequence, attention heads learn to associate early acoustic tokens (capturing the "wh" sound of "what's") with the later text tokens spelling "what". This cross-modal attention emerges from training on paired speech-text data, where the model learns that certain acoustic patterns predict certain text sequences. The attention patterns that form during processing are not pre-specified; they emerge from the training objective as the most efficient way to relate acoustic and linguistic content.

Step 4: Response Generation. The model autoregressively generates a response. For a speech response, it predicts audio tokens. For text, it predicts subword tokens. If generating speech, the output might be:

The generation process attends back to the input audio tokens, letting the model to match speaking style, emotional tone, or prosodic patterns from the input. If the input sounded like a question, the output might adopt a corresponding declarative or informative prosody. If the speaker sounded rushed, the model may generate a more concise response. This responsiveness to acoustic context is precisely what cascaded systems cannot provide.

Step 5: Audio Decoding. The EnCodec decoder converts the predicted discrete codes back to a waveform, applying vocoder processing to produce the final "It's sunny and 72 degrees" response in natural speech.

Throughout this process, the model maintains a unified latent representation. The attention weights connecting early audio tokens to late generated tokens reveal how the model associates specific acoustic patterns (the "w" sound in "weather") with generated text tokens ("weather", "sunny"). This learned association is the operational definition of a model that "understands" speech rather than merely transcribing it.

Let us implement a simplified speech-to-text interface using a pre-trained Whisper encoder connected to a small language model. While production systems like Qwen-Audio use proprietary architectures, we can demonstrate the core principles: audio encoding, projection, and text generation.

We will use PyTorch and create a minimal connector between Whisper's encoder and a small GPT-style decoder.

This implementation illustrates the core principle of modern speech-LLM integration: use frozen foundation models for their respective modalities, and train lightweight connectors to bridge the semantic gap. The connector learns a translation from "acoustic semantics" to "linguistic semantics" without destroying the pre-trained knowledge in either the speech encoder or the language model.

The speech-LLM connector we implemented above exposes several hyperparameters that significantly affect both capability and computational cost. Understanding these tradeoffs helps in adapting the architecture to a specific application.

audio_dim controls the dimension of the speech encoder output (512 for Whisper base, 1280 for Whisper large-v2). This is not a free parameter; it must match the encoder you are using. If you want to use a larger, more capable encoder, you will need to update this parameter and the projection matrix will correspondingly be larger. However, a larger encoder also provides richer features that may reduce the burden on the connector.

text_dim must match the LLM's hidden dimension (768 for GPT-2, 4096 for LLaMA-7B, 8192 for LLaMA-70B). Larger LLMs require larger projection matrices in the connector but generally produce better language understanding. The matrix connecting Whisper-large to LLaMA-70B contains over 10 million parameters, but this remains tiny compared to the 70 billion parameters of the LLM itself.

max_audio_len is set to 1500 because Whisper's CNN downsampler produces at most 1500 frames from its 30-second input window. If you are using a different encoder with higher temporal resolution, you will need to increase this. The memory cost scales linearly: doubling the maximum audio length doubles the memory used by the positional embedding table.

n_mels (80 for Whisper) determines the frequency resolution of the mel-spectrogram. Whisper uses 80 mel bins, which provides sufficient resolution for speech while keeping the feature dimension manageable. Increasing this to 128 or 256 bins can improve quality for music or environmental sound tasks but does not help for speech recognition.

N (Perceiver queries) is the most impactful parameter for the throughput-quality tradeoff. With $N = 64$ queries, you compress even long audio utterances into 64 vectors before the LLM, keeping the LLM's context window requirements fixed regardless of audio duration. With $N = 256$, you preserve more detail but consume more of the LLM's context budget. For understanding tasks, 32-64 queries often suffice; for generation or precise temporal tasks, more queries help.

Number of RVQ codebooks determines the acoustic fidelity of discrete token representations. A single codebook with a 1024-entry vocabulary can represent coarse content but sounds robotic when decoded. Eight codebooks with 1024 entries each ($8 \times 1024 = 8192$ total codes) can reproduce near-studio-quality audio but increases token sequence length by a factor of eight. Most deployed systems use 4-8 codebooks, with the first one or two codebooks handling semantic content and the remainder handling acoustic detail.

Despite remarkable progress, speech-language integration faces significant challenges that shape its practical deployment. The most immediate limitation is data scarcity. While text LLMs train on trillions of tokens, high-quality speech-text paired data remains limited to thousands of hours. Speech is expensive to record and transcribe, including verification. This constrains the scale of speech LLMs compared to their text-only counterparts, though synthetic data generation and self-supervised pre-training partially mitigate the gap.

The scarcity extends beyond mere volume to diversity. Text corpora capture niche domains, rare languages, and specialized vocabulary through web crawling. Speech data collection requires speaker consent, studio quality or clean recording conditions, and careful transcription. Underrepresented languages and accents, including less common speaking styles remain challenging for current models, potentially perpetuating biases toward dominant linguistic varieties. A model trained predominantly on American English audiobooks will struggle with Nigerian Pidgin, Scots English, or any of the thousands of world languages with no large-scale speech corpus.

Latency presents another necessary barrier. End-to-end speech LLMs operating on discrete audio tokens must generate sequences 50-100 times longer than their text equivalents for the same duration of speech. Autoregressive generation of 2,000 tokens for a 10-second response takes considerably longer than generating 20 text tokens. Techniques like non-autoregressive decoding, speculative decoding from Part XLII: Inference Optimization, and parallel codebook prediction help, but real-time conversational speech remains challenging.

The streaming nature of speech exacerbates latency concerns. Unlike text where the entire message arrives simultaneously, speech unfolds over time. Humans begin processing and responding before a speaker finishes; current models typically wait for the complete utterance. Research into streaming architectures that can generate incremental responses while maintaining coherence represents a important direction for natural conversation. Some systems address this by processing speech in fixed-size chunks (e.g., 500 milliseconds) and running the LLM incrementally, but this risks breaking the acoustic context that spans chunk boundaries.

The modality gap itself persists in subtle ways. Speech encoders optimized for phonetic recognition may miss paralinguistic cues that humans effortlessly process, such as sarcasm, uncertainty, and emotional state. When these acoustic features do not align with textual content (saying "great" in a flat, disappointed tone), models often default to the text interpretation. Training on emotionally diverse data and using semantic-acoustic factorization helps, but fully integrated understanding of "how something is said" versus "what is said" remains imperfect.

Error propagation manifests differently across architectures. Cascaded systems compound ASR and LLM errors sequentially. End-to-end systems can hallucinate acoustic content, generating plausible-sounding but incorrect speech, or "verbal pareidolia" where noise is interpreted as words. The unified model has no external transcript to check against, making error detection harder. This is particularly concerning in high-stakes applications: a medical voice assistant that mishears "30 mg" as "13 mg" has no downstream transcript check to catch the error.

Evaluation presents additional challenges. Standard ASR metrics like Word Error Rate do not capture semantic correctness; a model might substitute "their" for "there" without changing meaning, or correctly transcribe "bank" while missing that the speaker meant the financial institution rather than the river edge. Developing evaluation protocols that assess semantic, pragmatic, and acoustic quality simultaneously remains an open research problem. The field currently lacks a speech equivalent to BLEU, ROUGE, or perplexity that captures all the dimensions users care about.

A less-discussed but increasingly important limitation is security. Speech provides an additional attack surface compared to text-only systems. Adversarial audio examples, carefully crafted signals that are imperceptible to human listeners but cause models to misrecognize commands, are a real threat in deployed voice interfaces. Unlike text-based prompt injection, acoustic adversarial attacks can be embedded in background noise, music, or seemingly innocuous audio streams. As speech LLMs become more capable, the consequences of such attacks grow more severe, and defending against them requires attention to robustness that is orthogonal to the architectural advances described in this chapter.

Despite these limitations, the impact of speech-language integration is already substantial and continues to grow. Voice interfaces for LLMs are being deployed at scale, making AI assistance accessible to people who cannot type, people with disabilities, and contexts where hands-free interaction is needed. By removing the transcription bottleneck, these models enable natural conversation with AI systems that understand emotion, handle overlapping speech, and generate appropriate prosody. The architectural patterns developed here, including projection layers, discrete tokenization, and multimodal pre-training, extend beyond speech to other modalities, informing the design of multimodal agents that smoothly process vision, audio, and text. As we move toward the next chapter on text-to-speech synthesis, we carry forward the understanding that speech is a rich, continuous signal that benefits from direct neural processing rather than merely text rendered audible.

Speech-language integration bridges the basic divide between continuous acoustic signals and discrete linguistic representations. This chapter explored the spectrum of approaches connecting these modalities, from cascaded pipelines that chain ASR and LLM components to fully end-to-end models that process audio tokens natively.

Key architectural patterns emerged from this exploration. Speech encoders (Whisper, wav2vec 2.0, HuBERT) extract meaningful acoustic features through self-supervised pre-training on massive audio corpora. Projection layers and Perceiver resamplers align these features with LLM embedding spaces, either directly mapping high-dimensional audio to the LLM's input space or compressing variable-length audio into a fixed-size summary. Neural audio codecs (EnCodec, SoundStream, SpeechTokenizer) quantize waveforms into discrete tokens that transformers can process autoregressively alongside text tokens. The choice between continuous projections and discrete tokens involves tradeoffs between reconstruction fidelity, sequence length efficiency, and architectural complexity.

Training these systems requires staged optimization: first aligning modalities with frozen components, then multimodal pre-training on diverse speech-text tasks, and finally instruction tuning for conversational competence. The data requirements remain the primary constraint, as speech collection and transcription costs vastly exceed text scraping. Factorized tokenization that separates semantic from acoustic content offers a promising direction, letting models to reason about linguistic content using compact representations while preserving acoustic detail for generation.

The impact extends beyond improved ASR accuracy. Unified speech LLMs enable emotional awareness, acoustic reasoning, and natural voice interaction, capabilities impossible when speech must first be forced into text. As these architectures mature and training data scales, the distinction between "speaking to" and "typing to" AI systems will fade, creating conversational agents that comprehend the full richness of human communication rather than only its lexical content.

Looking forward, the most exciting direction in this space is the development of universal multimodal models that treat speech, text, images, and video as equally first-class modalities rather than adding audio as an afterthought to a text model. Systems like Gemini represent early steps in this direction, but the basic challenge of efficiently aligning modalities with wildly different information densities and temporal structures remains open. The architectural insights from speech-language integration will be needed building blocks for that broader project.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about speech-language integration.