Speech-Language Integration › Connecting Speech Encoders to LLMs

When using continuous speech representations rather than discrete tokens, the interface between speech encoder and LLM becomes the necessary design decision. Several architectural patterns have emerged, each with distinct computational and representational tradeoffs.

The interface must solve two problems simultaneously: dimensionality alignment (so speech features match the LLM's expected input size) and semantic alignment (so acoustic concepts map to linguistic concepts). Different architectures approach these challenges with varying degrees of complexity and computational cost.

The simplest interface applies a learned linear projection $W \in \mathbb{R}^{d_{\text{llm}} \times d_{\text{audio}}}$ to map audio features into the LLM's dimensionality:

$$\mathbf{h}_{\text{projected}} = W \mathbf{h}_{\text{audio}} + \mathbf{b}$$

where:

• $\mathbf{h}_{\text{projected}}$: the audio features projected into the LLM's embedding dimension
• $W \in \mathbb{R}^{d_{\text{llm}} \times d_{\text{audio}}}$: the learned projection matrix mapping from audio to text dimensions
• $\mathbf{h}_{\text{audio}}$: the continuous audio feature vector from the speech encoder
• $\mathbf{b}$: the bias vector

This linear transformation aligns the geometric structure of acoustic embeddings with the semantic manifold of the LLM's word embeddings, letting the model to process speech features as if they were text tokens.

However, speech encoders often output high temporal resolution (e.g., 50 features per second). Feeding every frame to the LLM creates long sequences. Perceiver resamplers, introduced in the Flamingo architecture for vision-language models and adapted for audio, solve this by learning a fixed number of latent queries that attend to the variable-length audio sequence.

Given audio features $\mathbf{A} \in \mathbb{R}^{T \times d_{\text{audio}}}$ and $N$ learned latent queries $\mathbf{Q} \in \mathbb{R}^{N \times d_{\text{llm}}}$, a cross-attention layer computes:

$$\begin{aligned} \text{Scores} &= \frac{\mathbf{Q}\mathbf{W}_Q \, (\mathbf{A}\mathbf{W}_K)^T}{\sqrt{d_k}} \\ \text{Attention}(\mathbf{Q}, \mathbf{A}, \mathbf{A}) &= \text{softmax}(\text{Scores}) \, \mathbf{A}\mathbf{W}_V \end{aligned}$$

where:

• $\mathbf{Q} \in \mathbb{R}^{N \times d_{\text{llm}}}$: the $N$ learned latent queries that compress the audio sequence
• $\mathbf{A} \in \mathbb{R}^{T \times d_{\text{audio}}}$: the audio features from the encoder with $T$ time frames
• $\mathbf{W}_Q \in \mathbb{R}^{d_{\text{llm}} \times d_k}$: projection matrix for queries
• $\mathbf{W}_K \in \mathbb{R}^{d_{\text{audio}} \times d_k}$: projection matrix for keys
• $\mathbf{W}_V \in \mathbb{R}^{d_{\text{audio}} \times d_{\text{llm}}}$: projection matrix for values
• $d_k$: the dimensionality of the key vectors (typically $d_{\text{audio}}$ or a compressed dimension)

The softmax operation computes a weighted average over audio positions, where each query attends to relevant acoustic features. The resulting $N$ output vectors summarize the entire audio sequence, drastically reducing sequence length from $T$ (potentially thousands) to $N$ (typically 32-64) tokens.

This compresses $T$ audio frames into $N$ fixed-size representations, typically 32-64 tokens regardless of audio duration. The LLM then processes these condensed "audio summaries" alongside text tokens. This approach sacrifices fine-grained temporal alignment for computational efficiency, which makes it suitable for understanding tasks rather than precise phonetic manipulation.

The key insight behind the Perceiver design is that for most understanding tasks, the LLM does not need to inspect every 20-millisecond frame of audio. It needs a small number of high-level summaries, similar to how a reader skimming a document focuses on topic sentences rather than every word. The learned queries develop functional roles during training: some specialize in extracting speaker identity, others in capturing intonation contours, and still others in distilling lexical content. This emergent specialization arises naturally from the training objective without explicit supervision.

The learned queries in the Perceiver act as information bottlenecks, forcing the model to compress the rich acoustic signal into a fixed number of summary vectors. During training, these queries learn to specialize, with different queries attending to different aspects of the audio such as speaker identity, emotional tone, or linguistic content. This specialization emerges naturally from the attention mechanism as the model discovers efficient ways to summarize the input for downstream tasks.

Building on BLIP-2's Q-Former, some speech LLMs employ a transformer-based query network that bridges frozen speech encoders and frozen LLMs. Unlike the simple Perceiver resampler, the Q-Former uses multiple transformer layers with cross-attention to audio features, letting deeper interaction before compression.

The Q-Former learns a set of "query tokens" that interact with audio features through cross-attention layers, while also attending to each other through self-attention. This bidirectional processing extracts richer information than single-layer resampling. The output query representations then project into the LLM's input space.

The architecture typically consists of several transformer blocks where each block contains a self-attention layer among the queries, followed by a cross-attention layer between queries and audio features, followed by a feed-forward network. This allows the queries to iteratively refine their understanding of the audio, with each layer building upon the abstractions learned by the previous layer.

The Q-Former is trained with a two-stage strategy: first aligning the audio representations to the LLM's space using contrastive learning, then fine-tuning for specific generative tasks. This staged approach prevents the powerful LLM from dominating the training dynamics early on, letting the connector to learn meaningful mappings before the LLM adapts to it.

In the first stage, contrastive learning encourages the query outputs to align with text representations of the audio content. This keeps audio of someone saying "hello" maps near the text embedding of the word "hello". In the second stage, generative training teaches the queries to extract the specific information needed for the LLM to perform tasks like transcription or question answering. This curriculum prevents the connector from simply passing through unprocessed audio features that the LLM cannot interpret.

The Q-Former approach is particularly valuable when both the speech encoder and the LLM are large, pre-trained, and expensive to fine-tune. By keeping both frozen and training only the lightweight connector, the method preserves the generalization capabilities of both foundation models while building a bridge between them. The trade-off is that the connector must accomplish all the cross-modal alignment in a relatively small number of parameters, which can limit performance on tasks requiring fine-grained acoustic understanding.

Some architectures, particularly those building on Whisper's encoder, simply feed the encoder outputs directly into early layers of the LLM, bypassing the input embeddings entirely. In this "early fusion" approach, the speech encoder's final hidden states undergo layer normalization and potentially dimensionality projection, then serve as the initial hidden states for the first $k$ layers of a deep LLM.

Mathematically, if the speech encoder produces $\mathbf{H}_{\text{audio}} \in \mathbb{R}^{T \times d}$ and the LLM has $L$ layers, the first $m$ layers of the LLM might process only audio:

$$\mathbf{H}^{(l+1)} = \text{TransformerLayer}^{(l)}(\mathbf{H}^{(l)}) \quad \text{for } l = 0, \ldots, m-1$$

where:

• $\mathbf{H}^{(l)}$: the hidden states at layer $l$
• $\mathbf{H}^{(0)}$: the projected audio features, computed as $\text{LayerNorm}(\mathbf{H}_{\text{audio}} \mathbf{W}_{\text{proj}})$
• $\text{TransformerLayer}^{(l)}$: the $l$-th transformer layer of the LLM (including self-attention and feed-forward networks)
• $m$: the number of initial layers dedicated to audio processing before text tokens are introduced
• $L$: the total number of layers in the LLM (with $m < L$)
• $T$: the number of time frames in the audio sequence
• $d$: the dimensionality of the audio features from the encoder
• $\mathbf{W}_{\text{proj}}$: the learned projection matrix mapping audio features to the LLM's hidden dimension

This progressive processing allows the lower layers to transform acoustic representations into linguistically meaningful features before the model handles multimodal interactions between speech and text in deeper layers. At layer $m$, text token embeddings join the sequence, and the remaining layers process the multimodal mixture.

This approach treats the lower LLM layers as learnable adapters, transforming speech representations into a format compatible with the upper layers' linguistic processing. It requires training the LLM from scratch or extensively fine-tuning, which makes it less suitable for integrating off-the-shelf frozen LLMs.

The intuition behind this approach is that the lower layers of a transformer learn more general, syntactic patterns while higher layers handle more abstract, semantic reasoning. By injecting audio directly into the lower layers, we allow the model to learn acoustic-to-phonetic and phonetic-to-lexical mappings using the same architectural components that normally learn character-to-subword and subword-to-word mappings in text-only training. This deep integration can capture subtle acoustic cues that might be lost in compression-based approaches, but at the cost of requiring full model training rather than lightweight adapter tuning.