Speech-Language Integration › Worked Example: Processing Speech Through an Integrated Model

To concretize these concepts, consider a 5-second audio clip containing the question "What's the weather like?" processed by a speech LLM using discrete audio tokens.

Step 1: Audio Encoding. The raw waveform (80,000 samples at 16 kHz) passes through an EnCodec encoder with 4 codebooks at 50 fps. This produces:

• 250 frames (5 seconds $\times$ 50 fps)
• 4 codebooks per frame
• Total: 1,000 discrete tokens
• Sequence: $[a_{1,1}, a_{1,2}, a_{1,3}, a_{1,4}, a_{2,1}, \ldots, a_{250,4}]$

The encoder compresses the raw audio by a factor of roughly 320:1 (80,000 samples to 1,000 tokens), discarding imperceptible details while preserving the needed acoustic structure needed for reconstruction. The first codebook captures coarse pitch and energy contours; the remaining three capture the fine spectral texture that distinguishes individual voices and recording conditions.

Step 2: Token Stream Construction. Special tokens delimit the modalities:

$$[\langle\text{AUDIO}\rangle, \, a_{1,1}, \ldots, a_{250,4}, \, \langle/\text{AUDIO}\rangle, \, \langle\text{TEXT}\rangle, \ldots]$$

These delimiters act like XML tags, signaling to the model when it is processing acoustic versus linguistic content. The model learns to treat tokens between <AUDIO> and </AUDIO> as compressed sound requiring acoustic interpretation, while tokens following <TEXT> require semantic interpretation.

Step 3: LLM Processing. The transformer processes the 1,000+ token sequence through its self-attention layers. Rotary Position Embeddings (RoPE) from Part XIV: Positional Encoding apply across the unified sequence, letting the model to relate acoustic events at specific times to generated words.

As the model processes this sequence, attention heads learn to associate early acoustic tokens (capturing the "wh" sound of "what's") with the later text tokens spelling "what". This cross-modal attention emerges from training on paired speech-text data, where the model learns that certain acoustic patterns predict certain text sequences. The attention patterns that form during processing are not pre-specified; they emerge from the training objective as the most efficient way to relate acoustic and linguistic content.

Step 4: Response Generation. The model autoregressively generates a response. For a speech response, it predicts audio tokens. For text, it predicts subword tokens. If generating speech, the output might be:

$$[\langle\text{AUDIO}\rangle, \, a'_{1,1}, \ldots, a'_{300,4}, \, \langle/\text{AUDIO}\rangle]$$

The generation process attends back to the input audio tokens, letting the model to match speaking style, emotional tone, or prosodic patterns from the input. If the input sounded like a question, the output might adopt a corresponding declarative or informative prosody. If the speaker sounded rushed, the model may generate a more concise response. This responsiveness to acoustic context is precisely what cascaded systems cannot provide.

Step 5: Audio Decoding. The EnCodec decoder converts the predicted discrete codes back to a waveform, applying vocoder processing to produce the final "It's sunny and 72 degrees" response in natural speech.

Throughout this process, the model maintains a unified latent representation. The attention weights connecting early audio tokens to late generated tokens reveal how the model associates specific acoustic patterns (the "w" sound in "weather") with generated text tokens ("weather", "sunny"). This learned association is the operational definition of a model that "understands" speech rather than merely transcribing it.