Speech-Language Integration › Speech-Language Integration

Speech and text represent the same underlying linguistic content through fundamentally different physical manifestations. Text exists as discrete symbolic sequences, naturally compatible with the token-based processing of large language models. Speech, by contrast, is a continuous acoustic waveform, a high-dimensional signal varying over time, with frequency and amplitude as its other dimensions. Bridging this gap requires architectures that can translate between the continuous acoustic domain and the discrete linguistic domain without losing prosody and other nuanced paralinguistic information that speech conveys.

In prior chapters, we explored how models like Whisper convert speech to text through encoder-decoder architectures, effectively collapsing the acoustic signal into a sequence of text tokens. While powerful, this approach discards information. The timbre of a voice, the hesitation in a pause, and the emotion carried in intonation vanish when audio becomes text. Speech-language integration seeks to create unified models that understand and generate speech directly, treating acoustic information as a first-class modality alongside text.

Think about what happens when a friend tells you "sure, that's fine" in a flat, resigned voice versus an enthusiastic one. The words are identical, but any human immediately grasps the difference in meaning. A cascaded system that transcribes speech to text loses this distinction entirely before the language model ever sees it. This chapter is about building systems that keep the acoustic signal alive through the reasoning process, rather than discarding it at the first opportunity.

This chapter examines the architectures that bind speech encoders to large language models, the methods for representing audio as discrete tokens, and the spectrum of approaches from cascaded pipelines to fully end-to-end speech LLMs. We will see how the field has evolved from connecting separate systems to training unified models that think in both sound and text.