Speech-Language Integration › Key Parameters

The speech-LLM connector we implemented above exposes several hyperparameters that significantly affect both capability and computational cost. Understanding these tradeoffs helps in adapting the architecture to a specific application.

audio_dim controls the dimension of the speech encoder output (512 for Whisper base, 1280 for Whisper large-v2). This is not a free parameter; it must match the encoder you are using. If you want to use a larger, more capable encoder, you will need to update this parameter and the projection matrix will correspondingly be larger. However, a larger encoder also provides richer features that may reduce the burden on the connector.

text_dim must match the LLM's hidden dimension (768 for GPT-2, 4096 for LLaMA-7B, 8192 for LLaMA-70B). Larger LLMs require larger projection matrices in the connector but generally produce better language understanding. The matrix connecting Whisper-large to LLaMA-70B contains over 10 million parameters, but this remains tiny compared to the 70 billion parameters of the LLM itself.

max_audio_len is set to 1500 because Whisper's CNN downsampler produces at most 1500 frames from its 30-second input window. If you are using a different encoder with higher temporal resolution, you will need to increase this. The memory cost scales linearly: doubling the maximum audio length doubles the memory used by the positional embedding table.

n_mels (80 for Whisper) determines the frequency resolution of the mel-spectrogram. Whisper uses 80 mel bins, which provides sufficient resolution for speech while keeping the feature dimension manageable. Increasing this to 128 or 256 bins can improve quality for music or environmental sound tasks but does not help for speech recognition.

N (Perceiver queries) is the most impactful parameter for the throughput-quality tradeoff. With $N = 64$ queries, you compress even long audio utterances into 64 vectors before the LLM, keeping the LLM's context window requirements fixed regardless of audio duration. With $N = 256$, you preserve more detail but consume more of the LLM's context budget. For understanding tasks, 32-64 queries often suffice; for generation or precise temporal tasks, more queries help.

Number of RVQ codebooks determines the acoustic fidelity of discrete token representations. A single codebook with a 1024-entry vocabulary can represent coarse content but sounds robotic when decoded. Eight codebooks with 1024 entries each ($8 \times 1024 = 8192$ total codes) can reproduce near-studio-quality audio but increases token sequence length by a factor of eight. Most deployed systems use 4-8 codebooks, with the first one or two codebooks handling semantic content and the remainder handling acoustic detail.