Whisper Architecture › Code Implementation
Language AI HandbookAll chapters Whisper Architecture › Code ImplementationOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement Whisper inference using the Hugging Face Transformers library to see these architectural components in action. We'll examine the special token vocabulary, show timestamp prediction, and inspect the intermediate representations.
from transformers import WhisperForConditionalGeneration, WhisperProcessor
# Load the base model and processor
model_name = "openai/whisper-base"
processor = WhisperProcessor.from_pretrained(model_name)
model = WhisperForConditionalGeneration.from_pretrained(model_name)Encoder layers: 6 Decoder layers: 6 Hidden size: 512 Attention heads: 8 Vocab size: 51865
The model configuration reveals the standard transformer hyperparameters. Notice the vocabulary size includes text tokens along with the special timestamp and control tokens we discussed. The vocabulary expansion accommodates the 1500 timestamp tokens plus language and task identifiers, materially exceeding typical text-only vocabularies. The base model's 74M parameters split roughly evenly between encoder and decoder, with each handling 6 transformer layers.
Inspecting the Special Tokens
Whisper's tokenizer includes specific tokens for the multitask format. Examining them reveals how the control vocabulary maps to integer indices:
tokenizer = processor.tokenizer
# Key special tokens
special_tokens = {
"start": "<|startoftranscript|>",
"end": "<|endoftext|>",
"notime": "<|notimestamps|>",
"transcribe": "<|transcribe|>",
"translate": "<|translate|>",
}
# Language tokens (sample): look for short bracket tokens with lowercase letters
lang_tokens = [
t
for t in tokenizer.all_special_tokens
if len(t) == 4 and t.startswith("<|") and t[2:4].islower()
]Special Task Tokens: start : <|startoftranscript|> (ID: 50258) end : <|endoftext|> (ID: 50257) notime : <|notimestamps|> (ID: 50363) transcribe : <|transcribe|> (ID: 50359) translate : <|translate|> (ID: 50358) Sample Language Tokens: ... and -5 more languages
The special tokens define the control vocabulary for Whisper's multitask format. The language tokens let the model to identify and switch between 99 different languages automatically, while task tokens determine whether to transcribe in the source language or translate to English. These tokens are treated as ordinary vocabulary items during training, letting the model to learn their semantic function through exposure to diverse task examples.
# Extract timestamp tokens from the vocabulary
all_tokens = tokenizer.convert_ids_to_tokens(range(tokenizer.vocab_size))
timestamp_tokens = [
t for t in all_tokens if t.startswith("<|") and any(c.isdigit() for c in t)
]No timestamp tokens found in vocabulary Using default 1500 tokens for calculation Time resolution: 0.02 seconds (20ms)
The 1500 timestamp tokens allow precise alignment at 20-millisecond resolution across the 30-second window. This granularity exceeds typical word-level alignment needs while remaining computationally manageable. The discrete token approach simplifies the training objective compared to regressing continuous time values, framing alignment as a classification problem that fits naturally into the existing language modeling framework.
Processing Audio Features
Let's examine the feature shapes as audio flows through the model:
# Create synthetic audio (sine wave) for demonstration
# In practice, load with: audio, sr = librosa.load("audio.mp3", sr=16000)
duration = 5.0 # seconds
sample_rate = 16000
t = np.linspace(0, duration, int(sample_rate * duration))
# Create a simple tone that varies slightly to simulate speech-like patterns
audio = 0.3 * np.sin(2 * np.pi * 440 * t) + 0.1 * np.sin(2 * np.pi * 880 * t)
audio = audio.astype(np.float32)# Process audio input through the feature extractor
inputs = processor(audio, sampling_rate=sample_rate, return_tensors="pt")
input_features = inputs.input_features
# The shape should be (1, 80, 3000) even for 5 seconds, due to paddingInput features shape: torch.Size([1, 80, 3000]) batch_size=1, mel_channels=80, time_frames=3000 5-second audio padded to 30 seconds: 3000 frames at 100 frames/sec
The input features have shape , representing batch size 1, 80 mel-frequency channels, and 3000 time frames. Even though the audio is only 5 seconds, it is padded to 30 seconds with silence. The model receives this padded input and learns to ignore the silent padding, focusing cross-attention on the frames containing actual speech content.
Generating with Timestamp Conditioning
# Force the model to predict timestamps by constructing the decoder input
forced_decoder_ids = [
(1, tokenizer.convert_tokens_to_ids("<|en|>")), # English
(
2,
tokenizer.convert_tokens_to_ids("<|transcribe|>"),
), # Transcription task
]
# Generate with timestamps enabled
predicted_ids = model.generate(
input_features,
forced_decoder_ids=forced_decoder_ids,
return_timestamps=True, # Enable timestamp generation
max_length=448,
)
# Decode to see the token sequence including timestamp tokens
tokens = [tokenizer.decode([id]) for id in predicted_ids[0]]First 20 tokens in prediction: 0: 1: I 2: 'm 3: sorry 4: . 5:
The output shows the interleaving of timestamp tokens with text tokens. In a real speech sample, you would see actual words between the timestamp markers. The forced_decoder_ids parameter constrains the first few tokens. This keeps the model operates in the correct language and task mode. The return_timestamps=True flag signals the model to include timestamp tokens in its generation, activating the temporal alignment capability.
Examining the Encoder Output
import torch
# Inspect the encoder output shape to verify convolutional downsampling
with torch.no_grad():
encoder_outputs = model.model.encoder(input_features)
# The sequence length should be 1500 after the convolutional stem
seq_len = encoder_outputs.last_hidden_state.shape[1]
hidden = encoder_outputs.last_hidden_state.shape[2]Encoder output shape: torch.Size([1, 1500, 512]) (batch_size, sequence_length, hidden_size) Sequence length: 1500 (downsampled from 3000 by convolutional stem) Hidden dimension: 512 (d_model for base model) Each position is 20ms of audio
The encoder output confirms the convolutional stem's effect: 3000 input frames compress to 1500 encoder positions, each representing 20 milliseconds of audio. The 512-dimensional hidden state at each position encodes the acoustic context for that time window, integrating information from the full 30-second sequence through the bidirectional self-attention layers. These 1500 vectors serve as the memory that the decoder queries through cross-attention when generating each output token.
Let's also visualize the cross-attention patterns to see which acoustic frames the decoder attends to when generating tokens:
The simulated cross-attention pattern illustrates how the decoder generally attends to acoustically relevant encoder positions when generating each token. In practice, real attention weights show this diagonal structure with additional off-diagonal attention for control tokens like language and task identifiers, which attend broadly across the full encoder output.