Whisper Training › Code Implementation: Working with Whisper
Language AI HandbookAll chapters Whisper Training › Code Implementation: Working with WhisperOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
While training Whisper from scratch requires massive computational resources (hundreds of thousands of GPU hours), we can explore the trained model to understand its behavior and capabilities. The following implementation uses the Hugging Face Transformers library to demonstrate Whisper's inference and examine its tokenization scheme.
The tiny model has 39 million parameters, while the large-v3 model has 1.5 billion parameters. All share the same architecture but differ in depth, width, and training data scale. The smaller models are suitable for edge devices or real-time applications where latency is necessary, while the large models maximize accuracy at the cost of computational requirements. Whisper medium (307M parameters) is a common compromise for production deployments. It offers accuracy close to the large model at significantly lower inference cost.
Examining the Tokenizer
Whisper's tokenizer is important to its multitask capabilities. Let's inspect the special tokens that control task specification:
# Create mock processor for inspection if not already defined
if "processor" not in globals():
processor = MockProcessor()
# Inspect special tokens
special_tokens = processor.tokenizer.special_tokens_mapThe vocabulary includes 99 language tokens (e.g., <|en|>, <|de|>, <|ja|>) plus task tokens for transcription, translation, and timestamp control. These tokens are used to prompt the decoder during inference, conditioning the generation on the desired task. The tokenizer treats these as standard vocabulary items, but they function as control codes that shift the model's behavior, similar to how instruction tokens work in text-based instruction tuning.
Forced Task Specification
We can force the model to perform specific tasks by manipulating the forced_decoder_ids parameter. This mirrors exactly how the training targets are structured, maintaining consistency between training and inference:
# Ensure processor is defined
if "processor" not in globals():
processor = MockProcessor()
# Demonstrate task specification
def get_forced_decoder_ids(
language="en", task="transcribe", return_timestamps=False
):
"""
Create forced decoder IDs for specific task configuration.
This mimics how Whisper is prompted during training and inference.
"""
forced_decoder_ids = []
# Start of transcript
start_token = processor.tokenizer.convert_tokens_to_ids(
"<|startoftranscript|>"
)
forced_decoder_ids.append((0, start_token))
# Language token
lang_token = processor.tokenizer.convert_tokens_to_ids(f"<|{language}|>")
forced_decoder_ids.append((1, lang_token))
# Task token
task_token = processor.tokenizer.convert_tokens_to_ids(f"<|{task}|>")
forced_decoder_ids.append((2, task_token))
# Timestamp option
if not return_timestamps:
no_timestamps = processor.tokenizer.convert_tokens_to_ids(
"<|notimestamps|>"
)
forced_decoder_ids.append((3, no_timestamps))
return forced_decoder_ids
# Example configurations
transcribe_en = get_forced_decoder_ids("en", "transcribe")
translate_de = get_forced_decoder_ids("de", "translate")The tuples represent (position, token_id) pairs that force the decoder to generate specific tokens at the beginning of the sequence. This is how Whisper maintains the multitask format during inference. This matches the training structure we discussed earlier. By forcing these specific tokens, we override the model's default behavior (which might be to auto-detect language) and explicitly set the task parameters. This keeps consistent output format.
Simulating Training Data Preparation
To understand how training data is prepared, let's simulate the preprocessing pipeline:
Mel spectrogram shape: (80, 3000) Label IDs shape: torch.Size([1, 5]) Target text: <|startoftranscript|><|en|><|transcribe|><|notimestamps|>This is a test transcription<|endoftext|>
This preprocessing would be applied to the massive weakly supervised dataset. The key insight is that the same code handles all languages and tasks, differing only in the special tokens prepended to the target text. This uniformity simplifies the training pipeline and ensures that the model learns a unified representation space across all languages and tasks, rather than developing isolated capabilities.
Analyzing Zero-Shot Capabilities
Whisper's zero-shot performance stems from its diverse training. Let's examine how the model handles different scenarios without fine-tuning:
import torch
# Ensure get_forced_decoder_ids is defined in this cell's namespace
if "get_forced_decoder_ids" not in globals():
def get_forced_decoder_ids(
language="en", task="transcribe", return_timestamps=False
):
"""
Create forced decoder IDs for specific task configuration.
This mimics how Whisper is prompted during training and inference.
"""
forced_decoder_ids = []
# Start of transcript
start_token = processor.tokenizer.convert_tokens_to_ids(
"<|startoftranscript|>"
)
forced_decoder_ids.append((0, start_token))
# Language token
lang_token = processor.tokenizer.convert_tokens_to_ids(
f"<|{language}|>"
)
forced_decoder_ids.append((1, lang_token))
# Task token
task_token = processor.tokenizer.convert_tokens_to_ids(f"<|{task}|>")
forced_decoder_ids.append((2, task_token))
# Timestamp option
if not return_timestamps:
no_timestamps = processor.tokenizer.convert_tokens_to_ids(
"<|notimestamps|>"
)
forced_decoder_ids.append((3, no_timestamps))
return forced_decoder_ids
def analyze_model_capabilities():
"""
Analyze the model's behavior across different prompting strategies.
This demonstrates the multitask training in action.
"""
# Create a dummy spectrogram (normally this comes from real audio)
dummy_input = torch.randn(1, 80, 3000) # (batch, mel_bins, time)
capabilities = {}
# Test 1: English transcription
forced_ids = get_forced_decoder_ids("en", "transcribe")
model.config.forced_decoder_ids = forced_ids
capabilities["english_transcription"] = "Forced to <|en|><|transcribe|>"
# Test 2: Spanish to English translation
forced_ids = get_forced_decoder_ids("es", "translate")
model.config.forced_decoder_ids = forced_ids
capabilities["spanish_translation"] = "Forced to <|es|><|translate|>"
# Test 3: Auto-detect language (no forced IDs after start token)
# During training, the model learns to predict the language token
# based on the audio encoder output
capabilities["auto_detect"] = "Model predicts <|lang_id|> from audio"
return capabilities
caps = analyze_model_capabilities()The model learns during training that the encoder's representation of Spanish audio should be followed by the <|es|> token and then <|translate|> if the target is English text, or <|transcribe|> if the target is Spanish text. This allows zero-shot task switching based purely on prompting. The model effectively learns a mapping from acoustic patterns to language identity, and from semantic content to target language text, letting it to perform tasks it was never explicitly optimized for individually, but rather learned as part of the joint multitask objective.