Whisper Architecture › Timestamp Prediction

Traditional ASR systems output text without temporal alignment, requiring forced alignment algorithms to map words to audio timestamps. These post-processing steps often involve dynamic programming algorithms like dynamic time warping or hidden Markov model alignment, adding complexity and potential error accumulation to the pipeline. Whisper integrates timestamp prediction directly into the generative process, making speech recognition and alignment an end-to-end task.

This design choice has direct practical implications. Applications like subtitle generation, meeting transcription with speaker turns, and audio indexing all require knowing what was said and when it was said. In the traditional pipeline, alignment was always an afterthought, estimated by a separate system that might disagree with the ASR output. Whisper's integrated approach means that timestamps and text are generated jointly. This keeps consistency between the two.

Whisper discretizes the 30-second audio window into 1500 possible timestamp positions, each representing 0.02 seconds (20 milliseconds). These become additional vocabulary tokens that the decoder can predict at any point during generation:

• <|0.00|>, <|0.02|>, <|0.04|>, ..., <|30.00|>

When the model predicts a timestamp token, it indicates that the following text occurs at that specific time offset in the audio. The model learns to place these tokens at word or phrase boundaries, effectively segmenting the transcript temporally. This quantization balances precision with vocabulary size: 20 milliseconds is finer than typical word durations, letting precise alignment, while 1500 tokens remains computationally manageable compared to predicting raw continuous values or finer granularities.

The decision to discretize time into tokens rather than regress continuous values is architecturally important. A regression head would require the model to produce a scalar output in addition to the categorical token vocabulary, complicating the output representation and requiring separate training objectives. By treating timestamps as tokens, Whisper maintains a single unified output head: softmax over the full vocabulary, where the vocabulary happens to include both text tokens and time tokens. The model learns which output token is appropriate at each step through the same cross-entropy training objective used for text prediction. This simplicity is one of the elegances of Whisper's design: the entire system, including timing, is trained through standard language modeling.

During decoding, text and timestamp tokens interleave to create a structured temporal transcript:

The decoder must learn two coupled skills: recognizing linguistic content (what was said) and predicting temporal boundaries (when it was said). This is particularly challenging because the model must attend to the acoustic encoding to determine precise timing while maintaining linguistic coherence in the text generation. The timestamp tokens act as anchors, grounding the text generation in the temporal flow of the audio.

When the model predicts <|3.45|>, it must have determined from the encoder representations that approximately 3.45 seconds of audio have elapsed, which requires understanding the relationship between acoustic sequence position and real time, accounting for the convolutional downsampling and variable speech rates. The cross-attention mechanism is central here: the decoder positions that should predict timestamps attend strongly to the corresponding acoustic frames in the encoder output, learning a correspondence between text segments and acoustic positions.

An interesting emergent behavior is that timestamp prediction lets the model to handle silences gracefully. When the speaker pauses, the timestamp advances without corresponding text tokens, and the model learns to predict this pattern from the acoustic evidence of silence in the encoder representations. This natural silence handling is something that traditional ASR systems must handle with explicit pause detection and silence models.

You can disable timestamp prediction by including the <|notimestamps|> token in the decoder prompt. In this mode, the model generates only text tokens, similar to traditional ASR systems. This flexibility allows Whisper to serve applications requiring precise alignment (subtitling, diarization) and those needing only transcription (note-taking, search indexing). The no-timestamp mode also reduces the sequence length and computational cost slightly, as the model does not need to predict additional tokens for timing information. During training, the model encounters both modes, learning to condition its output format on the presence or absence of the timestamp control token.