Whisper (2022) › Limitations

Error rates were uneven across languages. The amount of training audio differed greatly, and the paper found a relationship between language-specific data volume and transcription quality. A headline language count therefore says little about whether a particular language or dialect is usable.

Whisper can hallucinate text that was not spoken, especially in silence, noise, or ambiguous audio. It may also repeat phrases. A fluent transcript is not proof of an accurate one, so high-consequence use requires checking the output against the recording.

The 30-second window and autoregressive decoder were designed for batch transcription rather than low-latency streaming. Processing long recordings requires windowing and timestamp heuristics. Boundary errors can omit speech or duplicate it across adjacent segments.

Compute remains another constraint. The largest original checkpoint has roughly 1.55 billion parameters. It can be slow on a CPU or a small device, while the faster checkpoints trade some accuracy for lower latency and memory use.

The web-sourced training set was not released, and its composition cannot be independently audited in full. This limits analysis of consent, copyrighted recordings, representational gaps, and label errors. Publishing model weights does not resolve those data questions.

Finally, Whisper's translation task targets English. It does not provide arbitrary speech-to-speech or speech-to-text translation between every pair of supported languages.