Whisper (OpenAI)

Multilingual Speech Recognition

Whisper is a general-purpose speech recognition model trained on a large dataset of diverse audio. It performs multilingual speech recognition across many languages, with model sizes ranging from tiny to large and English-only variants available.

Transcribe audio in multiple languages with a single model, choosing the size that balances speed and accuracy for your use case.

Speech Translation

Whisper can translate non-English speech into English. The multilingual models (tiny, base, small, medium, large) support translation tasks, while the turbo model is not trained for translation and will return the original language even if translation is requested.

Translate spoken content from various languages into English text, using medium or large models for the best translation results.

Spoken Language Identification

Whisper is a multitasking model that can perform spoken language identification. It uses special tokens as task specifiers or classification targets, allowing it to identify the language being spoken in an audio clip.

Automatically detect the language of audio input, enabling downstream processing without manual language selection.

Voice Activity Detection

Whisper includes voice activity detection as one of its jointly represented tasks. The model is trained on various speech processing tasks, including voice activity detection, all represented as sequences of tokens to be predicted by the decoder.

Identify segments of speech within audio, helping to filter out silence or non-speech portions during transcription.

Multiple Model Sizes

Whisper offers six model sizes (tiny, base, small, medium, large, turbo) with four English-only versions. The turbo model is an optimized version of large-v3, providing faster transcription with minimal accuracy loss. Memory requirements range from ~1 GB for tiny to ~10 GB for large.

Choose the right model for your hardware and speed needs, from lightweight tiny for quick tasks to large for maximum accuracy.

Simple Installation and Usage

Whisper can be installed via pip with 'pip install -U openai-whisper' and requires ffmpeg. It supports command-line usage (e.g., 'whisper audio.mp3 --model turbo') and Python usage via 'whisper.load_model' and 'model.transcribe'. The codebase is compatible with Python 3.8-3.11 and recent PyTorch versions.

Quickly integrate speech recognition into your workflow with straightforward installation and both CLI and Python APIs.

• General-purpose speech recognitionWhisper is a general-purpose speech recognition model trained on a large dataset of diverse audio. It can perform multilingual speech recognition, speech translation, and language identification.
• Multitask Transformer architectureA Transformer sequence-to-sequence model is trained on various speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. These tasks are jointly represented as a sequence of tokens, allowing a single model to replace many stages of a traditional speech-processing pipeline.
• Multiple model sizesThere are six model sizes, four with English-only versions, offering speed and accuracy tradeoffs. The relative speeds are measured by transcribing English speech on an A100.

• Python and PyTorch compatibilityThe codebase is expected to be compatible with Python 3.8-3.11 and recent PyTorch versions, but the developers used Python 3.9.9 and PyTorch 1.10.1 for training and testing. Ensure your environment meets these requirements.
• External dependenciesWhisper requires the command-line tool ffmpeg to be installed on your system. It also depends on OpenAI's tiktoken for its fast tokenizer implementation. If tiktoken does not provide a pre-built wheel for your platform, you may need to install Rust and configure the PATH environment variable.
• Installation and updatesInstall the latest release with 'pip install -U openai-whisper'. To pull the latest commit from the repository, use 'pip install git+https://github.com/openai/whisper.git'. For updates, run 'pip install --upgrade --no-deps --force-reinstall git+https://github.com/openai/whisper.git'. If installation fails with 'No module named setuptools_rust', install setuptools-rust.

Multilingual Speech Recognition

Whisper transcribes audio in multiple languages. It supports languages like Japanese, and you can specify the language directly, e.g., 'whisper japanese.wav --language Japanese'. The model handles diverse audio and is trained on a large dataset.

Speech Translation

Whisper can translate non-English speech into English. Use a multilingual model like medium or large with the --task translate flag, e.g., 'whisper japanese.wav --model medium --language Japanese --task translate'. The turbo model is not trained for translation.

Spoken Language Identification

Whisper is a multitasking model that can perform spoken language identification. It uses special tokens as task specifiers, allowing it to identify the language being spoken in an audio clip.

Voice Activity Detection

Whisper is trained on voice activity detection as one of its tasks. This allows it to detect when speech is present in an audio stream, which is useful for preprocessing or segmenting audio before transcription.

Define the exact aI Open-source Tools workflow Whisper (OpenAI) should support.

Compare it with closely related AI tools in the same category before committing.

Set review rules for accuracy, privacy, brand voice, compliance, and final approval.

Connect useful outputs to the wider stack instead of leaving them inside the AI tool.