mlx-examples/whisper

Whisper

Speech recognition with Whisper in MLX. Whisper is a set of open source speech recognition models from OpenAI, ranging from 39 million to 1.5 billion parameters.¹

Setup

First, install the dependencies:

pip install -r requirements.txt

Install ffmpeg:

# on macOS using Homebrew (https://brew.sh/)
brew install ffmpeg

Next, download the Whisper PyTorch checkpoint and convert the weights to the MLX format. For example, to convert the tiny model use:

python convert.py --torch-name-or-path tiny --mlx-path mlx_models/tiny

Note you can also convert a local PyTorch checkpoint which is in the original OpenAI format.

To generate a 4-bit quantized model, use -q. For a full list of options:

python convert.py --help

By default, the conversion script will make the directory mlx_models/tiny and save the converted weights.npz and config.json there.

Tip

Alternatively, you can also download a few converted checkpoints from the MLX Community organization on Hugging Face and skip the conversion step.

Run

Transcribe audio with:

import whisper

text = whisper.transcribe(speech_file)["text"]

The transcribe function also supports word-level timestamps. You can generate these with:

output = whisper.transcribe(speech_file, word_timestamps=True)
print(output["segments"][0]["words"])

To see more transcription options use:

>>> help(whisper.transcribe)

---

1. Refer to the arXiv paper, blog post, and code for more details. ↩︎