whisper.cpp: Whisper Speech Recognition in C/C++

What whisper.cpp is

whisper.cpp is a C/C++ inference engine for OpenAI's Whisper automatic speech recognition (ASR) model. The README sums up the goal in its first line: high-performance inference of Whisper as a "plain C/C++ implementation without dependencies". There is no Python runtime, no PyTorch and no separate model server in the loop. A compiled binary or a linked library loads a single model file and turns audio into timestamped text.

The project lives under the ggml-org organisation on GitHub. According to the README, the whole high-level implementation of the model sits in two files, include/whisper.h and src/whisper.cpp, and everything else is part of the ggml machine learning library. That split explains most of the project's character: the speech-specific logic is small, and the heavy lifting (tensor maths, quantization, CPU SIMD paths and GPU backends) comes from ggml.

The audience is developers who want speech-to-text inside their own software rather than as a call to a hosted API. The README lists macOS (Intel and Arm), iOS, Android, Java, Linux and FreeBSD, WebAssembly, Windows (MSVC and MinGW), Raspberry Pi and Docker as supported targets, and it shows the model running fully offline on an iPhone 13. If the requirement is "transcribe audio on the machine where it was recorded, without shipping it anywhere", whisper.cpp is one of the most direct ways to do that.

How whisper.cpp works

Whisper is an encoder-decoder transformer. Audio is converted into a log-mel spectrogram, the encoder turns that into an audio representation, and the decoder generates text tokens conditioned on it. whisper.cpp reimplements that inference path on top of ggml and exposes it through a C-style API declared in whisper.h. The README's "Limitations" section is a single bullet: "Inference only". Training and fine-tuning happen elsewhere; whisper.cpp only runs models.

The ggml model file

The original PyTorch checkpoints are converted into a custom binary format. The README says this packs everything needed into one file: model parameters, mel filters, vocabulary and weights. models/README.md describes three ways to get such a file: run download-ggml-model.sh to fetch a pre-converted model, download one manually from the ggerganov/whisper.cpp repository on Hugging Face, or convert an OpenAI checkpoint yourself with convert-pt-to-ggml.py. A separate convert-h5-to-ggml.py script handles fine-tuned models in the Hugging Face Transformers format, and there is "initial support" for distil-whisper models.

Precision, memory and backends

The README lists mixed F16/F32 precision, integer quantization and "zero memory allocations at runtime" among the core properties. On the CPU side it advertises ARM NEON and the Accelerate framework on Apple Silicon, AVX intrinsics on x86 and VSX intrinsics on POWER. Accelerators are opt-in at build time: Metal on Apple Silicon (where the README says inference runs fully on the GPU), Core ML and ANEForge for the Apple Neural Engine, NVIDIA via CUDA, Vulkan as a cross-vendor option, AMD via HIP/ROCm, OpenVINO for Intel CPUs and GPUs, AMD Ryzen AI NPUs via VitisAI, Huawei Ascend via CANN, Moore Threads via MUSA and OpenBLAS for CPU encoder acceleration. Several of these, Core ML, OpenVINO and VitisAI among them, accelerate only the encoder and leave the decoder on the regular path.

Voice activity detection

whisper-cli can run a separate Silero-VAD model (also converted to ggml) before transcription. The README explains the flow: the VAD model detects speech segments, only those segments are extracted and passed to Whisper, and that "can significantly speed up the transcription process" because less audio is processed. Tuning knobs include the speech threshold, minimum speech and silence durations, maximum segment length, padding and overlap between segments.

Key features

• Dependency-free core. The library is plain C/C++, and the README's quick start builds the CLI with two CMake commands.
• A family of command-line tools. The examples table covers whisper-cli (transcribe and translate), whisper-bench (encoder benchmark), whisper-stream (live microphone capture), whisper-command (voice commands), whisper-server (HTTP server) and whisper-talk-llama (talking to a LLaMA bot).
• Many output formats. The whisper-cli help lists plain text, VTT, SRT, LRC, CSV, JSON and an extended JSON mode, plus a script that renders a karaoke-style video with ffmpeg.
• Timestamps down to word level. The experimental -ml (max segment length) flag controls segment length; -ml 1 gives word-level timestamps, and -dtw computes token-level timestamps.
• Integer quantization. Models can be quantized (for example to q5_0); the README says quantized models need less memory and disk and "depending on the hardware can be processed more efficiently".
• Speaker turns. With a small.en-tdrz model and the -tdrz flag, output is marked with [SPEAKER_TURN] via tinydiarize (experimental). A separate -di flag does stereo-audio diarization.
• Decoding controls. Beam size, best-of, temperature and fallback, entropy and log-probability thresholds, suppression regexes and even a GBNF grammar to guide decoding are exposed as flags.
• HTTP server with an OpenAI-like API. whisper-server accepts multipart uploads at /inference and can swap models through /load.
• Bindings and packaging. In-tree bindings exist for JavaScript (npm package whisper.cpp, built with Emscripten), Go and Ruby; the README links community bindings for Rust, Java, Swift, .NET, Python, R and Unity, plus a Conan package, Docker images and an XCFramework for Apple platforms.
• Parakeet support. The tree now also includes a parakeet-cli example that runs NVIDIA's Parakeet TDT 0.6B v3 model converted to ggml, with its own download and conversion steps.

Getting started with whisper.cpp

The README's quick start is three steps: clone, download a model, build and transcribe. Clone the repository and enter it:

git clone https://github.com/ggml-org/whisper.cpp.git
cd whisper.cpp

Download a pre-converted model. base.en is the English-only base model, about 142 MiB on disk:

sh ./models/download-ggml-model.sh base.en

Build with CMake and run the bundled JFK sample:

# build the project
cmake -B build
cmake --build build -j --config Release

# transcribe an audio file
./build/bin/whisper-cli -f samples/jfk.wav

The README notes that whisper-cli "currently runs only with 16-bit WAV files" and suggests converting input with ffmpeg first:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

GPU builds swap in a ggml flag at configure time. For NVIDIA:

cmake -B build -DGGML_CUDA=1
cmake --build build -j --config Release

For any GPU whose driver supports the Vulkan API:

cmake -B build -DGGML_VULKAN=1
cmake --build build -j --config Release

To quantize a model and run it, the README gives this sequence (note the binary name; see the hands-on section below):

# quantize a model with Q5_0 method
./build/bin/quantize models/ggml-base.en.bin models/ggml-base.en-q5_0.bin q5_0

# run the examples as usual, specifying the quantized model file
./build/bin/whisper-cli -m models/ggml-base.en-q5_0.bin ./samples/gb0.wav

Live microphone transcription needs SDL2 and a separate build flag:

cmake -B build -DWHISPER_SDL2=ON
cmake --build build -j --config Release
./build/bin/whisper-stream -m ./models/ggml-base.en.bin -t 8 --step 500 --length 5000

To enable VAD, download the Silero model and pass it alongside the Whisper model:

./models/download-vad-model.sh silero-v6.2.0
./build/bin/whisper-cli -vm /path/models/ggml-silero-v6.2.0.bin --vad -f samples/jfk.wav -m models/ggml-base.en.bin

Once whisper-server is running, a transcription request is a multipart POST, per the server README:

curl 127.0.0.1:8080/inference \
-H "Content-Type: multipart/form-data" \
-F file="@<file-path>" \
-F temperature="0.0" \
-F temperature_inc="0.2" \
-F response_format="json"

Models, sizes and quantization

The model choice dominates both quality and resource use. The README's memory table gives approximate disk and RAM figures for each size:

models/README.md adds naming rules worth knowing before downloading anything. Models are multilingual unless the name contains .en. Names ending in -q5_0 are pre-quantized: large-v3-q5_0 is listed at 1.1 GiB against 2.9 GiB for large-v3, and large-v3-turbo-q5_0 at 547 MiB against 1.5 GiB for large-v3-turbo. Names ending in -tdrz support tinydiarize speaker-turn marking. Each listed model has a published SHA, which makes it easy to verify a file copied in from elsewhere.

Hands-on: v1.9.5 on a 2-vCPU CPU-only VM

In a test run on 14 November 2026, release v1.9.5 (tag cloned with --depth 1) was built and run on a Linux x86_64 cloud VM with 2 vCPUs, AVX2 and AVX-512, and no GPU. Every number below is from a single run of each command. Treat it as an orientation point for small CPU-only machines, not a benchmark.

The transcription command was ./build/bin/whisper-cli -m models/ggml-base.en.bin -f samples/jfk.wav -t 2. It produced the expected single segment: [00:00:00.000 --> 00:00:11.000] And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.

The model was then quantized with ./build/bin/whisper-quantize models/ggml-base.en.bin models/ggml-base.en-q5_0.bin q5_0, which took 523 ms. Both models were run on the same 11-second clip with 2 threads:

Two practical takeaways. First, the quantized file was well under half the size and gave the same words on this clip, but it was slower on this particular CPU. That matches the README's careful wording that quantized models "depending on the hardware can be processed more efficiently": quantization reliably saves disk and RAM, while the speed effect depends on the machine. Measure on the target hardware before assuming a quantized model is faster. Second, the README's quantize example calls ./build/bin/quantize, but the v1.9.5 CMake build produced the tool as whisper-quantize. Adjust scripts copied from the README accordingly.

Adding -osrt wrote an .srt subtitle file. The first cue was numbered 1, timed 00:00:00,000 --> 00:00:11,000, and contained the same sentence, ready to drop into a video player or subtitle editor.

One step did not work as documented, and it is the most transferable lesson from the run. models/download-ggml-model.sh failed inside the VM with an HTTP 403 from curl, because the VM's egress proxy blocked huggingface.co. The script itself was fine; the network policy was not. Corporate networks and locked-down CI runners that block Hugging Face will hit the same wall. The workaround was to fetch ggml-base.en.bin on another machine and copy it in. For production builds, plan to mirror model files into internal artifact storage and check them against the SHA values in models/README.md rather than downloading at deploy time.

Use cases

• Offline transcription on laptops and desktops. Meeting recordings, interviews or lecture audio can be transcribed locally with whisper-cli, writing TXT, SRT, VTT or JSON for downstream tools.
• On-device speech in mobile apps. The repository ships iOS (whisper.objc, whisper.swiftui) and Android (whisper.android) examples, and the README links Swift and React Native bindings.
• Subtitles and captioning pipelines. SRT and VTT output, segment-length control and word-level timestamps cover most captioning needs, and the karaoke script can render word-highlighted video.
• A self-hosted transcription endpoint. whisper-server or the Docker images give a small internal HTTP service for teams that cannot send audio to a third-party API.
• Voice commands and assistants. whisper-command and whisper-talk-llama show how to wire microphone input into command handling or an LLM chat loop.
• Browser-side speech recognition. Several examples, including whisper.wasm and stream.wasm, run in the browser through WebAssembly.

How whisper.cpp compares

The neighbours below solve related but different problems. The table stays at category level; check each project's own documentation for specifics.

The practical rule of thumb: choose whisper.cpp when the deliverable is a binary or library that must run without Python, on unusual hardware or offline. Look at a higher-level toolkit when pipeline features matter more than footprint. Choose a generic runtime when Whisper is one of several model types in the same serving layer.

Limits and trade-offs

• Inference only. The README's limitations list contains exactly that. Fine-tuning has to happen in another framework, followed by conversion with convert-h5-to-ggml.py.
• Accuracy versus size. The jump from tiny (75 MiB, ~273 MB RAM) to large (2.9 GiB, ~3.9 GB RAM) is large. English-only .en models exist for the smaller sizes, and quantized variants trade some file size for a different precision profile. Test accuracy on your own audio before settling on a size.
• Quantization is not a guaranteed speed-up. As the hands-on run showed, q5_0 cut file size to about 37% but ran slower than f16 on a 2-vCPU AVX-512 machine. The README only promises efficiency "depending on the hardware".
• Accelerators require rebuilding. GPU and NPU support is chosen at configure time with flags such as -DGGML_CUDA=1, -DGGML_VULKAN=1, -DGGML_HIP=1, -DWHISPER_COREML=1, -DWHISPER_OPENVINO=1 or -DWHISPER_VITISAI=1, often with vendor SDKs installed first. Core ML and OpenVINO also require generating a separate encoder model with Python tooling, and the first run on a device is slow while the model is compiled.
• Audio input caveats. The README still says whisper-cli runs only with 16-bit WAV and recommends converting with ffmpeg, while the same README says the examples now decode audio with miniaudio by default, and the CLI help lists flac, mp3, ogg and wav. Broader format support needs a build with WHISPER_COMMON_FFMPEG. Converting to 16 kHz mono 16-bit WAV remains the safest input path.
• The streaming example is deliberately simple. The README calls whisper-stream "a naive example" that samples audio every half second. Its README warns that the default output rewrites one line in place with ANSI escapes, "fine for a terminal, wrong for a pipe"; parseable timestamped blocks only appear in VAD mode (--step 0). It also needs SDL2.
• Hallucinations on non-speech audio. The CLI README documents phantom text on music or silence, sometimes in an unexpected language. Its suggested mitigations are pinning the language with -l, --suppress-nst, --no-fallback, tighter thresholds, VAD and, as a last resort, --suppress-regex.
• Server hardening is up to you. The server README warns not to run whisper-server with administrative privileges and to operate it in a sandbox, because it accepts file uploads and can invoke ffmpeg for conversion.
• Models come from Hugging Face. The download scripts, the VAD models, the VitisAI encoder caches and the Parakeet model all fetch from Hugging Face. Restricted networks need a mirror, as the 403 in the hands-on run showed.
• Distilled models are partially supported. models/README.md says the chunk-based transcription strategy is not implemented, so quality with distil-whisper models can be sub-optimal.

Project activity and licence

Checked 14 November 2026: latest release v1.9.5 (6 October 2026), MIT licence, about 55,200 stars on the ggml-org/whisper.cpp repository. The repository was created on 25 September 2022, is written primarily in C++, uses master as its default branch and received its most recent push on 13 November 2026.

The LICENSE file is the standard MIT text with copyright held by "The ggml authors" (2023-2026), and the Go bindings README states that the bindings share the same MIT licence. The project has no separate homepage; documentation lives in the README, the per-example READMEs and the GitHub Discussions, where a frequently asked questions thread (#126) collects common questions.

• Repository: github.com/ggml-org/whisper.cpp
• Releases: github.com/ggml-org/whisper.cpp/releases
• Pre-converted models: huggingface.co/ggerganov/whisper.cpp
• Underlying tensor library: github.com/ggml-org/ggml