whisper.cpp: High-Performance Speech Recognition in C and C++

What it is

whisper.cpp is an open-source, high-performance port of OpenAI’s Whisper automatic speech recognition (ASR) architecture written directly in plain C and C++. Automatic speech recognition refers to technology that processes spoken human speech from audio recordings and converts those vocal sounds into written text. While modern machine learning tools frequently depend on large runtime runtimes and multiple external libraries, whisper.cpp is intentionally structured without third-party dependencies, providing a self-contained and compact implementation.

The architecture of whisper.cpp is cleanly divided. The high-level logic of the model is encapsulated within two main files: whisper.h, which provides a straightforward C-style application programming interface (API), and whisper.cpp, which implements the speech recognition pipeline. Lower-level computation and tensor operations are managed by ggml, a specialized machine learning library designed specifically for efficient local model inference.

A key design focus of whisper.cpp is runtime efficiency. The software is architected to perform zero memory allocations during execution, ensuring that memory usage remains steady and runtime overhead is minimized during continuous audio transcription. By keeping the codebase lightweight and avoiding external framework dependencies, whisper.cpp makes it straightforward for software engineers to embed speech recognition capabilities directly into desktop programs, mobile apps, web browsers, and embedded systems.

What it can do

whisper.cpp translates spoken words from audio files into text transcripts and provides a broad range of operational features and hardware acceleration options:

• Broad Model Support: The engine works with speech models converted into the custom ggml format. Supported model sizes include tiny, base, small, medium, and large variants (including large-v1, large-v2, large-v3, and large-v3-turbo). For tasks restricted to English speech, developers can choose specialized English-only models such as tiny.en, base.en, small.en, and medium.en.
• Voice Activity Detection (VAD): The project features integrated voice activity detection, which helps applications detect and isolate active speech segments within an incoming audio stream.
• Precision Modes and Quantization: The system supports mixed precision arithmetic using 16-bit floating-point (F16) and 32-bit floating-point (F32) formats. Additionally, it supports integer quantization (such as the Q5_0 format). Quantization is a process that converts full floating-point model weights into smaller integer values, significantly reducing the amount of disk storage and working memory required to load and execute the model.
• Hardware Acceleration Options:
  • Apple Silicon: Optimized as a first-class citizen on Apple hardware through ARM NEON vector instructions, the Accelerate framework, and GPU execution via Metal. It also supports running the encoder on the Apple Neural Engine (ANE) using Core ML or ANEForge. According to the project README, Core ML can deliver a significant speedup of more than three times compared with CPU-only execution, while ANEForge communicates directly with the ANE and is described as approximately two times faster than the Core ML encoder for tiny to medium models.
  • x86 and POWER Processors: Compatible x86 systems can leverage AVX vector intrinsics for faster calculations. On POWER architectures, the software supports VSX intrinsics; according to the project README, this code significantly speeds operations on Linux running on POWER9 or POWER10 hardware and makes the software capable of faster-than-realtime transcription on an underclocked Raptor Talos II system.
  • Graphics Processors (GPUs): GPU acceleration is implemented across multiple hardware ecosystems. It provides NVIDIA card acceleration via cuBLAS and custom CUDA kernels, AMD GPU acceleration via HIP and ROCm, Moore Threads card acceleration via muBLAS and custom MUSA kernels, and cross-vendor GPU support through the Vulkan graphics API.
  • Neural Processing Units and Specialized Frameworks: whisper.cpp supports OpenVINO for execution on Intel CPUs and Intel integrated or discrete GPUs, which according to the project README can result in a significant speedup in encoder performance. On AMD Ryzen AI 300 and 400 series processors featuring a dedicated NPU, the encoder can be offloaded via VitisAI, delivering what the project README describes as significant speedup over CPU-only inference. In addition, Huawei Ascend NPUs are supported via the CANN toolkit.
• Sample Implementations: The project provides practical demonstration examples, including on-device transcription on an iPhone 13 operating completely offline, offline voice command assistants, and containerized Docker configurations.

How to get started

Setting up whisper.cpp involves cloning the official repository, downloading a pre-converted model, building the project with CMake, and running transcription against an audio sample.

First, clone the repository from GitHub:

git clone https://github.com/ggml-org/whisper.cpp.git

Navigate into the newly created directory:

cd whisper.cpp

Next, download one of the pre-converted ggml models. For example, you can download the English base model using the provided script:

sh ./models/download-ggml-model.sh base.en

After the model finishes downloading, build the project and transcribe an audio file using the command-line utility:

# build the project
cmake -B build
cmake --build build -j --config Release

# transcribe an audio file
./build/bin/whisper-cli -f samples/jfk.wav

For a quick automated demonstration, you can run make base.en, which automatically retrieves the base.en model and executes transcription on all sample audio files located in the samples directory.

By default, the whisper-cli sample tool expects input audio to be formatted as 16-bit WAV files. If your audio is currently in a different format, such as MP3, you can convert it using ffmpeg:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

If you wish to test integer quantization to reduce resource requirements, you can build the quantizer and evaluate the smaller model file:

# quantize a model with Q5_0 method
cmake -B build
cmake --build build -j --config Release
./build/bin/quantize models/ggml-base.en.bin models/ggml-base.en-q5_0.bin q5_0

# run the examples as usual, specifying the quantized model file
./build/bin/whisper-cli -m models/ggml-base.en-q5_0.bin ./samples/gb0.wav

Pricing and requirements

whisper.cpp is published as an open-source project under the permissive MIT license. Commercial arrangements, plans, and official pricing terms are subject to the official project repository page at whisper.cpp.

The software supports an extensive range of operating platforms:

• macOS (Intel and Apple Silicon)
• iOS
• Android
• Java runtime environments
• Linux and FreeBSD
• WebAssembly (for web browser execution)
• Windows (via MSVC and MinGW toolchains)
• Raspberry Pi
• Docker containers

Hardware resource consumption depends on the model tier being used. According to the project README, the disk space and working memory requirements for standard models are:

• tiny: Requires 75 MiB of disk storage and approximately 273 MB of working memory.
• base: Requires 142 MiB of disk storage and approximately 388 MB of working memory.
• small: Requires 466 MiB of disk storage and approximately 852 MB of working memory.
• medium: Requires 1.5 GiB of disk storage and approximately 2.1 GB of working memory.
• large: Requires 2.9 GiB of disk storage and approximately 3.9 GB of working memory.

Standard builds require a functional C/C++ compiler and CMake. Audio decoding in the example programs relies on the included miniaudio library by default, with an optional compilation flag available to leverage system FFmpeg libraries (libavcodec-dev, libavformat-dev, and libavutil-dev). Specific hardware acceleration paths require respective vendor dependencies, such as the NVIDIA CUDA toolkit for CUDA builds, ROCm for AMD GPUs, OpenVINO packages for Intel acceleration, Python and Xcode for Core ML model generation on macOS, XRT and FlexML runtimes for AMD Ryzen AI NPUs, or the CANN toolkit for Ascend NPUs.

Who it is for

whisper.cpp is designed for developers, systems engineers, and technical creators who require a standalone, local speech recognition engine:

• Software Developers: Engineers who want to integrate on-device speech transcription into existing desktop, server, or enterprise software via a lightweight C-style interface without adding heavyweight runtime environments.
• Mobile and Embedded Creators: Programmers building speech tools on devices with limited computing resources, such as smartphones running iOS or Android, single-board computers like the Raspberry Pi, or web applications executing in browsers via WebAssembly.
• Privacy-Centric Builders: Teams developing applications, voice assistants, or dictation tools where spoken audio must remain entirely on the local device rather than being transmitted across networks to cloud APIs.
• Infrastructure Operators: Teams deploying containerized transcription services inside Docker environments across varied hardware platforms, ranging from standard CPU servers to specialized GPU and NPU accelerators.

Alternatives

OpenAI Whisper is an open-source speech recognition model developed by OpenAI.

faster-whisper is an alternative open-source implementation of the Whisper speech recognition model.

Sources

• https://github.com/ggml-org/whisper.cpp
• https://github.com/openai/whisper
• https://github.com/SYSTRAN/faster-whisper