llama.cpp: LLM inference in plain C/C++

What llama.cpp is

llama.cpp is a C/C++ engine for running large language models, and increasingly vision-language models, on ordinary hardware. The project describes its main goal as enabling LLM (and VLM) inference "with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud." In practice that means a laptop with an Apple silicon chip, a Linux box with a consumer NVIDIA or AMD card, an Intel or AMD server CPU, a Snapdragon phone, or an IBM Z mainframe can all load the same model file and generate text without a Python stack, a container runtime or a vendor SDK being mandatory.

The repository ships three things that matter to developers. First, the llama library itself, whose C-style interface lives in include/llama.h. Second, a set of tools built on that library: an interactive CLI, a completion tool, and llama-server, an HTTP server with OpenAI- and Anthropic-compatible endpoints and a built-in web UI. Third, the build system and backends that let all of it run on CPUs, GPUs and NPUs from many vendors. Underneath sits ggml, the tensor library maintained by the same organisation, which llama.cpp is built on top of.

The audience is broad: application developers who want a local model behind an API, platform teams who want to self-host inference on hardware they already own, mobile and embedded developers who need an engine they can compile into an app, and researchers who want to try a newly released open-weights model the day it lands on Hugging Face.

How it works

llama.cpp is a plain C/C++ implementation "without any dependencies", in the README's words. Models are loaded from GGUF files, the single-file format used across the ggml ecosystem, and the tools can download those files directly from Hugging Face with the -hf flag. When a repository contains several quantisations, the server documentation says the default is Q4_K_M, falling back to the first file in the repo if that variant does not exist, and a multimodal projector (mmproj) is fetched automatically when one is available.

Performance comes from two directions. On the model side, llama.cpp supports 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit and 8-bit integer quantisation, which shrinks memory use and speeds up inference at some cost in output quality. On the hardware side, the compute graph is executed by one or more ggml backends: hand-tuned CPU paths (ARM NEON, Accelerate, AVX, AVX2, AVX512, AMX, and several RISC-V vector extensions), Metal on Apple silicon, custom CUDA kernels on NVIDIA GPUs, HIP for AMD GPUs, MUSA for Moore Threads GPUs, plus Vulkan, SYCL, OpenCL, WebGPU and others.

A key design choice is hybrid execution. If a model is larger than the available VRAM, llama.cpp can keep some layers on the GPU and run the rest on the CPU. The --n-gpu-layers option controls how many layers live in VRAM (it accepts a number, auto or all, with auto as the default), and there are finer controls for Mixture-of-Experts models, such as --cpu-moe to keep all expert weights in system RAM or --n-cpu-moe N to do that only for the first N layers. The KV cache data type can be changed independently for keys and values (for example to q8_0) to trade precision for memory.

The server adds its own layer: parallel decoding with multiple "slots", continuous batching, a prompt cache that can save and restore slot state to disk, speculative decoding with a draft model, and grammar- or schema-constrained output. A "router mode" lets one llama-server process expose many models and load or unload them on demand.

Key features

• Dependency-free C/C++ core: the library and tools build with CMake and do not require Python at runtime, which makes them easy to embed or ship as a single binary.
• Wide quantisation range: integer formats from 1.5-bit to 8-bit let the same model fit very different memory budgets.
• Apple silicon as a first-class target: the README calls it out explicitly, with optimisation through ARM NEON, Accelerate and Metal.
• Many GPU and accelerator backends: CUDA, HIP, MUSA, Metal, Vulkan, SYCL, OpenCL (Adreno), CANN (Ascend NPU), Hexagon (Snapdragon), IBM zDNN, ZenDNN, WebGPU, and an RPC backend for spreading work across machines; OpenVINO support is marked as in progress.
• CPU+GPU hybrid inference: partially accelerate models larger than total VRAM instead of failing to load them.
• OpenAI-compatible server: llama-server exposes chat completions, the Responses API, completions, embeddings and model listing under /v1, so existing OpenAI client libraries can point at it.
• Anthropic Messages compatibility: a /v1/messages endpoint plus token counting accepts Anthropic-style requests.
• Reranking, embeddings, infill and TTS: dedicated endpoints cover retrieval pipelines, code infilling and speech generation for supported models.
• Function calling and structured output: tool use "for ~any model", schema-constrained JSON responses, and GBNF grammars that can force output to match a formal grammar.
• Multimodal input: image and audio inputs through the OpenAI-compatible chat endpoint, documented as experimental.
• Monitoring: a public /health check, a /slots endpoint with per-slot metrics, and a Prometheus-compatible /metrics exporter.
• Router mode: start llama-server without a model to serve several models from the cache, a directory or an INI preset file, with a cap on how many stay loaded at once.

Supported hardware backends

The README lists the backends and the devices each one targets. A condensed version:

Backend	Target devices
Metal	Apple silicon
CUDA	NVIDIA GPUs
HIP	AMD GPUs
Vulkan	GPUs in general
SYCL	Intel GPUs
MUSA	Moore Threads GPUs
OpenCL	Adreno GPUs
CANN	Ascend NPUs
Hexagon	Snapdragon
IBM zDNN	IBM Z and LinuxONE
ZenDNN	AMD CPUs
BLAS, BLIS, RPC, WebGPU	All

Each backend is selected at build time, and the build guide has a section per backend with the CMake flags and environment variables it needs.

Getting started

The README offers an installer script for macOS and Linux and a PowerShell equivalent for Windows:

curl -LsSf https://llama.app/install.sh | sh
irm https://llama.app/install.ps1 | iex

Once installed, the quick start downloads a small model from Hugging Face and either chats with it or serves it over an OpenAI-compatible API:

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Package managers are an alternative. The install guide lists Homebrew (macOS and Linux), Winget (Windows), conda-forge, MacPorts and Nix:

brew install llama.cpp
winget install llama.cpp
conda install -c conda-forge llama.cpp

Building from source uses CMake. The CPU build is two commands after cloning the repository; GPU backends add flags described in the build guide:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

For containers, the project publishes images on GitHub Container Registry in full, light and server flavours, each with CUDA 12, CUDA 13, ROCm, MUSA, SYCL (Intel), Vulkan and OpenVINO variants. The server README shows this invocation:

docker run -p 8080:8080 -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:server -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080

Note that the README's quick start uses a unified llama command with cli and serve subcommands, while the per-tool documentation and the Docker images still refer to the individual binaries such as llama-server, llama-cli and llama-completion. Both names show up in current docs, so check which one your installation provides.

Running llama-server as a service

The server is where most application integrations happen, and its defaults are worth knowing. According to the server README, it listens on 127.0.0.1 and port 9931 by default; the Docker example above overrides both with --host 0.0.0.0 --port 8080. Every option can also be set through an environment variable (for example LLAMA_ARG_PORT), which suits container and systemd deployments.

Authentication is opt-in. --api-key accepts one or more keys, and --api-key-file reads them from a file. The /health endpoint is deliberately public and skips the key check. TLS requires building with OpenSSL 3 (-DLLAMA_OPENSSL=ON) and passing --ssl-key-file and --ssl-cert-file; the docs' recommendation for public deployments is to set an API key and put the server behind a reverse proxy.

CORS is permissive by default: the server reflects any Origin header back with credentials allowed, which the docs say is fine as long as only stateless, read-only endpoints are exposed. That changes when agent features are switched on. The experimental --tools flag gives the model built-in tools such as read_file, write_file, edit_file and exec_shell_command, and --mcp-servers-config spawns stdio MCP servers as child processes with the server's own privileges. In those modes CORS defaults to localhost only, and every one of these flags carries the warning "do not enable in untrusted environments".

For multi-model setups, router mode is started simply by omitting the model. It looks for models in the cache controlled by LLAMA_CACHE, in a directory passed with --models-dir, or in an INI preset file passed with --models-preset. --models-max limits how many models are loaded simultaneously (default 4), and --models-memory-margin unloads least-recently-used models to keep a set amount of device memory free. In router mode, requests to /metrics must name the model with a ?model= query parameter.

Multi-GPU inference

A dedicated guide covers running across more than one GPU, which is worth reaching for when a model does not fit in a single card's VRAM or when more throughput is needed. The split mode is set with --split-mode. The default, layer, is pipeline parallelism: each GPU holds a contiguous slice of layers and their KV cache, which tolerates slow interconnects and favours fast prompt processing. tensor is tensor parallelism that splits weights and KV cache across GPUs; it targets fast token generation but is marked experimental, and the guide says performance should be good for multiple NVIDIA GPUs on CUDA with "no guarantees otherwise". The older row mode is deprecated. --tensor-split sets the proportion of the model each GPU receives, and --main-gpu picks the device when only one should be used.

Use cases

• Local development against an OpenAI-style API: run llama-server on a workstation and point an existing OpenAI or Anthropic client library at it to build and test features without sending data to a hosted provider.
• Self-hosted inference on existing hardware: serve quantised open-weights models from servers that have consumer GPUs, AMD or Intel accelerators, or only CPUs, with Prometheus metrics feeding existing dashboards.
• Retrieval pipelines: use the embeddings and reranking endpoints to build search or RAG systems on the same engine that generates answers.
• Constrained generation: force JSON that matches a schema, or any custom grammar written in GBNF, for extraction and classification jobs that must produce parseable output.
• On-device and mobile apps: compile the library into Android or Apple apps (the docs include Android build notes and an XCFramework guide) to run models without a network connection.
• Trying new models quickly: pull a GGUF file straight from Hugging Face with -hf and chat with it in the CLI or web UI minutes after it is published.

How it compares

The README does not position llama.cpp against named competitors, so comparisons have to be drawn at the category level and with care. Python-first serving frameworks generally assume a CUDA-class GPU and a Python environment; llama.cpp's distinguishing traits are its dependency-free native build, its long list of non-NVIDIA backends, aggressive low-bit quantisation and CPU+GPU offloading for models that do not fit in VRAM. Desktop model runners and chat apps often sit a layer above an engine like this and add model management and polished UIs; llama.cpp itself now ships a web UI and router mode but remains closer to infrastructure than to an end-user app. Teams that need managed scaling, autoscaling or multi-tenant billing will still need to build or buy that around it.

Things to know before adopting

• Licence: the project is MIT-licensed. Bundled single-header dependencies listed in the acknowledgements are MIT or public domain (cpp-httplib, nlohmann/json, stb, miniaudio, subprocess.h). Model weights carry their own licences, which are separate from the engine's.
• Two release streams: the README links to versioned v* releases and to nightly b* builds. As of early November 2026 the newest versioned release is v0.8.0 (published 28 October 2026), while nightly builds are tagged several times a day (b12032 on 1 November 2026). Pin a version for production.
• Fast-moving surface: command names (llama serve versus llama-server), the default server port and individual flags have changed over time, and some options are marked deprecated. Read the release notes before upgrading.
• Experimental features: multimodal support, tensor-parallel multi-GPU, the built-in agent tools, MCP integration and the web UI's MCP proxy are documented as experimental.
• Security defaults: no API key is set by default, CORS reflects any origin unless agent features are enabled, and agent tools can read, write and execute on the host. Bind to localhost or put the server behind an authenticated proxy.
• GPU images are build-tested only: the Docker docs say the GPU-enabled images are not tested by CI beyond being built, and custom CUDA, ROCm or MUSA versions require building images locally.
• Installer scripts: the quick start pipes a remote script into a shell; teams with stricter policies can use package managers, release binaries or source builds instead.

Project activity

As of November 2026 the repository has about 133,000 stars and around 25,000 forks, which puts it among the most-starred AI infrastructure projects on GitHub. It was created in March 2023, is written mainly in C++, and is released under the MIT licence. Commits land daily, nightly builds are cut continuously, and the contributing guide describes a model where contributors open pull requests, collaborators are invited based on contributions, and maintainers merge into master. The source is at github.com/ggml-org/llama.cpp and the project site is llama.app.

[![Featured on TechLogHub](https://techloghub.com/badge-neutral.svg)](https://techloghub.com/open-source/llama-cpp?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)