ggml: Tensor Library for Machine Learning in C/C++
What ggml is
ggml is a tensor library for machine learning written in C and C++. Its README states the goal in one sentence: "to be a simple, portable, and efficient tensor library for machine learning with minimal setup." It is the layer underneath llama.cpp and whisper.cpp: the llama.cpp README says that project "is built on top of the ggml library", and the whisper.cpp README says the high-level model code lives in whisper.h and whisper.cpp while "the rest of the code is part of the ggml machine learning library". Anyone who has run a GGUF model locally has, directly or indirectly, run ggml.
The README lists six headline properties: a plain C/C++ implementation without dependencies; cross-platform builds for x86, ARM, RISC-V, LoongArch, PowerPC, s390x and WebAssembly; SIMD-optimised kernels for x86, ARM and RISC-V; backends for CPU, GPU, NPU and browser; 2- to 8-bit integer quantization plus the MXFP4 and NVFP4 microscaling formats; and "zero memory allocations during runtime". It is not a training framework in the PyTorch sense, nor a model server. It is the piece a C or C++ program links to when it wants to define a network as a graph of tensor operations, load quantized weights and run that graph on whatever hardware is present.
The repository, ggml-org/ggml, was created on 18 September 2022 and is MIT-licensed, with the copyright line "Copyright (c) 2023-2026 The ggml authors". The auto-generated AUTHORS file, last regenerated on 4 November 2026, lists roughly 950 contributors. The project tagged v1.0.0 on 13 November 2026.
How it works: contexts, tensors and computation graphs
Contexts and memory
Everything in ggml starts with a context. ggml_init() takes a ggml_init_params struct with three fields: mem_size in bytes, an optional mem_buffer ("if NULL, memory will be allocated internally") and a no_alloc flag that tells the context not to allocate memory for tensor data. Tensor metadata and graphs are carved out of that pool, which is how the library keeps its promise of no allocations at runtime: the memory is reserved up front and reused. With no_alloc set, the context only holds metadata and the actual data lives in backend buffers, which is the pattern the newer examples use.
Tensors
The ggml_tensor struct in include/ggml.h is small and readable. It carries a type (one of the ggml_type values such as GGML_TYPE_F32, GGML_TYPE_F16, GGML_TYPE_Q4_0 or GGML_TYPE_Q8_0), a pointer to the backend buffer, and two arrays of length GGML_MAX_DIMS, which is 4: ne, the number of elements per dimension, and nb, the stride in bytes. The header comment spells out the stride rule: nb[0] = ggml_type_size(type), nb[1] = nb[0] * (ne[0] / ggml_blck_size(type)) + padding, and each higher stride is the previous one times the previous element count. Dimension 0 is the innermost, contiguous one, which is the reverse of the row-major shape order NumPy and PyTorch print.
A tensor also records the operation that produces it (op), up to GGML_MAX_SRC (10) source tensors, operation parameters, flags that mark it as a graph input, output, trainable parameter or loss, and a view_src/view_offs pair for views into other tensors. Calling an operation such as ggml_add or ggml_mul_mat does not compute anything; it returns a new tensor node whose sources point at the operands.
Graphs and execution
Computation is deferred. A program builds the forward graph by calling ggml_build_forward_expand(graph, result), which walks the sources of result and adds them to a ggml_cgraph; ggml_build_backward_expand extends a graph with gradients for training. Graphs come from ggml_new_graph (default size GGML_DEFAULT_GRAPH_SIZE, 2048 nodes) or ggml_new_graph_custom. They can be printed with ggml_graph_print or dumped to Graphviz with ggml_graph_dump_dot, which is useful when debugging a model port. Execution is a separate step: the backend API's ggml_backend_graph_compute(backend, graph) runs the whole graph on one backend and returns a ggml_status.
Backends and the scheduler
include/ggml-backend.h describes a layered model: buffer types and buffers (memory on a device), backends (an execution stream), devices (with a type of CPU, GPU, integrated GPU, accelerator or a "meta" device that wraps several others for tensor parallelism) and a registry that can load backends from dynamic libraries via ggml_backend_load_all(). ggml_backend_init_best() picks a GPU device if one is available and falls back to CPU.
For mixed hardware there is a backend scheduler. The header says it "allows for multiple backend devices to be used together" and "handles compute buffer allocation, assignment of tensors to backends, and copying of tensors between backends", choosing a backend for each operation based on which backend supports it and where pre-allocated tensors such as weights already live. Backends with a lower index get priority. This is how an example can request CUDA or Metal and still run operations that the GPU backend does not implement on the CPU, a fallback the MNIST README warns "may result in bad performance".
Quantized blocks
Quantized types store values in fixed-size blocks rather than one value at a time. In src/ggml-common.h, block_q8_0 is a 16-bit ggml_half scale d followed by int8_t qs[32], so 32 weights take 34 bytes. block_q4_0 is the same scale followed by uint8_t qs[16], two 4-bit values per byte, so 32 weights take 18 bytes. Both definitions carry static_asserts on the block size. The header also defines the K-quants (Q2_K to Q8_K), the IQ family, ternary TQ1_0/TQ2_0, MXFP4 and NVFP4. A comment on the ggml_type enum says new types are always appended "to keep backward compatibility", and several retired entries are left in place as comments marked "support has been removed".
Key features
- No runtime dependencies. The core is C11 and C++17 (set in
CMakeLists.txt) with only a threads library required; BLAS, CUDA, Vulkan and the rest are opt-in. - Deferred graph execution. Operations build a graph; nothing runs until the graph is computed, which lets the allocator and scheduler plan memory and device placement for the whole graph.
- Wide quantization coverage. Classic Q4/Q5/Q8 block formats, K-quants, IQ formats, ternary types, MXFP4, NVFP4 and FP8 E4M3 alongside F32, F16 and BF16.
- Many backends behind one API. The
src/tree at the tested commit holds backend directories for BLAS, CANN, CPU, CUDA, HIP, Hexagon, Metal, MUSA, OpenCL, OpenVINO, RPC, SYCL, Vulkan, WebGPU, zDNN, ZenDNN, a virtual GPU backend and one namedggml-et. - Backend scheduler with CPU fallback. One graph can be split across devices, with unsupported operations falling back to a lower-priority backend.
- Automatic differentiation and optimisers.
ggml_build_backward_expandand themnist-trainexample show training on top of the same graph, though the MNIST README calls training "a work-in-progress and not production ready". - Build-time CPU tuning.
GGML_NATIVE(on by default unless cross-compiling) targets the build machine;GGML_BACKEND_DLwithGGML_CPU_ALL_VARIANTSbuilds several CPU variants as loadable libraries for portable binaries.
Getting started
The README's quick start builds from source with CMake:
git clone https://github.com/ggml-org/ggml
cd ggml
mkdir build && cd build
cmake ..
cmake --build . --config Release -j 8
CMakeLists.txt requires CMake 3.14 or newer and turns on GGML_BUILD_TESTS and GGML_BUILD_EXAMPLES when ggml is the top-level project, so this produces the test binaries and examples as well as the libraries. On Apple platforms GGML_METAL and GGML_BLAS (with the Accelerate vendor) default to on; elsewhere GPU backends are off until enabled with options such as GGML_CUDA, GGML_VULKAN, GGML_HIP, GGML_SYCL or GGML_WEBGPU.
The README points first to examples/simple, "a minimal, fully commented example (matrix multiplication)". It ships in two forms: simple-ctx, which keeps everything in a ggml context and "doesn't support gpu acceleration", and simple-backend, which "demonstrates how to use other backends like CUDA and Metal". After that, the MNIST example is the most complete tour. Its README trains a small network in PyTorch, saves it as GGUF, and evaluates it with ggml from inside examples/mnist:
$ python3 mnist-train-fc.py mnist-fc-f32.gguf
$ ../../build/bin/mnist-eval mnist-fc-f32.gguf data/MNIST/raw/t10k-images-idx3-ubyte data/MNIST/raw/t10k-labels-idx1-ubyte
The same README shows training directly in ggml:
$ ../../build/bin/mnist-train mnist-fc mnist-fc-f32.gguf data/MNIST/raw/train-images-idx3-ubyte data/MNIST/raw/train-labels-idx1-ubyte
The GPT-2 example covers a full transformer with weight quantization. Its README converts F16 weights to Q4_0 with the bundled quantizer:
./bin/gpt-2-quantize models/gpt-2-1558M/ggml-model-f16.bin models/gpt-2-1558M/ggml-model-q4_0.bin 2
Note that this example still uses the older single-file ggml-model.bin format rather than GGUF, and that its README shows a ./bin/gpt-2 binary, while the build at the tested commit produced gpt-2-ctx, gpt-2-alloc, gpt-2-backend, gpt-2-sched and gpt-2-batched instead.
Hands-on: building ggml 1.0.0 and running its tests on arm64
The build and tests below ran on 17 November 2026 on a Linux arm64 virtual machine with 4 vCPUs and about 3.9 GB of RAM on an Apple Silicon host, using GCC 11.4.0 on Ubuntu 22.04 and CMake 4.4.5 (installed from PyPI because the VM had no CMake). The source was a shallow clone of master at commit d9fe7bf, committed on 16 November 2026.
Configure and build
| Step | Command | Result |
|---|---|---|
| Configure | cmake -B build -DCMAKE_BUILD_TYPE=Release | 0.66 s on a re-run; printed ggml version: 1.0.0, ggml commit: d9fe7bf, ARM detected and Adding CPU backend variant ggml-cpu: -mcpu=native |
| Build (clean, separate build directory) | cmake --build build2 --config Release -j4 | 27 s wall clock, exit code 0 |
| Shared libraries | - | libggml-base.so 850,808 B, libggml-cpu.so 919,248 B, libggml.so 48,800 B |
Executables in build/bin | - | 38 binaries, about 4.9 MB: 15 examples (gpt-2-*, gpt-j, gpt-j-quantize, magika, mnist-eval, mnist-train, sam, simple-backend, simple-ctx, yolov3-tiny) and about 23 test-* programs |
The split into libggml-base (core and graph code), libggml-cpu (the CPU backend) and a thin libggml reflects the backend registry design: a GPU build adds further backend libraries next to these rather than growing one monolith.
CPU feature probes
The configure step probes the compiler for Arm extensions. On this VM the results did not line up with what the CPU advertises:
| Probe | CMake result | Related flag in /proc/cpuinfo |
|---|---|---|
HAVE_FMA | Success | - |
HAVE_DOTPROD | Failed | asimddp present |
HAVE_MATMUL_INT8 | Failed | i8mm present |
HAVE_FP16_VECTOR_ARITHMETIC | Failed | fphp, asimdhp present |
HAVE_SVE | Failed | - |
HAVE_SME | Failed | - |
The cause was not investigated further. With GCC 11 and -mcpu=native in this VM the probes did not enable these paths; a newer compiler or explicit architecture flags may give a different result. The practical point is to read the configure output on any new build machine rather than assume the native build picked up every extension.
The transposed mul_mat convention
Both ./bin/simple-ctx and ./bin/simple-backend printed the same result:
| Output header | Row 1 | Row 2 | Row 3 |
|---|---|---|---|
mul mat (4 x 3) (transposed result) | 60.00 55.00 50.00 110.00 | 90.00 54.00 54.00 126.00 | 42.00 29.00 28.00 64.00 |
This matches examples/simple/README.md. A conventional 4x2 by 2x3 product gives a 4x3 matrix whose first column is 60, 55, 50, 110. In ggml, "we pass the matrix B in transposed form and multiply row-by-row. The result C is also transposed": ggml_mul_mat(A, B^T) = C^T. The comment above ggml_mul_mat in ggml.h says the same in shape terms: A has k columns and n rows, B has k columns and m rows ("we transpose it internally"), and the result has n columns and m rows. Both inputs share their innermost dimension ne[0], which keeps the dot products contiguous in memory. Code ported from NumPy, where A @ B pairs columns with rows, is where this convention usually bites.
Operator tests
| Command | Result |
|---|---|
./bin/test-quantize-fns | Exit 0, no failures; covered types include tq2_0, mxfp4, nvfp4, q1_0, q2_0 and f8_e4m3, plus "degenerate imatrix" checks for q2_K, q4_K, q5_K, q4_1 and q5_1 |
./bin/test-backend-ops test -o ADD -b CPU | 173/173 tests passed, 1/1 backends passed |
./bin/test-backend-ops --list-ops | Total: 127 operations |
test-backend-ops --help lists four modes: test (compare a backend's results with the CPU backend), grad (check gradients with finite differences), perf and support. Options include -o to filter by operation, -b to pick a backend by name (for example CPU, MTL0 or CUDA0), -p for a parameter regex, --output with console, sql or csv, and -j for parallel workers. For anyone evaluating a GPU backend, support mode is the quickest way to see which of the 127 operations it implements before relying on it.
One decode-step matrix-vector product
To see what quantization does on the CPU backend, test-backend-ops perf timed a 4096 x 14336 weight matrix multiplied by a single f32 vector, the shape of one feed-forward projection during single-token decoding (117.44 MFLOP per run). The command was ./bin/test-backend-ops perf -o MUL_MAT -b CPU -p 'type_a=(f32|f16|q8_0|q4_0),type_b=f32,m=4096,n=1,k=14336', and the tool reported 3906 MB of device memory. Weight sizes are computed from the block layouts above (4 bytes per value for f32, 2 for f16, 34 bytes per 32 values for q8_0, 18 bytes per 32 values for q4_0).
| Weight type | Time per run | GFLOPS | Weight size |
|---|---|---|---|
| f32 | 2065.05 us | 56.87 | 234.9 MB |
| f16 | 2226.71 us | 52.74 | 117.4 MB |
| q8_0 | 1116.43 us | 105.19 | 62.4 MB |
| q4_0 | 1335.75 us | 87.92 | 33.0 MB |
This is a single run on a single 4-vCPU virtual machine, so treat it as an illustration, not a benchmark. On this VM q8_0 was the fastest, about 1.85 times quicker than f32 while using roughly a quarter of the memory, and q4_0 came second despite being half the size of q8_0. f16 was not faster than f32, which is consistent with the FP16 vector arithmetic probe failing at configure time. The q4_0 result should not be generalised either: GGML_CPU_REPACK ("use runtime weight conversion of Q4_0 to Q4_X_X") is on by default, and the extension probes above failed, so a different compiler or CPU could reorder these rows.
A measurement gotcha
./bin/test-quantize-perf --type q4_0 --op vec_dot_q -i 20 reported min cycles/32 vals: 0.00 and avg cycles/32 vals: 0.00 on this arm64 VM, because it had no usable cycle counter. Only the throughput lines from that tool were meaningful there, and the small 4096-value tests that fit in L1 cache produced noisy numbers. On virtualised Arm hosts, test-backend-ops perf with wall-clock timing is the more trustworthy of the two.
GGUF: the model format
docs/gguf.md defines GGUF as "a file format for storing models for inference with GGML and executors based on GGML", designed for fast loading and saving. Models are "traditionally developed using PyTorch or another framework, and then converted to GGUF". It succeeds the earlier GGML, GGMF and GGJT formats; the key change, according to the spec, is a typed key-value structure for metadata instead of a list of untyped hyperparameters, so new fields can be added without breaking existing readers.
The layout, summarised at the top of include/gguf.h, is: the magic GGUF, a version (currently 3), tensor and key-value counts, the key-value pairs, a tensor-info table (name, dimensions, ggml_type, offset) and an aligned data blob. Alignment defaults to 32 bytes unless general.alignment says otherwise, which is what makes mmap loading possible. The spec says files are little-endian by default, that version 3 added big-endian support, and that "there is no way to determine if a model is big-endian" from the file itself. general.architecture is required, and general.quantization_version must be present if any tensor is quantized.
The spec also sets a filename convention, <BaseName><SizeLabel><FineTune><Version><Encoding><Type><Shard>.gguf with optional mmproj or mtp sidecar prefixes, and gives a validation regex. One limit is worth knowing: GGUF stores weights and metadata, not the computation graph. The spec lists embedding a graph as "a future extension" whose "primary blocker is the stabilization of the computation graph format", so every executor still has to implement each model architecture in code.
Use cases
- Building a local inference engine for a new model family. This is what llama.cpp and whisper.cpp do, and what the gpt-2, gpt-j, SAM, YOLO and Magika examples in the repository demonstrate at smaller scale.
- Embedding inference in a native application. A C API, no Python runtime and no required dependencies make ggml suitable for desktop apps, mobile apps and command-line tools. Projects such as Jan credit llama.cpp, and LocalAI's README says the team wrote 18 of its own "native C, C++ and GGML engines with no Python at inference".
- Running quantized models on CPUs and modest GPUs. Block quantization cut the weight footprint of the test matrix above from 234.9 MB to 33 MB, which is the main reason ggml-based tools fit large models into laptop memory.
- Browser and edge targets. The MNIST README shows compiling the evaluation code to WebAssembly with Emscripten, and the backend list includes WebGPU, Hexagon and other NPU targets.
- Writing and testing new kernels.
test-backend-opsgives backend authors a ready-made correctness and performance harness against the CPU reference.
How ggml compares
The table positions ggml against libraries it is often weighed against. Each row reflects only what that project's own README says; "not stated" means the README read did not say, not that the feature is missing.
| Project | Language and API | Primary target, per its README | Model input | Quantization in README | Licence |
|---|---|---|---|---|---|
| ggml 1.0.0 | C/C++ library with a C API; simple autogenerated cffi Python bindings in examples/python | "Tensor library for machine learning" with minimal setup, on CPU, GPU, NPU and browser | Graphs built in code; weights usually from GGUF | 2- to 8-bit integer, MXFP4, NVFP4 | MIT |
| ONNX Runtime | Cross-platform runtime | "Cross-platform inference and training machine-learning accelerator" for models from PyTorch, TensorFlow/Keras, scikit-learn and others | ONNX models | Not stated in README | MIT; README notes it may collect usage data |
| PyTorch | Python first, with C/C++ extension API | Tensor computation "with strong GPU acceleration" and "tape-based autograd" | Models defined in Python | Not stated in README | BSD-style |
| MLX | Python API following NumPy, plus C++, C and Swift | "Array framework for machine learning on Apple silicon", lazy evaluation, unified memory | Models defined in code | Not stated in README | MIT (LICENSE file) |
| Apache TVM | Python-first compiler | "Open machine learning compilation framework" producing "minimum deployable modules" | Models compiled through its pipeline | Not stated in README | Apache-2.0 |
The broad split is between frameworks that take a model from elsewhere (ONNX Runtime with ONNX files, TVM through compilation) and libraries where the model is written in code (PyTorch, MLX, ggml). Within the second group, ggml is the only one here aimed at C and C++ programs with no Python in the loop, and the only one whose README leads with low-bit quantization. PyTorch is the natural choice for research and training; MLX targets Apple silicon specifically. On the serving side, vLLM lists GGUF among the quantization formats it accepts, so ggml's format reaches beyond ggml's own runtime.
Limits and when not to use it
- Not a training framework. Autodiff and optimisers exist, but the MNIST README says training is "a work-in-progress and not production ready". Train in PyTorch or similar and convert.
- Low-level API. Every model architecture has to be written by hand as graph construction code; GGUF carries no graph. For running an existing LLM, llama.cpp is the higher-level entry point.
- Backend coverage varies. The MNIST README warns that if a backend lacks an operation, "a CPU fallback is used instead which may result in bad performance". Some CMake options are labelled "experimental, unstable" (MUSA graphs) or "llama.cpp only" (CUDA graphs). Check
test-backend-ops supportfor the target backend. - Native builds can miss CPU features. In the hands-on run, GCC 11 probes failed for dot-product, int8 matrix multiply and FP16 arithmetic even though the CPU listed them, and f16 ran no faster than f32.
- Contributions go elsewhere. The README and
CONTRIBUTING.mdask that changes to core ggml, including the CMake build, be opened as pull requests in llama.cpp, "doing so will make your PR more visible, better tested, and more likely to be reviewed". Review of core changes therefore happens in the llama.cpp repository, which matters for anyone planning to upstream a fix. - API stability is not spelled out. The project moved from frequent 0.x tags (v0.28.0 on 28 October, v0.29.0 on 4 November 2026) to v1.0.0 on 13 November 2026. Neither the README nor
CONTRIBUTING.mdstates a compatibility policy for 1.x; the only explicit rule found is the append-onlyggml_typeenum. Pin a tag or commit. - Uneven documentation. The README links a Hugging Face introduction and a llama.cpp wiki page for most conceptual material. Several example READMEs are dated: the GPT-2 README uses a non-GGUF
.binformat and a binary name the current build does not produce, and the Python bindings README builds against llama.cpp with olderLLAMA_*CMake flags.
Project activity
Checked 17 November 2026: latest release v1.0.0 (13 November 2026), MIT licence, about 16,100 stars (16,098) and 2,075 forks on the ggml-org/ggml repository, per the GitHub API. The repository was created on 18 September 2022, is written mainly in C++, develops on master and is not archived; the most recent commit at the time of testing, d9fe7bf, was made on 16 November 2026. The GGUF specification is in docs/gguf.md, and the sibling projects llama.cpp and whisper.cpp live in the same ggml-org organisation.
Enjoying this project?
Discover more amazing open-source projects on TechLogHub. We curate the best developer tools and projects.
Repository:https://github.com/ggml-org/ggml
GitHub - ggml-org/ggml: ggml: Tensor Library for Machine Learning in C/C++
ggml is the MIT-licensed C/C++ tensor library under llama.cpp and whisper.cpp. Version 1.0.0 built and tested on arm64: graphs, backends, GGUF, the transposed m...
github - ggml-org/ggml
Related Projects
llama.cpp: LLM inference in plain C/C++
Dependency-free C/C++ LLM inference with quantization, many GPU backends and an OpenAI-compatible server.
Claude-Mem: Persistent Memory for AI Coding Agents
Persistent cross-session memory for Claude Code, OpenCode, Codex and other agents, with SQLite and MCP search.
vLLM: High-Throughput LLM Inference and Serving Engine
Python engine for fast LLM inference and OpenAI-compatible serving, built around PagedAttention.
Maintain ggml: Tensor Library for Machine Learning in C/C++? Add the badge to your README
Show that the project is featured on TechLogHub and send readers to this write-up.
[](https://techloghub.com/open-source/ggml?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)