ggml: Tensor Library for Machine Learning in C/C++
GitHub Repo
MIT
November 17, 2026 at 01:41 PM
0 views

ggml: Tensor Library for Machine Learning in C/C++

@ggml-orgProject Author

What ggml is

ggml is a tensor library for machine learning written in C and C++. Its README states the goal in one sentence: "to be a simple, portable, and efficient tensor library for machine learning with minimal setup." It is the layer underneath llama.cpp and whisper.cpp: the llama.cpp README says that project "is built on top of the ggml library", and the whisper.cpp README says the high-level model code lives in whisper.h and whisper.cpp while "the rest of the code is part of the ggml machine learning library". Anyone who has run a GGUF model locally has, directly or indirectly, run ggml.

The README lists six headline properties: a plain C/C++ implementation without dependencies; cross-platform builds for x86, ARM, RISC-V, LoongArch, PowerPC, s390x and WebAssembly; SIMD-optimised kernels for x86, ARM and RISC-V; backends for CPU, GPU, NPU and browser; 2- to 8-bit integer quantization plus the MXFP4 and NVFP4 microscaling formats; and "zero memory allocations during runtime". It is not a training framework in the PyTorch sense, nor a model server. It is the piece a C or C++ program links to when it wants to define a network as a graph of tensor operations, load quantized weights and run that graph on whatever hardware is present.

The repository, ggml-org/ggml, was created on 18 September 2022 and is MIT-licensed, with the copyright line "Copyright (c) 2023-2026 The ggml authors". The auto-generated AUTHORS file, last regenerated on 4 November 2026, lists roughly 950 contributors. The project tagged v1.0.0 on 13 November 2026.

How it works: contexts, tensors and computation graphs

Contexts and memory

Everything in ggml starts with a context. ggml_init() takes a ggml_init_params struct with three fields: mem_size in bytes, an optional mem_buffer ("if NULL, memory will be allocated internally") and a no_alloc flag that tells the context not to allocate memory for tensor data. Tensor metadata and graphs are carved out of that pool, which is how the library keeps its promise of no allocations at runtime: the memory is reserved up front and reused. With no_alloc set, the context only holds metadata and the actual data lives in backend buffers, which is the pattern the newer examples use.

Tensors

The ggml_tensor struct in include/ggml.h is small and readable. It carries a type (one of the ggml_type values such as GGML_TYPE_F32, GGML_TYPE_F16, GGML_TYPE_Q4_0 or GGML_TYPE_Q8_0), a pointer to the backend buffer, and two arrays of length GGML_MAX_DIMS, which is 4: ne, the number of elements per dimension, and nb, the stride in bytes. The header comment spells out the stride rule: nb[0] = ggml_type_size(type), nb[1] = nb[0] * (ne[0] / ggml_blck_size(type)) + padding, and each higher stride is the previous one times the previous element count. Dimension 0 is the innermost, contiguous one, which is the reverse of the row-major shape order NumPy and PyTorch print.

A tensor also records the operation that produces it (op), up to GGML_MAX_SRC (10) source tensors, operation parameters, flags that mark it as a graph input, output, trainable parameter or loss, and a view_src/view_offs pair for views into other tensors. Calling an operation such as ggml_add or ggml_mul_mat does not compute anything; it returns a new tensor node whose sources point at the operands.

Graphs and execution

Computation is deferred. A program builds the forward graph by calling ggml_build_forward_expand(graph, result), which walks the sources of result and adds them to a ggml_cgraph; ggml_build_backward_expand extends a graph with gradients for training. Graphs come from ggml_new_graph (default size GGML_DEFAULT_GRAPH_SIZE, 2048 nodes) or ggml_new_graph_custom. They can be printed with ggml_graph_print or dumped to Graphviz with ggml_graph_dump_dot, which is useful when debugging a model port. Execution is a separate step: the backend API's ggml_backend_graph_compute(backend, graph) runs the whole graph on one backend and returns a ggml_status.

Backends and the scheduler

include/ggml-backend.h describes a layered model: buffer types and buffers (memory on a device), backends (an execution stream), devices (with a type of CPU, GPU, integrated GPU, accelerator or a "meta" device that wraps several others for tensor parallelism) and a registry that can load backends from dynamic libraries via ggml_backend_load_all(). ggml_backend_init_best() picks a GPU device if one is available and falls back to CPU.

For mixed hardware there is a backend scheduler. The header says it "allows for multiple backend devices to be used together" and "handles compute buffer allocation, assignment of tensors to backends, and copying of tensors between backends", choosing a backend for each operation based on which backend supports it and where pre-allocated tensors such as weights already live. Backends with a lower index get priority. This is how an example can request CUDA or Metal and still run operations that the GPU backend does not implement on the CPU, a fallback the MNIST README warns "may result in bad performance".

Quantized blocks

Quantized types store values in fixed-size blocks rather than one value at a time. In src/ggml-common.h, block_q8_0 is a 16-bit ggml_half scale d followed by int8_t qs[32], so 32 weights take 34 bytes. block_q4_0 is the same scale followed by uint8_t qs[16], two 4-bit values per byte, so 32 weights take 18 bytes. Both definitions carry static_asserts on the block size. The header also defines the K-quants (Q2_K to Q8_K), the IQ family, ternary TQ1_0/TQ2_0, MXFP4 and NVFP4. A comment on the ggml_type enum says new types are always appended "to keep backward compatibility", and several retired entries are left in place as comments marked "support has been removed".

Key features

  • No runtime dependencies. The core is C11 and C++17 (set in CMakeLists.txt) with only a threads library required; BLAS, CUDA, Vulkan and the rest are opt-in.
  • Deferred graph execution. Operations build a graph; nothing runs until the graph is computed, which lets the allocator and scheduler plan memory and device placement for the whole graph.
  • Wide quantization coverage. Classic Q4/Q5/Q8 block formats, K-quants, IQ formats, ternary types, MXFP4, NVFP4 and FP8 E4M3 alongside F32, F16 and BF16.
  • Many backends behind one API. The src/ tree at the tested commit holds backend directories for BLAS, CANN, CPU, CUDA, HIP, Hexagon, Metal, MUSA, OpenCL, OpenVINO, RPC, SYCL, Vulkan, WebGPU, zDNN, ZenDNN, a virtual GPU backend and one named ggml-et.
  • Backend scheduler with CPU fallback. One graph can be split across devices, with unsupported operations falling back to a lower-priority backend.
  • Automatic differentiation and optimisers. ggml_build_backward_expand and the mnist-train example show training on top of the same graph, though the MNIST README calls training "a work-in-progress and not production ready".
  • Build-time CPU tuning. GGML_NATIVE (on by default unless cross-compiling) targets the build machine; GGML_BACKEND_DL with GGML_CPU_ALL_VARIANTS builds several CPU variants as loadable libraries for portable binaries.

Getting started

The README's quick start builds from source with CMake:

git clone https://github.com/ggml-org/ggml
cd ggml

mkdir build && cd build
cmake ..
cmake --build . --config Release -j 8

CMakeLists.txt requires CMake 3.14 or newer and turns on GGML_BUILD_TESTS and GGML_BUILD_EXAMPLES when ggml is the top-level project, so this produces the test binaries and examples as well as the libraries. On Apple platforms GGML_METAL and GGML_BLAS (with the Accelerate vendor) default to on; elsewhere GPU backends are off until enabled with options such as GGML_CUDA, GGML_VULKAN, GGML_HIP, GGML_SYCL or GGML_WEBGPU.

The README points first to examples/simple, "a minimal, fully commented example (matrix multiplication)". It ships in two forms: simple-ctx, which keeps everything in a ggml context and "doesn't support gpu acceleration", and simple-backend, which "demonstrates how to use other backends like CUDA and Metal". After that, the MNIST example is the most complete tour. Its README trains a small network in PyTorch, saves it as GGUF, and evaluates it with ggml from inside examples/mnist:

$ python3 mnist-train-fc.py mnist-fc-f32.gguf
$ ../../build/bin/mnist-eval mnist-fc-f32.gguf data/MNIST/raw/t10k-images-idx3-ubyte data/MNIST/raw/t10k-labels-idx1-ubyte

The same README shows training directly in ggml:

$ ../../build/bin/mnist-train mnist-fc mnist-fc-f32.gguf data/MNIST/raw/train-images-idx3-ubyte data/MNIST/raw/train-labels-idx1-ubyte

The GPT-2 example covers a full transformer with weight quantization. Its README converts F16 weights to Q4_0 with the bundled quantizer:

./bin/gpt-2-quantize models/gpt-2-1558M/ggml-model-f16.bin models/gpt-2-1558M/ggml-model-q4_0.bin 2

Note that this example still uses the older single-file ggml-model.bin format rather than GGUF, and that its README shows a ./bin/gpt-2 binary, while the build at the tested commit produced gpt-2-ctx, gpt-2-alloc, gpt-2-backend, gpt-2-sched and gpt-2-batched instead.

Hands-on: building ggml 1.0.0 and running its tests on arm64

The build and tests below ran on 17 November 2026 on a Linux arm64 virtual machine with 4 vCPUs and about 3.9 GB of RAM on an Apple Silicon host, using GCC 11.4.0 on Ubuntu 22.04 and CMake 4.4.5 (installed from PyPI because the VM had no CMake). The source was a shallow clone of master at commit d9fe7bf, committed on 16 November 2026.

Configure and build

StepCommandResult
Configurecmake -B build -DCMAKE_BUILD_TYPE=Release0.66 s on a re-run; printed ggml version: 1.0.0, ggml commit: d9fe7bf, ARM detected and Adding CPU backend variant ggml-cpu: -mcpu=native
Build (clean, separate build directory)cmake --build build2 --config Release -j427 s wall clock, exit code 0
Shared libraries-libggml-base.so 850,808 B, libggml-cpu.so 919,248 B, libggml.so 48,800 B
Executables in build/bin-38 binaries, about 4.9 MB: 15 examples (gpt-2-*, gpt-j, gpt-j-quantize, magika, mnist-eval, mnist-train, sam, simple-backend, simple-ctx, yolov3-tiny) and about 23 test-* programs

The split into libggml-base (core and graph code), libggml-cpu (the CPU backend) and a thin libggml reflects the backend registry design: a GPU build adds further backend libraries next to these rather than growing one monolith.

CPU feature probes

The configure step probes the compiler for Arm extensions. On this VM the results did not line up with what the CPU advertises:

ProbeCMake resultRelated flag in /proc/cpuinfo
HAVE_FMASuccess-
HAVE_DOTPRODFailedasimddp present
HAVE_MATMUL_INT8Failedi8mm present
HAVE_FP16_VECTOR_ARITHMETICFailedfphp, asimdhp present
HAVE_SVEFailed-
HAVE_SMEFailed-

The cause was not investigated further. With GCC 11 and -mcpu=native in this VM the probes did not enable these paths; a newer compiler or explicit architecture flags may give a different result. The practical point is to read the configure output on any new build machine rather than assume the native build picked up every extension.

The transposed mul_mat convention

Both ./bin/simple-ctx and ./bin/simple-backend printed the same result:

Output headerRow 1Row 2Row 3
mul mat (4 x 3) (transposed result)60.00 55.00 50.00 110.0090.00 54.00 54.00 126.0042.00 29.00 28.00 64.00

This matches examples/simple/README.md. A conventional 4x2 by 2x3 product gives a 4x3 matrix whose first column is 60, 55, 50, 110. In ggml, "we pass the matrix B in transposed form and multiply row-by-row. The result C is also transposed": ggml_mul_mat(A, B^T) = C^T. The comment above ggml_mul_mat in ggml.h says the same in shape terms: A has k columns and n rows, B has k columns and m rows ("we transpose it internally"), and the result has n columns and m rows. Both inputs share their innermost dimension ne[0], which keeps the dot products contiguous in memory. Code ported from NumPy, where A @ B pairs columns with rows, is where this convention usually bites.

Operator tests

CommandResult
./bin/test-quantize-fnsExit 0, no failures; covered types include tq2_0, mxfp4, nvfp4, q1_0, q2_0 and f8_e4m3, plus "degenerate imatrix" checks for q2_K, q4_K, q5_K, q4_1 and q5_1
./bin/test-backend-ops test -o ADD -b CPU173/173 tests passed, 1/1 backends passed
./bin/test-backend-ops --list-opsTotal: 127 operations

test-backend-ops --help lists four modes: test (compare a backend's results with the CPU backend), grad (check gradients with finite differences), perf and support. Options include -o to filter by operation, -b to pick a backend by name (for example CPU, MTL0 or CUDA0), -p for a parameter regex, --output with console, sql or csv, and -j for parallel workers. For anyone evaluating a GPU backend, support mode is the quickest way to see which of the 127 operations it implements before relying on it.

One decode-step matrix-vector product

To see what quantization does on the CPU backend, test-backend-ops perf timed a 4096 x 14336 weight matrix multiplied by a single f32 vector, the shape of one feed-forward projection during single-token decoding (117.44 MFLOP per run). The command was ./bin/test-backend-ops perf -o MUL_MAT -b CPU -p 'type_a=(f32|f16|q8_0|q4_0),type_b=f32,m=4096,n=1,k=14336', and the tool reported 3906 MB of device memory. Weight sizes are computed from the block layouts above (4 bytes per value for f32, 2 for f16, 34 bytes per 32 values for q8_0, 18 bytes per 32 values for q4_0).

Weight typeTime per runGFLOPSWeight size
f322065.05 us56.87234.9 MB
f162226.71 us52.74117.4 MB
q8_01116.43 us105.1962.4 MB
q4_01335.75 us87.9233.0 MB

This is a single run on a single 4-vCPU virtual machine, so treat it as an illustration, not a benchmark. On this VM q8_0 was the fastest, about 1.85 times quicker than f32 while using roughly a quarter of the memory, and q4_0 came second despite being half the size of q8_0. f16 was not faster than f32, which is consistent with the FP16 vector arithmetic probe failing at configure time. The q4_0 result should not be generalised either: GGML_CPU_REPACK ("use runtime weight conversion of Q4_0 to Q4_X_X") is on by default, and the extension probes above failed, so a different compiler or CPU could reorder these rows.

A measurement gotcha

./bin/test-quantize-perf --type q4_0 --op vec_dot_q -i 20 reported min cycles/32 vals: 0.00 and avg cycles/32 vals: 0.00 on this arm64 VM, because it had no usable cycle counter. Only the throughput lines from that tool were meaningful there, and the small 4096-value tests that fit in L1 cache produced noisy numbers. On virtualised Arm hosts, test-backend-ops perf with wall-clock timing is the more trustworthy of the two.

GGUF: the model format

docs/gguf.md defines GGUF as "a file format for storing models for inference with GGML and executors based on GGML", designed for fast loading and saving. Models are "traditionally developed using PyTorch or another framework, and then converted to GGUF". It succeeds the earlier GGML, GGMF and GGJT formats; the key change, according to the spec, is a typed key-value structure for metadata instead of a list of untyped hyperparameters, so new fields can be added without breaking existing readers.

The layout, summarised at the top of include/gguf.h, is: the magic GGUF, a version (currently 3), tensor and key-value counts, the key-value pairs, a tensor-info table (name, dimensions, ggml_type, offset) and an aligned data blob. Alignment defaults to 32 bytes unless general.alignment says otherwise, which is what makes mmap loading possible. The spec says files are little-endian by default, that version 3 added big-endian support, and that "there is no way to determine if a model is big-endian" from the file itself. general.architecture is required, and general.quantization_version must be present if any tensor is quantized.

The spec also sets a filename convention, <BaseName><SizeLabel><FineTune><Version><Encoding><Type><Shard>.gguf with optional mmproj or mtp sidecar prefixes, and gives a validation regex. One limit is worth knowing: GGUF stores weights and metadata, not the computation graph. The spec lists embedding a graph as "a future extension" whose "primary blocker is the stabilization of the computation graph format", so every executor still has to implement each model architecture in code.

Use cases

  • Building a local inference engine for a new model family. This is what llama.cpp and whisper.cpp do, and what the gpt-2, gpt-j, SAM, YOLO and Magika examples in the repository demonstrate at smaller scale.
  • Embedding inference in a native application. A C API, no Python runtime and no required dependencies make ggml suitable for desktop apps, mobile apps and command-line tools. Projects such as Jan credit llama.cpp, and LocalAI's README says the team wrote 18 of its own "native C, C++ and GGML engines with no Python at inference".
  • Running quantized models on CPUs and modest GPUs. Block quantization cut the weight footprint of the test matrix above from 234.9 MB to 33 MB, which is the main reason ggml-based tools fit large models into laptop memory.
  • Browser and edge targets. The MNIST README shows compiling the evaluation code to WebAssembly with Emscripten, and the backend list includes WebGPU, Hexagon and other NPU targets.
  • Writing and testing new kernels. test-backend-ops gives backend authors a ready-made correctness and performance harness against the CPU reference.

How ggml compares

The table positions ggml against libraries it is often weighed against. Each row reflects only what that project's own README says; "not stated" means the README read did not say, not that the feature is missing.

ProjectLanguage and APIPrimary target, per its READMEModel inputQuantization in READMELicence
ggml 1.0.0C/C++ library with a C API; simple autogenerated cffi Python bindings in examples/python"Tensor library for machine learning" with minimal setup, on CPU, GPU, NPU and browserGraphs built in code; weights usually from GGUF2- to 8-bit integer, MXFP4, NVFP4MIT
ONNX RuntimeCross-platform runtime"Cross-platform inference and training machine-learning accelerator" for models from PyTorch, TensorFlow/Keras, scikit-learn and othersONNX modelsNot stated in READMEMIT; README notes it may collect usage data
PyTorchPython first, with C/C++ extension APITensor computation "with strong GPU acceleration" and "tape-based autograd"Models defined in PythonNot stated in READMEBSD-style
MLXPython API following NumPy, plus C++, C and Swift"Array framework for machine learning on Apple silicon", lazy evaluation, unified memoryModels defined in codeNot stated in READMEMIT (LICENSE file)
Apache TVMPython-first compiler"Open machine learning compilation framework" producing "minimum deployable modules"Models compiled through its pipelineNot stated in READMEApache-2.0

The broad split is between frameworks that take a model from elsewhere (ONNX Runtime with ONNX files, TVM through compilation) and libraries where the model is written in code (PyTorch, MLX, ggml). Within the second group, ggml is the only one here aimed at C and C++ programs with no Python in the loop, and the only one whose README leads with low-bit quantization. PyTorch is the natural choice for research and training; MLX targets Apple silicon specifically. On the serving side, vLLM lists GGUF among the quantization formats it accepts, so ggml's format reaches beyond ggml's own runtime.

Limits and when not to use it

  • Not a training framework. Autodiff and optimisers exist, but the MNIST README says training is "a work-in-progress and not production ready". Train in PyTorch or similar and convert.
  • Low-level API. Every model architecture has to be written by hand as graph construction code; GGUF carries no graph. For running an existing LLM, llama.cpp is the higher-level entry point.
  • Backend coverage varies. The MNIST README warns that if a backend lacks an operation, "a CPU fallback is used instead which may result in bad performance". Some CMake options are labelled "experimental, unstable" (MUSA graphs) or "llama.cpp only" (CUDA graphs). Check test-backend-ops support for the target backend.
  • Native builds can miss CPU features. In the hands-on run, GCC 11 probes failed for dot-product, int8 matrix multiply and FP16 arithmetic even though the CPU listed them, and f16 ran no faster than f32.
  • Contributions go elsewhere. The README and CONTRIBUTING.md ask that changes to core ggml, including the CMake build, be opened as pull requests in llama.cpp, "doing so will make your PR more visible, better tested, and more likely to be reviewed". Review of core changes therefore happens in the llama.cpp repository, which matters for anyone planning to upstream a fix.
  • API stability is not spelled out. The project moved from frequent 0.x tags (v0.28.0 on 28 October, v0.29.0 on 4 November 2026) to v1.0.0 on 13 November 2026. Neither the README nor CONTRIBUTING.md states a compatibility policy for 1.x; the only explicit rule found is the append-only ggml_type enum. Pin a tag or commit.
  • Uneven documentation. The README links a Hugging Face introduction and a llama.cpp wiki page for most conceptual material. Several example READMEs are dated: the GPT-2 README uses a non-GGUF .bin format and a binary name the current build does not produce, and the Python bindings README builds against llama.cpp with older LLAMA_* CMake flags.

Project activity

Checked 17 November 2026: latest release v1.0.0 (13 November 2026), MIT licence, about 16,100 stars (16,098) and 2,075 forks on the ggml-org/ggml repository, per the GitHub API. The repository was created on 18 September 2022, is written mainly in C++, develops on master and is not archived; the most recent commit at the time of testing, d9fe7bf, was made on 16 November 2026. The GGUF specification is in docs/gguf.md, and the sibling projects llama.cpp and whisper.cpp live in the same ggml-org organisation.

Enjoying this project?

Discover more amazing open-source projects on TechLogHub. We curate the best developer tools and projects.

Project
ggml
Created
November 17
Last Updated
November 17, 2026 at 01:41 PM

Maintain ggml: Tensor Library for Machine Learning in C/C++? Add the badge to your README

Show that the project is featured on TechLogHub and send readers to this write-up.

Featured on TechLogHub
[![Featured on TechLogHub](https://techloghub.com/badge-neutral.svg)](https://techloghub.com/open-source/ggml?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)

Find more projects like this

One email a week: new and trending developer tools, fresh comparisons, and what shipped. Unsubscribe in one click.