MNN: Alibaba's Lightweight On-Device Inference Engine
What MNN is
MNN is Alibaba's deep learning inference engine, written in C++ and licensed under Apache-2.0. The README calls it "a highly efficient and lightweight deep learning framework" that "supports inference and training of deep learning models", with the stated focus on running models on the device itself: phones, PCs and embedded boards. The same README says MNN "has been integrated into more than 30 apps of Alibaba Inc, such as Taobao, Tmall, Youku, DingTalk, Xianyu", covering "more than 70 usage scenarios" such as live streaming, short video, search recommendation and product search by image. These are the project's own claims.
The workflow has three stages, which the quick-start overview spells out: train a model in another framework, convert it to MNN's own .mnn format with the converter, then load and run it with the MNN runtime on the target device. On top of that core engine the repository now ships two runtimes for generative models. MNN-LLM runs exported large language models locally, and MNN-Diffusion does the same for Stable Diffusion-style models.
The audience is mobile and embedded developers who need a small runtime with GPU and NPU paths, ML engineers who want to convert ONNX, TensorFlow, TFLite, Caffe or TorchScript models into one deployable format, and teams that want on-device LLM inference on Android, iOS or desktop. The README links an MLSys 2020 paper on the original engine and an OSDI 2022 paper on Walle, Alibaba's device-cloud ML system, which it says contains benchmarks against TensorFlow, TensorFlow Lite, PyTorch, PyTorch Mobile and TVM.
Most detailed documentation is in Chinese: the docs/ tree behind the Read the Docs site is almost entirely Chinese, and the README says the DingTalk groups are "predominantly Chinese" but welcome English speakers. This page uses the raw Markdown files from the repository.
How it works
Converter and the .mnn format
Models enter MNN through MNNConvert, a C++ tool that the Python wheel wraps as mnnconvert. The converter reference lists the source formats TF, CAFFE, ONNX, TFLITE, MNN, TORCH and JSON. The README counts the supported operators as 178 TensorFlow, 52 Caffe, 163 TorchScript and 158 ONNX ops. The tool optimises the graph while converting and can compress weights. --fp16 stores conv, matmul and LSTM weights as half precision. --weightQuantBits 2 to 8 quantises those weights, optionally per block with --weightQuantBlock, with the HQQ algorithm via --hqq, or asymmetrically, which is the default. The result is a FlatBuffers file; the acknowledgements credit flatbuffers. An MNN model can be dumped to JSON, edited, and converted back.
Runtime APIs
The docs recommend the high-level Module API, which supports control flow, over the low-level Interpreter/Session API, whose Python bindings are marked deprecated. A module loads with MNN.nn.load_module_from_file(path, inputs, outputs, runtime_manager=...). The runtime manager takes a small config dictionary. backend selects the hardware (0 CPU, 1 Metal, 2 CUDA, 3 OpenCL, 5 NPU, 7 Vulkan). numThread sets CPU threads (1 to 32). precision and memory each take 0 (normal), 1 (high) or 2 (low).
Those last two settings do most of the performance tuning. precision = 2 asks for FP16 compute where the hardware supports it. The Python guide names iPhone 8 and later and Arm Cortex-A75 and later, and promises "50%-100%" speed-ups there. memory = 2 on a weight-quantised model turns on what the docs call dynamic quantisation: weights stay in int8 or int4 form and are used in integer kernels, which cuts memory. The docs say it gives a "1-2x" speed-up when the model is compute-heavy and the CPU has sdot or smmla. On x86 the picture is different, as the hands-on section shows.
Backends and precision support
The README's support matrix rates each architecture and precision from S (deeply optimised) to C (unsupported). It covers x86 (SSE4.1, AVX2, AVX512), ARMv7a, ARMv8, ARMv9 with SME2 and RISC-V; OpenCL, Vulkan, Metal and CUDA GPUs; and CoreML, Hexagon, NeuroPilot, HIAI, NNAPI and QNN NPUs. Two rows matter for x86 users: FP16 is "C" (not supported) on every x86 CPU path, and AVX512 is rated "S" for FP32 and INT8. Many paths are compile-time options. The CMake reference shows MNN_AVX512, MNN_LOW_MEMORY, MNN_SUPPORT_BF16, MNN_KLEIDIAI and MNN_BUILD_LLM all defaulting to OFF, so a prebuilt package only does what it was built with.
Beyond inference
The same engine also powers MNN-Express (control flow and numpy-style computation), MNN-CV (an OpenCV-like image library of about 100 KB, per the README) and MNN-Train, which the docs index labels experimental. In Python these appear as MNN.expr, MNN.numpy, MNN.cv, MNN.nn, MNN.optim, MNN.llm and MNN.audio.
Key features in 3.6.2
- Multi-format conversion: ONNX, TensorFlow frozen graphs, TFLite, Caffe and TorchScript into
.mnn, with graph optimisation and a JSON round-trip for inspecting or patching models. - Post-training compression without data: FP16 weight storage (about 50% smaller) and 2 to 8-bit weight quantisation (75 to 87% smaller per the compression guide), with block-wise and HQQ variants and an
auto_quant.pyscript that picks per-layer settings against a test set. - Offline int8 quantisation with a calibration set (KL, ADMM or EMA), and dynamic quantisation at load time for weight-quantised models.
- Wide hardware coverage: x86 and ARM CPUs with hand-written SIMD and assembly, Metal, OpenCL, Vulkan and CUDA GPUs, and several NPU backends, with support levels published per precision.
- Small mobile footprint, per the README: about 800 KB for the Android core
.so(armv7a, c++_shared), about 12 MB for a full-option iOS static library, and roughly 25% less again withMNN_BUILD_MINI, which fixes input sizes. - MNN-LLM: export from Hugging Face or ModelScope checkpoints, 4-bit weights by default, an
llm_demochat tool, anllm_benchtool whose test modes are documented as aligned with llama.cpp'sllama-bench, and a PythonMNN.llmmodule. - MNN-Diffusion for local Stable Diffusion-style image generation, plus Android and iOS chat apps built on MNN-LLM.
Getting started
The Python quick start installs the wheel from PyPI:
pip install MNN
It then converts an ONNX model with 8-bit weight quantisation through the bundled converter:
mnnconvert -f ONNX --modelFile mobilenet_v1.onnx --MNNModel mobilenet_v1.mnn --weightQuantBits 8
The compression guide gives the C++ converter equivalents for FP16 storage, weight quantisation and block-wise quantisation:
./MNNConvert -f ONNX --modelFile model.onnx --MNNModel model_fp16.mnn --fp16
./MNNConvert -f ONNX --modelFile model.onnx --MNNModel model_quant.mnn --weightQuantBits 8
./MNNConvert ... --weightQuantBits 8 --weightQuantBlock 128
Offline int8 quantisation with a calibration JSON, per the same guide:
mnnquant model_float.mnn model_quant_int8.mnn quant.json
Loading a weight-quantised model with dynamic quantisation enabled, from the Python guide:
config['backend'] = 0
config['memory'] = 2
rt = MNN.nn.create_runtime_manager((config,))
net = MNN.nn.load_module_from_file(sys.argv[1], ["input"], ["MobilenetV1/Predictions/Reshape_1"], runtime_manager=rt)
Building the C++ converter and quantisation tool from source, and the build flag needed for dynamic quantisation in a C++ build:
cmake .. -DMNN_BUILD_CONVERTER=ON -DMNN_BUILD_QUANTOOLS=ON
cmake .. -DMNN_LOW_MEMORY=ON
benchmark.out, the testMNNFromOnnx.py correctness scripts and the LLM tools need a source build or checkout; they are not in the wheel.
Hands-on: MNN 3.6.2 on a Linux VM
Everything in this section was run on 22 November 2026 in a cloud VM: Ubuntu 24.04.5 LTS, Linux 6.18, an Intel Xeon at 2.10 GHz exposing AVX2, AVX512 and AVX512-VNNI, 2 vCPUs, about 7.8 GiB RAM and Python 3.13. Hugging Face and the PyTorch wheel index were blocked, so no pretrained model and no PyTorch were available. Instead, three ONNX models with random weights were built with the onnx package (opset 17):
- mbv2: a MobileNetV2-shaped CNN (1×3×224×224 in, 1,000 logits out, 3.49 M parameters).
- tfm: a 4-layer transformer encoder with d=256, 4 heads, a 1,024-wide feed-forward block, sequence length 128, LayerNormalization and erf-based GELU (3.16 M parameters).
- mlp: a batch-8 MLP, 1024→4096→4096→1000 (25.1 M parameters).
The reference is ONNX Runtime 1.32.0 from PyPI on the CPU execution provider. Latency is the median of repeated forward calls (about 2 s per configuration after 5 warm-up calls), each configuration in a fresh process, using the Module API with backend = 0. For much of the session, a process from another job on the same VM kept one of the two vCPUs busy. Single-thread numbers were stable across rounds. Two-thread numbers were stable only in an early uncontended window, so they are reported for mbv2 alone. All figures come from one machine and are only indicative.
Install footprint and tools
| Check | Result |
|---|---|
| Wheel | mnn-3.6.2-cp313-cp313-manylinux2014_x86_64, 18,187,277 bytes; the only dependency is numpy |
| Installed size | _tools extension (converter and quantiser) 35.0 MB, _mnncengine extension (engine) 24.1 MB, bundled libgomp 168 KB, Python package 260 KB |
| Console scripts | mnn (lists tools), mnnconvert, mnnquant, as the Python tools page documents. No benchmark or LLM binaries |
MNN.version() / mnnconvert --version | 3.6.2 / 3.6.2 |
| Import side effect | import MNN prints "The device supports: i8sdot:0, fp16:0, i8mm: 0, sve2: 0, sme2: 0, perfCores: 0" to stdout; the console tools print it twice |
| CPU paths in the engine | Symbols for _AVX2_ kernels are present; disassembling _mnncengine found no instructions that use zmm registers, so this wheel appears to have no AVX512 kernels even though the CPU supports them |
Process RSS after import MNN | about 43.7 MB, against 64.6 MB after import onnxruntime (the Python interpreter plus numpy alone measured 32.5 MB) |
The stdout banner gets in the way of scripts that emit JSON or CSV on stdout. The missing AVX512 path matters for x86 servers: the README rates AVX512 as the best x86 path, the LLM guide suggests -DMNN_AVX512=ON, and CMake defaults it to OFF, so a modern Xeon appears to need a source build.
Conversion and file sizes
All conversions used mnnconvert -f ONNX and printed "Converted Success!". Sizes are in bytes:
| Variant (converter flags) | mbv2 | tfm | mlp |
|---|---|---|---|
| ONNX source | 13,962,954 | 12,647,168 | 100,307,334 |
FP32 .mnn (no flags) | 13,963,836 (0.24 s) | 12,674,040 (0.24 s) | 100,311,728 (1.3 s) |
--fp16 | 7,026,700 (−50%) | 6,383,684 (−50%) | 50,176,872 (−50%) |
--weightQuantBits 8 | 3,715,804 (−73%) | 3,318,220 (−74%) | 25,183,704 (−75%) |
--weightQuantBits 8 --hqq | 3,715,804 (2.4 s) | 3,318,220 (1.5 s) | 25,183,704 (14.2 s) |
--weightQuantBits 4 | 1,968,204 (−86%) | 1,739,596 (−86%) | 12,649,224 (−87%) |
--weightQuantBits 4 --weightQuantBlock 128 | 2,044,812 | 1,862,476 | 14,142,412 |
Offline int8 (quantized.out path, 16 images) | 3,647,668 (3.6 s) | not run | not run |
The measured ratios match the compression guide: FP16 halves the file, 8-bit cuts about 75% and 4-bit about 87%. The README's summary line ("FP16 / Int8 quantize, can reduce model size 50%-70%") understates the 8-bit and 4-bit savings. HQQ produced files of identical size and took about 10 times longer to convert. mnnconvert -f MNN --info correctly printed the input name, NCHW layout, shape, output name and "Model Version: 3.6.2".
Numerical agreement with ONNX Runtime
The same random input went through ONNX Runtime and each MNN variant, single-threaded. Max abs is the largest absolute difference across outputs. Relative error is that maximum divided by the largest absolute reference value. Cosine is the cosine similarity of the flattened outputs.
| MNN variant / runtime setting | mbv2 (max abs, rel, cosine) | tfm | mlp |
|---|---|---|---|
FP32, precision 0, 1 or 2 | 9.5e-6, 1.8e-6, 1.000000 | 8.6e-4, 2.0e-4, 1.000000 | 6.2e-6, 1.1e-6, 1.000000 |
--fp16 storage | 0.050, 9.3e-3, 0.999962 | 3.2e-3, 7.6e-4, 1.000000 | 5.9e-3, 1.1e-3, 1.000000 |
| 8-bit weights | 0.28, 5.2e-2, 0.998428 | 0.048, 1.1e-2, 0.999939 | 0.081, 1.5e-2, 0.999908 |
| 8-bit weights + HQQ | 0.28, 5.2e-2, 0.998840 | 0.043, 1.0e-2, 0.999949 | 0.068, 1.2e-2, 0.999924 |
8-bit weights, memory 2 (dynamic quant) | 0.32, 6.0e-2, 0.998242 | 0.072, 1.7e-2, 0.999857 | 0.115, 2.1e-2, 0.999814 |
4-bit weights, memory 2 | 1.18, 0.22, 0.971463 | 0.75, 0.18, 0.980961 | 1.18, 0.21, 0.974746 |
4-bit, block 128, memory 2 | 1.06, 0.20, 0.974362 | 0.64, 0.15, 0.987706 | 0.94, 0.17, 0.986121 |
| Offline int8 | 0.57, 0.11, 0.993641 | n/a | n/a |
FP32 conversion is numerically faithful. The transformer's larger absolute difference (about 1e-3) comes from LayerNorm, erf and softmax and is still at float rounding level. On x86, precision = 2 gave outputs bit-identical to precision = 0, which matches the README's statement that x86 has no FP16 path. The low-precision setting changes nothing on this class of hardware. Weight quantisation behaved as documented: 8-bit is close to the original, HQQ improves it slightly, block-wise 4-bit beats plain 4-bit, and plain 4-bit without calibration loses a lot on these random-weight networks. Random weights are harsher than trained ones, so treat the 4-bit rows as a worst case. The int8 model was calibrated on synthetic images, so its error is only a smoke test.
Latency and memory
Median single-thread latency in milliseconds (range across rounds in brackets), with peak process RSS after load plus 20 forward passes:
| Engine and setting | mbv2 ms | tfm ms | mlp ms | Peak RSS mbv2 / mlp |
|---|---|---|---|---|
| ONNX Runtime 1.32.0, FP32 | 7.1 (6.4–11.3) | 9.3 (7.2–11.6) | 9.2 (8.3–17.1) | 118 / 267 MB |
| MNN FP32 | 11.4 (10.8–14.4) | 14.1 (13.5–22.2) | 12.2 (9.9–25.7) | 84 / 237 MB |
MNN FP32, precision 2 | 11.8 | 14.0 | 12.9 | same as FP32 |
MNN --fp16 storage | 12.9 | 14.4 | 12.0 | 76 / 291 MB |
MNN 8-bit weights, memory 0 | 14.2 | 14.9 | 14.1 | 77 / 231 MB |
MNN 8-bit weights, memory 2 | 26.2 (19.8–36.1) | 15.1 | 6.6 (4.9–8.8) | 61 / 124 MB |
MNN 4-bit weights, memory 2 | 21.9 | 16.2 | 4.1 (3.8–6.6) | 57 / 83 MB |
| MNN offline int8 | 15.9 (15.7–16.4) | n/a | n/a | 58 MB |
In the uncontended window at the start of the session, mbv2 scaled from 1 to 2 threads as follows: ONNX Runtime 9.4 → 5.2 ms, MNN FP32 11.5 → 7.3 ms, MNN 8-bit weights 11.2 → 6.6 ms, and MNN 8-bit with dynamic quantisation 32.1 → 12.0 ms. Loading took about 30 ms for FP32 mbv2 and 170 to 210 ms for the 100 MB MLP (17 to 23 ms at 4-bit); the first forward pass cost about twice a steady-state call.
Four conclusions for this x86 setup. First, on FP32 the wheel was 1.3 to 1.6 times slower than ONNX Runtime single-threaded, consistent with the missing AVX512 kernels; ARM phones, MNN's main target, could not be tested. Second, MNN used less memory than ONNX Runtime for the same FP32 model (84 vs 118 MB on mbv2), and weight quantisation with memory = 2 cut the MLP's peak RSS from 237 MB to 83 MB. Third, dynamic quantisation is workload-dependent. On the bandwidth-bound MLP it made MNN the fastest option tested (4.1 ms at 4-bit vs 9.2 ms for ONNX Runtime FP32). On the convolution-heavy mbv2 it more than doubled latency, and the offline int8 model was also slower than FP32. The FAQ explains this directly: on x86 without VNNI kernels, int8 math widens to int16 before accumulating and "is not as efficient as floating point", and VNNI needs MNN_AVX512 at build time. Fourth, --fp16 storage did not speed anything up, as the FAQ says (weights are expanded to FP32 at load). It also raised the MLP's peak RSS to 291 MB, presumably from holding both copies during decompression.
What failed or differed from the docs
mnnquantcrashes in the wheel. Everymnnquant src.mnn dst.mnn quant.jsoncall aborted within 0.1 s with "terminate called after throwing an instance of 'std::length_error' … cannot create std::vector larger than max_size()". It never opened the model or the config. The 3.6.2 source shows why:MNN/tools/mnnquant.pycallsTools.mnnquant(src_mnn, dst_mnn, config_json)with three strings, while the C binding inpymnn/src/MNNTools.ccexpects one argv-style list. Calling the extension directly with a list (_tools.mnnquant(["quantized.out", src, dst, cfg])) worked and produced the int8 model above. The same crash occurred with the 3.6.1 wheel. The compression guide nevertheless presentsmnnquantas working afterpip install MNN.- Telemetry in older wheels. As a comparison, the 3.6.1 wheel was installed in a separate venv. Its first
mnnquantrun executedpip install -U aliyun-log-python-sdkby itself, pulling in about 20 packages, and then tried to reach an Alibaba Cloud Function Compute host incn-hangzhou; the VM's proxy refused the connection. The 3.6.1log.pyin the repository shows the intent. It fetches temporary credentials, then uploads to an Aliyun log store whatever the wrappers pass in (converter or quantiser arguments, model sizes and a model UUID), plus the OS and a machine ID read from/etc/machine-idon Linux. In 3.6.2 the same module is a stub whose docstring reads "Cloud telemetry is disabled. Importing this module does not fetch credentials, install packages, or collect machine information", and no such activity was seen with the 3.6.2 wheel. PinMNN>=3.6.2for the Python tools. - Licence metadata.
LICENSE.txtis Apache License 2.0 and the README says "Apache 2.0", but the PyPI metadata andpymnn/pip_package/setup.pydeclareBSD-3with the "BSD License" classifier. Licence scanners that read wheel metadata will report BSD-3. The repository's licence file is the authoritative one.
MNN-LLM and MNN-Diffusion (from the docs, not tested)
Nothing in this section was run. Every route needs a model download from Hugging Face or ModelScope, and Hugging Face was blocked from the VM. The description comes from docs/transformers/llm.md, transformers/README.md and docs/pymnn/llm.md.
MNN-LLM is split into export and inference. Export runs llmexport.py on a local checkpoint. It writes llm.mnn, an external llm.mnn.weight file, a tokenizer, llm_config.json and a runtime config.json. Weights default to 4 bits with block size 64, and --hqq, --awq or --omni select quantisation algorithms. LoRA merging and GPTQ weights are supported. A gguf2mnn.py script converts GGUF files, the format used by llama.cpp, but not for multimodal models. The guide's export example:
python llmexport.py \
--path /path/to/Qwen2-0.5B-Instruct \
--export mnn --hqq
Inference needs MNN built with -DMNN_BUILD_LLM=ON, or the Python package, whose MNN.llm module was present in the 3.6.2 wheel and exposes create, Llm and Context. The command-line tools:
./llm_demo /path/to/model_dir/config.json
./llm_bench -m ./model/config.json -a cpu,opencl -t 4,8 -pg 64,32 -rep 3
From Python, per the module docs:
import MNN.llm as mllm
model = mllm.create("./exported-model/config.json")
model.set_config({"max_new_tokens": 64})
model.load()
response = model.response("Hello, how are you?")
print(response)
The guide documents Flash Attention (on by default in llm_bench), KV-cache int8 quantisation, token compression for long prompts, a NeedleBench long-context test, an Android Hexagon backend and multimodal prompts with <img> and <audio> tags. Its sample llm_bench table shows Qwen3-0.6B on Metal at about 5,073 tokens/s prefill and 346 tokens/s decode. The README's November 2026 news item claims 350 tokens/s decode for the new MNN-Pocket-0.3B INT4 build on an iPhone 18 Pro. Both are vendor numbers. The docs also disagree on file names. The English transformers/README.md lists embeddings_bf16.bin and tokenizer.txt among the exported files. The Chinese guide lists tokenizer.mtok and says embeddings_bf16.bin is no longer exported by default for models under 8B, which reuse the LM head weights unless --seperate_embed is passed.
Use cases
- Shipping a vision or audio model inside a mobile app, where binary size and GPU or NPU access through one API matter more than desktop throughput.
- Converting a mixed model zoo (ONNX, TFLite, TorchScript, older Caffe) into one runtime format for Android, iOS and embedded Linux.
- Shrinking memory for large fully connected layers with 4- or 8-bit weights and
memory = low; the MLP test cut RSS to a third and latency by more than half. - On-device LLM chat on phones and laptops with MNN-LLM, especially for teams already using MNN for other models.
- Embedded and IoT targets with POSIX, RISC-V vector or Qualcomm Hexagon hardware, according to the support matrix.
How MNN compares
Only ONNX Runtime was run here, as the reference above; the other rows rest on each project's README and licence file. Check current docs before relying on a row.
| Aspect | MNN 3.6.2 | ONNX Runtime | OpenVINO | ncnn | LiteRT (formerly TensorFlow Lite) | llama.cpp |
|---|---|---|---|---|---|---|
| Primary focus | On-device inference across mobile, PC and embedded; LLM and diffusion runtimes on top | "A performance-focused scoring engine" for ONNX models (PyPI description) | Optimising and deploying models from edge to cloud; CPU (x86, ARM), Intel GPU and NPU per its README | Inference "optimized for mobile, embedded, and desktop deployment" with no third-party runtime dependencies | Google's "on-device runtime for high-performance ML & GenAI deployment on edge platforms" | "LLM inference in C/C++" on the ggml tensor library |
| Model input | Converts ONNX, TF, TFLite, Caffe, TorchScript to .mnn | ONNX directly | PyTorch, TensorFlow, ONNX, Keras, PaddlePaddle, JAX/Flax per README | PyTorch or ONNX through pnnx to .param/.bin | PyTorch, TensorFlow and JAX per README | GGUF models |
| GPU / NPU | Metal, OpenCL, Vulkan, CUDA; CoreML, Hexagon, NNAPI, QNN, HIAI, NeuroPilot | Execution providers (not reviewed here) | Intel GPU and NPU | Vulkan GPU | GPU and NPU acceleration through its Compiled Model API | Many backends (not reviewed here) |
| Measured here (x86, 1 thread, FP32 mbv2) | 11.4 ms, 84 MB peak RSS | 7.1 ms, 118 MB peak RSS | not tested | not tested | not tested | not applicable |
| Licence | Apache-2.0 (wheel metadata says BSD-3) | MIT (PyPI metadata) | Apache-2.0 | BSD 3-Clause, with third-party components under other licences | Apache-2.0 | MIT |
ONNX Runtime is the obvious alternative on servers and desktops. It reads ONNX without a conversion step, and on this x86 VM it beat the MNN wheel on FP32 latency for all three models. MNN's advantages in this test were lower memory and, for matmul-heavy models, weight-only quantisation with dynamic int8 or int4 compute. OpenVINO is the natural choice when the target is Intel CPUs, GPUs or NPUs. ncnn, from Tencent, is MNN's closest peer in purpose: a dependency-free mobile engine with Vulkan GPU support, which MNN's README credits among the projects it drew on. LiteRT continues TensorFlow Lite, with GenAI support through LiteRT-LM. For LLMs specifically, llama.cpp on ggml is the reference that MNN's own llm_bench lines up with, and MNN can import its GGUF files. For running open models locally on a desktop, Ollama is the more direct tool; high-throughput GPU serving is vLLM territory.
Things to know before adopting
- Prebuilt packages are not tuned builds. The PyPI wheel showed AVX2 kernels and no AVX512 code. Dynamic quantisation, BF16, KleidiAI, LLM support and GPU backends all depend on CMake options that default to
OFF. For production on a given platform, plan to build MNN yourself and benchmark that build. - Speed claims are ARM-centric. The guide's 50–100% FP16 and 1–2× dynamic-quantisation gains rely on ARM features; on x86, FP16 compute does not exist and int8 was slower than FP32 for a CNN. Judge MNN on the devices you ship to.
- Dynamic quantisation can hurt.
memory = lowmore than halved MLP latency and memory but doubled CNN latency. Measure per model before turning it on globally. - Quantise with a check. Plain 4-bit weights cost real accuracy here. Use block-wise quantisation, HQQ,
auto_quant.pywith an error budget, or thetestMNNFromOnnx.pycomparison script. - Python tooling rough edges.
mnnquantfrom the 3.6.2 wheel crashes,import MNNwrites a banner to stdout, and the 3.6.1 wheel sent converter telemetry to Alibaba Cloud and installed a package at runtime. Earlier versions were not checked. Use 3.6.2 or later, and audit any older pinned version in CI images. - Documentation is mostly Chinese, and English and Chinese pages sometimes disagree (see the LLM export file names above).
- Model protection is limited. The FAQ admits that device-side encryption cannot be absolutely secure. Its suggestion is to build a private schema by reordering the FlatBuffers definitions and recompiling MNN and its tools.
- Licence is Apache-2.0, with a patent grant; mind the BSD-3 wheel metadata if compliance tooling reads package metadata.
Project activity
Checked 22 November 2026: latest release 3.6.2 (published on GitHub on 29 October 2026 per the GitHub releases API; the wheels reached PyPI the same day), licence Apache-2.0. The README's news section still names 3.6.1 (22 July 2026, Hexagon backend) as the latest engine release announcement, and its newest item is the MNN-Pocket-0.3B model from 4 November 2026. PyPI shows a steady cadence through 2026: 3.4.0 in February, 3.4.1 in March, 3.5.0 in April, 3.6.0 in June, 3.6.1 in July and 3.6.2 in October.
The alibaba/MNN repository had about 16,600 stars (16,592 at the time of writing) and 2,545 forks. It is written in C++, was created on 15 April 2019, develops on master and was last pushed on 20 November 2026. The GitHub metadata lists no homepage; the README points to mnn.zone for the MNN Workbench and to Read the Docs for documentation.
Enjoying this project?
Discover more amazing open-source projects on TechLogHub. We curate the best developer tools and projects.
Repository:https://github.com/alibaba/MNN
GitHub - alibaba/MNN: MNN: Alibaba's Lightweight On-Device Inference Engine
MNN is Alibaba's Apache-2.0 on-device inference engine. Version 3.6.2 tested on x86: ONNX conversion, accuracy vs ONNX Runtime, FP16 and 4/8-bit weights, dynami...
github - alibaba/mnn
Related Projects
llama.cpp: LLM inference in plain C/C++
Dependency-free C/C++ LLM inference with quantization, many GPU backends and an OpenAI-compatible server.
Claude-Mem: Persistent Memory for AI Coding Agents
Persistent cross-session memory for Claude Code, OpenCode, Codex and other agents, with SQLite and MCP search.
vLLM: High-Throughput LLM Inference and Serving Engine
Python engine for fast LLM inference and OpenAI-compatible serving, built around PagedAttention.
Maintain MNN: Alibaba's Lightweight On-Device Inference Engine? Add the badge to your README
Show that the project is featured on TechLogHub and send readers to this write-up.
[](https://techloghub.com/open-source/alibaba-mnn?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)