OpenVINO: Open-Source Toolkit for Deep Learning Inference
What OpenVINO is
OpenVINO is an inference toolkit: it takes a trained deep learning model, converts it into its own graph representation, optimises that graph for a target device and runs it. The README describes it as an "open-source software toolkit for optimizing and deploying deep learning models", covering computer vision, speech recognition, natural language processing and generative AI workloads. It does not train models. Training stays in PyTorch, TensorFlow or JAX; OpenVINO runs the result on a laptop, edge box or server.
The repository is openvinotoolkit/openvino, written mostly in C++, created on 15 October 2018 and licensed under Apache-2.0. The README lists three device families: "CPU (x86, ARM), GPU (Intel integrated & discrete GPU) and AI accelerators (Intel NPU)". Most of the stack is tuned for Intel hardware, but the CPU plugin also targets Arm, and the PyPI wheel ships for macOS on Apple silicon.
This page covers the core runtime and openvino Python package; GenAI, Model Server, NNCF and Optimum Intel live in separate repositories.
How it works
Three representations of a model
The model preparation guide describes three states. A model is either saved on disk (an OpenVINO IR pair of .xml and .bin files, or a framework format such as .onnx), loaded but not compiled as an ov.Model object that can still be reshaped, quantised or given preprocessing steps, or loaded and compiled as an ov.CompiledModel bound to a device. Only the compiled form runs inference.
Frontends and conversion
Frontends read framework formats. The docs list PyTorch, TensorFlow, TensorFlow Lite, ONNX, PaddlePaddle, JAX/Flax (marked experimental) and OpenVINO IR. ONNX, TensorFlow, TensorFlow Lite and PaddlePaddle files can be read directly at runtime, but the guide says that path "offers lower performance and stability, as well as fewer optimization options" and recommends explicit conversion to IR with openvino.convert_model plus openvino.save_model, or the ovc command-line tool. PyTorch and JAX models convert only through the Python API, which takes an example_input for tracing.
When saving IR, compress_to_fp16 is on by default: per the conversion parameters page, "all produced IRs are saved using FP16 data type for weights", which "saves up to 2x storage space" and "in most cases doesn't sacrifice model accuracy". The test below confirms the halving, and a small change in outputs.
Plugins, hints and precision
Each device is a plugin (libopenvino_intel_cpu_plugin.so, the GPU and NPU plugins, plus the virtual AUTO, HETERO and batching plugins). Instead of hand-tuning threads and streams, the docs recommend two performance hints: LATENCY, the runtime default, and THROUGHPUT. The hints page says the device "translates" the hint into concrete settings, such as the number of CPU streams, and that "models may load slower and consume more memory" with THROUGHPUT.
Precision is chosen the same way. The CPU device page says that on hardware with native half-precision support, "the plugin uses the half-precision type (bf16 or f16) automatically instead of f32 to improve performance", under the default PERFORMANCE execution mode. Setting ov::hint::inference_precision to f32, or the execution mode to ACCURACY, forces full precision.
Key features
- Multi-framework model import. Frontends for PyTorch, TensorFlow, TensorFlow Lite, ONNX, PaddlePaddle and JAX/Flax; the installed 2026.4.1 package reported
jax,paddle,onnx,pytorch,tflite,irandtfas available frontends. - OpenVINO IR. A portable
.xmlgraph plus.binweights file, with FP16 weight compression on by default when saving. - Device plugins. CPU (Intel x86-64 and Arm), Intel GPU and Intel NPU, with automatic device selection, heterogeneous execution across devices and automatic batching (the last one GPU-only, per the supported-devices table).
- Performance hints.
LATENCYandTHROUGHPUThints that pick streams, threads and request counts per model and device; results can be read back from the compiled model. - Automatic low precision.
bf16orf16execution on CPUs that support it, plus INT8 and other quantised types for models prepared with NNCF. - Dynamic shapes and stateful models. Full dynamic-shape support on CPU (but not dynamically changing rank), partial on GPU; stateful models on all three device families.
- Bundled tools. The PyPI wheel installs
ovc(model conversion),benchmark_app(performance measurement) andopt_in_out(telemetry consent) as console commands. - APIs and integrations. C++, Python, C and Node.js APIs, plus README-linked integrations with
torch.compile, ONNX Runtime, ExecuTorch, vLLM, Keras 3, LangChain and LlamaIndex.
Getting started
The README's quick install is a single pip command:
pip install -U openvino
and its verification snippet is:
import openvino as ov
print(ov.__version__)
The PyPI install guide recommends a virtual environment first, then lists available devices as the check:
python3 -m venv openvino_env
source openvino_env/bin/activate
python -m pip install openvino
python -c "from openvino import Core; print(Core().available_devices)"
The same page notes that the PyPI distribution "offers the Python API only" and that on macOS it "offers support only for CPU inference". GPU and NPU use need drivers that, per the system requirements, are not included in the package.
Converting a model on the command line, with and without the default FP16 weight compression, per the conversion parameters page:
ovc path_to_your_model
ovc path_to_your_model --compress_to_fp16=False
The Python equivalent:
ov_model = ov.convert_model(original_model)
ov.save_model(ov_model, 'model.xml', compress_to_fp16=False)
Benchmarking a model with each performance hint, from the Benchmark Tool page:
benchmark_app -m model.xml -hint latency
benchmark_app -m model.xml -hint throughput
To turn off telemetry, which is on by default (see the hands-on findings), the README gives:
opt_in_out --opt_out
Hands-on: OpenVINO 2026.4.1 on a 2 vCPU Linux VM
Everything in this section was run on 20 November 2026 in a cloud VM: Ubuntu 24.04.5 LTS, Linux 6.18 x86_64, 2 vCPUs reported as "Intel(R) Xeon(R) Processor @ 2.10GHz" (1 thread per core), about 7.8 GiB RAM, Python 3.13.18 in a fresh venv. The CPU flags include avx512_bf16, amx_bf16, amx_int8 and avx512_fp16, but not amx_fp16. The VM cannot reach Hugging Face, so no pretrained models were used: all models were built in-process with the openvino.opset13 graph API or the onnx package, with random weights. All numbers are single-machine measurements on a small shared VM and only indicative; repeat runs moved some latencies by up to about 1.7 times.
Install footprint and devices
| Check | Result |
|---|---|
pip install --no-cache-dir openvino into an empty venv | 5.2 s wall clock; installed openvino-2026.4.1, numpy-2.5.5 and openvino-telemetry-2025.2.0 |
| Wheel used | cp313-manylinux_2_28_x86_64, 59,142,703 bytes; a Python 3.13 wheel exists, so no fallback interpreter was needed |
| Disk added to the venv | 236.6 MB in total: 183.5 MB for openvino/ (169.6 MB of it in libs/) and 52.6 MB for NumPy |
| Largest libraries | CPU plugin 63.7 MB, GPU plugin 41.7 MB, core libopenvino.so 20.5 MB, NPU plugin 10.6 MB, ONNX frontend 6.8 MB, PyTorch frontend 3.3 MB, plus bundled oneTBB and hwloc |
import openvino | about 0.16 s cumulative, per python -X importtime |
Core().available_devices | ['CPU'] (no GPU or NPU in the VM, as expected) |
CPU OPTIMIZATION_CAPABILITIES | BF16, WINOGRAD, FP32, FP16, INT8, BIN, EXPORT_IMPORT |
| CPU defaults before compiling | PERFORMANCE_HINT LATENCY, INFERENCE_PRECISION_HINT bfloat16, EXECUTION_MODE_HINT PERFORMANCE, KV_CACHE_PRECISION uint8_t, CPU pinning on |
The default inference precision on this CPU is bfloat16, not f32. That matches the CPU device page (half precision is used automatically where the hardware supports it), but it means the same application produces slightly different numbers on a machine with AMX than on one without it.
Two test models
- Small CNN: input
[1,3,224,224], four stride-2 3x3 convolutions (3 to 32 to 64 to 128 to 256 channels) with bias and ReLU, global average pooling and a 256x1000MatMul; about 0.64 million weights. - MLP block: input
[32,1024](a batch of 32 rows), sixMatMullayers alternating 1024x4096 and 4096x1024 with GELU after each expansion; 25.2 million weights, roughly the size of the feed-forward layers in a small transformer.
Each configuration was warmed up with 10 calls, timed over 500 synchronous infer() calls, then given 1,000 requests through an AsyncInferQueue sized to OPTIMAL_NUMBER_OF_INFER_REQUESTS, with a fixed input per model.
Performance hints and inference precision
| Model, hint | Precision | Streams / requests | Sync p50 / p90 latency | Async throughput |
|---|---|---|---|---|
CNN, LATENCY | f32 | 1 / 1 | 1.66 / 2.24 ms | 511 inferences/s |
CNN, LATENCY | bf16 | 1 / 1 | 0.92 / 1.00 ms | 1,382 inferences/s |
CNN, LATENCY | f16 | 1 / 1 | 2.55 / 3.19 ms | 352 inferences/s |
CNN, LATENCY | not set (became bf16) | 1 / 1 | 0.91 / 0.98 ms | 1,049 inferences/s |
CNN, THROUGHPUT | f32 | 2 / 2 | 3.79 / 4.20 ms | 463 inferences/s |
CNN, THROUGHPUT | bf16 | 2 / 2 | 1.45 / 1.63 ms | 1,573 inferences/s |
CNN, THROUGHPUT | not set (became bf16) | 2 / 2 | 1.01 / 1.70 ms | 1,465 inferences/s |
MLP, LATENCY | f32 | 1 / 1 | 9.64 / 12.53 ms | 94 batches/s |
MLP, LATENCY | bf16 | 1 / 1 | 1.87 / 3.18 ms | 393 batches/s |
MLP, LATENCY | f16 | 1 / 1 | 13.87 / 16.20 ms | 71 batches/s |
MLP, THROUGHPUT | f32 | 2 / 2 | 20.67 / 24.20 ms | 100 batches/s |
MLP, THROUGHPUT | bf16 | 2 / 2 | 5.97 / 7.97 ms | 450 batches/s |
MLP, THROUGHPUT | not set (became bf16) | 2 / 2 | 5.55 / 10.01 ms | 537 batches/s |
Compilation took 0.02 to 0.06 s for the CNN and 0.07 to 0.10 s for the MLP in every configuration. Three things stand out:
- bf16 on AMX was the big lever. On the 25-million-weight MLP,
bf16cut single-request latency about five times againstf32(1.87 against 9.64 ms) and raised throughput about 4.5 times. Because the default is alreadybf16here, the code that wrote nothing got the fast path; code that forcedf32for accuracy paid for it. - Explicit f16 was the slowest option. The CPU advertises
FP16and acceptedINFERENCE_PRECISION_HINT=f16, but it ran 1.4 to 1.5 times slower thanf32. This VM's CPU hasavx512_fp16but noamx_fp16flag; the docs tie the AMX speed-up to "4th-gen and newer" Xeons without listing which half type each generation accelerates, so measure before forcingf16. - THROUGHPUT barely helps on 2 vCPUs. The hint created 2 streams and 2 requests, roughly doubling per-request latency for a throughput gain of 0 to 40 percent, and once a loss (CNN
f32: 463 against 511 inferences/s).
What lower precision did to outputs
| Model | bf16 against f32 (max absolute difference, relative to largest output) | f16 against f32 |
|---|---|---|
| CNN logits | 1.08e-3 (0.32 percent) | 1.39e-4 (0.04 percent) |
| MLP output | 2.23e-3 (0.56 percent) | 2.87e-4 (0.07 percent) |
Outputs were identical between LATENCY and THROUGHPUT at the same precision, and the "not set" runs matched bf16 exactly. f16 was about eight times closer to f32 than bf16, as expected from its longer mantissa. Random weights say nothing about real accuracy, but the docs warn that bf16 "may affect inference accuracy in LLM models"; validate on real data.
Saving IR and model caching
| Model | compress_to_fp16 | .xml | .bin | Weights after reload | Max difference vs in-memory model (both run in f32) |
|---|---|---|---|---|---|
| CNN | default (True) | 17,102 B | 1,288,848 B | float16, 9 Convert nodes | 1.0e-4 |
| CNN | False | 12,127 B | 2,577,680 B | float32 | 0 |
| MLP | default (True) | 9,983 B | 50,331,648 B | float16, 6 Convert nodes | 1.9e-4 |
| MLP | False | 7,008 B | 100,663,296 B | float32 | 0 |
The default exactly halves the weights file, and the IR stays readable in milliseconds (read_model took 2 to 5 ms). The cost is that an FP16-compressed IR no longer reproduces the original model bit for bit, even when executed in f32. Model caching with CACHE_DIR wrote a 50,353,690-byte blob for the MLP, but compile times across three runs (0.077, 0.047 and 0.083 s) showed no consistent gain; caching is aimed at large models where compilation takes seconds.
benchmark_app
The wheel installs the Python benchmark_app. Run with only -m on the CNN IR, it printed a warning that the hint "will be set to PerformanceMode.THROUGHPUT", used bfloat16, 2 streams and 2 requests, ran for 60,002.78 ms and reported 98,296 iterations: median latency 1.01 ms, max 71.80 ms, 1,638.19 FPS. Runs of 15 s on the MLP IR (-t 15):
| Flags | Precision | Compile time, RAM | First inference | Median latency | Throughput |
|---|---|---|---|---|---|
-hint latency | bfloat16 | 88.23 ms, 216,580 KB | 9.23 ms | 1.78 ms | 424.30 FPS |
-hint throughput | bfloat16 | 89.02 ms, 355,848 KB | 7.09 ms | 3.17 ms | 481.74 FPS |
-hint latency -infer_precision f32 | float32 | 78.33 ms, 219,652 KB | 15.86 ms | 11.97 ms | 83.54 FPS |
-hint throughput -infer_precision f32 | float32 | 104.81 ms, 358,920 KB | 24.96 ms | 19.97 ms | 95.47 FPS |
"FPS" here is inferences per second; the [32,1024] input has no layout, so each counted frame is a 32-row batch. The figures agree with the hand-written loop, and "Compile model ram used" confirms the hints page: THROUGHPUT used about 64 percent more memory at compile time.
ONNX import and ovc
A 256-512-10 MLP with softmax and a dynamic batch dimension was built with onnx 1.23.2 (installed from PyPI in 11.4 s) and saved as a 547,069-byte .onnx file.
| Step | Result |
|---|---|
ovc tiny.onnx --output_model ovc_out | 0.28 s; wrote ovc_out.xml (4,334 B) and ovc_out.bin (272,384 B) with FP16 weights; input shape kept as [?,256] |
Same with --compress_to_fp16=False | .bin of 544,768 B |
core.read_model('tiny.onnx') directly | 4.8 ms, no conversion step needed |
Accuracy against a NumPy reference, f32 | ONNX and FP32 IR: max error 1.2e-7; FP16 IR: 7.6e-5 |
Accuracy, default bf16, batch of 64 | max error 1.1e-3 to 1.2e-3; top-1 class changed for 1 of 64 rows |
| Dynamic batch | Batches of 1 and 64 both ran on the same compiled model |
Two ovc quirks: its FP16 notice says to disable compression "by removing argument "compress_to_fp16"", which had not been passed, and links a docs.openvino.ai/2023.0 page. And although the docs say a directory passed to --output_model receives <model>.xml, that only happened for an existing directory or a trailing slash (outdir/); a new name without one became a file prefix.
Telemetry on import
The README says usage data is collected "directly by OpenVINO or through the use of Google Analytics 4", by default per the telemetry page. The trigger is less obvious. The package's __init__.py imports convert_model from openvino.tools.ovc, and that module initialises telemetry with enable_opt_in_dialog=False and sends an import event. Observed:
- A bare
python -c "import openvino"created~/intel/openvino_ga_cid(a client ID) and~/intel/stats("usage_count": 1). With the HTTPS proxy pointed at a local listener, the process opened aCONNECT www.google-analytics.com:443. - With
CI=trueset, no files were created and no connection was made; the code also checksTF_BUILDandJENKINS_URL. opt_in_out --opt_outwrote0to~/intel/openvino_telemetry, and itself made one connection (the code force-sends a "declined" event). Later imports made no connection.
The openvino-telemetry package on PyPI describes itself as "sending statistics with user's consent", but in this flow a missing consent file counts as consent. For containers, opt out at image build time or set CI=true.
Use cases
- CPU inference on Intel servers. Vision, audio and embedding models on Xeons, where the automatic
bf16path on AMX gave the test MLP a 4 to 5 times speed-up. - Edge and client devices. Laptops and gateways with Intel integrated GPUs or NPUs, using
AUTOto pick a device and the IR format to avoid shipping a training framework. - Computer vision pipelines. Running detection or classification models next to OpenCV preprocessing; the README's tutorial list includes YOLO optimisation notebooks.
- Speech recognition. The README links Whisper notebooks, and the 2026.4 release notes add an
ASRPipelineto the Node.js GenAI API; for a plain C/C++ Whisper engine without OpenVINO, see whisper.cpp. - Accelerating an existing ONNX Runtime app. The README links an OpenVINO execution provider for ONNX Runtime, which keeps ONNX Runtime's API while using OpenVINO underneath.
- LLM serving on Intel hardware. Through OpenVINO GenAI, OpenVINO Model Server or the
vllm-openvinobackend linked from the README; the core runtime alone is a building block, not an LLM server.
How OpenVINO compares
Descriptions below come from each project's README, licence file or PyPI metadata as read on 20 November 2026; none of the alternatives were run here, and no performance comparison is implied.
| Project | How it describes itself | Model formats | Hardware focus | Licence |
|---|---|---|---|---|
| OpenVINO | "Open-source software toolkit for optimizing and deploying deep learning models" | PyTorch, TensorFlow, TFLite, ONNX, PaddlePaddle, JAX (experimental), own IR | x86 and Arm CPUs, Intel GPUs, Intel NPUs | Apache-2.0 |
| ONNX Runtime | "a performance-focused scoring engine for Open Neural Network Exchange (ONNX) models" (PyPI description) | ONNX | Pluggable execution providers, one of which is OpenVINO | MIT |
| llama.cpp | aims "to enable LLM (and MLLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware" | GGUF | Broad CPU and GPU support, built on ggml | MIT |
| vLLM | "a fast and easy-to-use library for LLM inference and serving" | Hugging Face model architectures | NVIDIA, AMD and Intel GPUs, x86/Arm/PowerPC CPUs, hardware plugins | Apache-2.0 |
| ggml | "Tensor library for machine learning" | Library-level; models are built in code | CPU, GPU, NPU and browser backends | MIT |
ONNX Runtime is the closest general-purpose peer: both run conventional neural networks across many model types. The main split is format and scope. ONNX Runtime is built around ONNX and swaps hardware back ends through execution providers; OpenVINO imports several frameworks, converts them to its own IR and concentrates on Intel devices. OpenVINO also ships as an ONNX Runtime execution provider, so the two can be combined.
llama.cpp and vLLM are LLM engines rather than general model runtimes. llama.cpp targets minimal-setup local inference on GGUF files; vLLM focuses on high-throughput serving with PagedAttention and continuous batching, and OpenVINO's README links a vLLM backend for Intel hardware. OpenVINO's LLM equivalents are the separate GenAI library and Model Server, not tested here. ggml, the tensor library under llama.cpp, works one level lower: models are assembled in C, much like the opset graph API used in the test above.
Limits, trade-offs and when not to use it
- Defaults depend on the CPU. On this AMX-capable Xeon the runtime silently used
bf16; on a CPU without native half precision it usesf32. Results, accuracy and speed can therefore change when the same container moves between hosts. PinINFERENCE_PRECISION_HINTorEXECUTION_MODE_HINTif reproducibility matters. - FP16 IR and forced precisions. Default weight compression changed outputs by up to 1.9e-4 even in
f32; usecompress_to_fp16=Falsewhen outputs must match the source framework. Forcedf16was slower thanf32here, andTHROUGHPUTon 2 vCPUs cost latency and memory for little gain. - Telemetry fires on import. Any process importing
openvinocontacted Google Analytics unless consent was declined orCI=truewas set. - Intel-centric acceleration. GPU and NPU support covers Intel devices only; other accelerators need different runtimes. Arm CPUs are supported but, per the CPU page, run quantised models "in simulation mode" in floating point, and Windows on ARM64 has no CPU inference.
- Hardware floor raised. The 2026 deprecation notes say the CPU plugin now requires AVX2 and SSE is no longer supported, which rules out CPUs without AVX2.
- Not an LLM server by itself. Tokenisation, KV-cache management and serving live in GenAI, Tokenizers and Model Server; the core runtime is lower level.
- API churn. The 2026 notes list removals such as the
openvino.runtimenamespace andopenvino.Type.undefined, an NNCF 3.0 API break, and Python 3.10 support ending in 2026.5.
Documentation discrepancies found
- The system requirements page still lists "Intel Atom processor with Intel SSE4.2 support" as supported hardware, while the release notes' deprecation section says the CPU plugin now requires AVX2.
- The system requirements give Python 3.11 to 3.15. The 2026.4.1 wheels on PyPI cover CPython 3.10 to 3.14 (including a free-threaded 3.14t build), declare
Requires-Python >=3.10, and there is no 3.15 wheel. - The PyPI project's "Homepage" link points to the
docs.openvino.ai/2023.0documentation, andovc's FP16 message links to a 2023.0 page as well. - The supported-devices page's API coverage table is dated "as of OpenVINO 2024.5, 20 Nov. 2024", and its distribution table is labelled 2024.6.
- The Benchmark Tool page says
benchmark_app"creates 4 and 16 inference requests for CPU and GPU respectively" by default; on this 2 vCPU VM the default run used 2, derived from theTHROUGHPUThint. - The wheel ships
libopenvino_gguf_frontend.so, butFrontEndManager().get_available_front_ends()did not list a GGUF frontend, and the model preparation page does not mention GGUF. - The release notes on
masterdescribe 2026.4 (16 September 2026) but have no separate section for the 2026.4.1 patch that PyPI and GitHub published on 1 October 2026.
Releases, security and project activity
Checked 20 November 2026: latest release 2026.4.1, published 1 October 2026 on GitHub and on PyPI (2026.4.0 appeared on 16 September 2026, the date of the 2026.4 release notes), licence Apache-2.0. Releases are frequent: PyPI shows 2026.0.0 (23 February), 2026.1.0 (7 April), 2026.2.0 (28 May), 2026.2.1 (17 June), 2026.3.0 (4 August), 2026.3.1 (26 August), 2026.4.0 (16 September) and 2026.4.1 (1 October). The 2026.4 release notes mention a possible move to C++20 as the default standard from release 27.0, and list known issues including a "slight performance regression" on some convolution-heavy models on Xeon after the oneDNN 3.13 upgrade.
SECURITY.md routes vulnerability reports to the Intel Security Center and shows OpenSSF Best Practices and Coverity badges. The 2026.4 notes also mention hardening against malformed IR files, so a corrupted .xml "now produces a predictable error rather than a crash"; IR from untrusted sources should still be treated as untrusted input.
The openvinotoolkit/openvino repository had about 11,700 stars (11,672 at the time of writing) and 3,631 forks, with the latest push on 20 November 2026. It is written in C++, was created on 15 October 2018 and develops on master. User documentation is at docs.openvino.ai; its sources live under docs/ in the repository, and the README points to GitHub Issues, the Intel DevHub Discord and the openvino Stack Overflow tag for support.
Enjoying this project?
Discover more amazing open-source projects on TechLogHub. We curate the best developer tools and projects.
Repository:https://github.com/openvinotoolkit/openvino
GitHub - openvinotoolkit/openvino: OpenVINO: Open-Source Toolkit for Deep Learning Inference
OpenVINO is Intel's Apache-2.0 toolkit for converting and running deep learning models on CPUs, GPUs and NPUs. Version 2026.4.1 tested on Linux: install footpri...
github - openvinotoolkit/openvino
Related Projects
llama.cpp: LLM inference in plain C/C++
Dependency-free C/C++ LLM inference with quantization, many GPU backends and an OpenAI-compatible server.
Claude-Mem: Persistent Memory for AI Coding Agents
Persistent cross-session memory for Claude Code, OpenCode, Codex and other agents, with SQLite and MCP search.
vLLM: High-Throughput LLM Inference and Serving Engine
Python engine for fast LLM inference and OpenAI-compatible serving, built around PagedAttention.
Maintain OpenVINO: Open-Source Toolkit for Deep Learning Inference? Add the badge to your README
Show that the project is featured on TechLogHub and send readers to this write-up.
[](https://techloghub.com/open-source/openvino?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)