OpenVINO: Open-Source Toolkit for Deep Learning Inference
GitHub Repo
Apache-2.0
November 20, 2026 at 01:46 PM
0 views

OpenVINO: Open-Source Toolkit for Deep Learning Inference

@openvinotoolkitProject Author

What OpenVINO is

OpenVINO is an inference toolkit: it takes a trained deep learning model, converts it into its own graph representation, optimises that graph for a target device and runs it. The README describes it as an "open-source software toolkit for optimizing and deploying deep learning models", covering computer vision, speech recognition, natural language processing and generative AI workloads. It does not train models. Training stays in PyTorch, TensorFlow or JAX; OpenVINO runs the result on a laptop, edge box or server.

The repository is openvinotoolkit/openvino, written mostly in C++, created on 15 October 2018 and licensed under Apache-2.0. The README lists three device families: "CPU (x86, ARM), GPU (Intel integrated & discrete GPU) and AI accelerators (Intel NPU)". Most of the stack is tuned for Intel hardware, but the CPU plugin also targets Arm, and the PyPI wheel ships for macOS on Apple silicon.

This page covers the core runtime and openvino Python package; GenAI, Model Server, NNCF and Optimum Intel live in separate repositories.

How it works

Three representations of a model

The model preparation guide describes three states. A model is either saved on disk (an OpenVINO IR pair of .xml and .bin files, or a framework format such as .onnx), loaded but not compiled as an ov.Model object that can still be reshaped, quantised or given preprocessing steps, or loaded and compiled as an ov.CompiledModel bound to a device. Only the compiled form runs inference.

Frontends and conversion

Frontends read framework formats. The docs list PyTorch, TensorFlow, TensorFlow Lite, ONNX, PaddlePaddle, JAX/Flax (marked experimental) and OpenVINO IR. ONNX, TensorFlow, TensorFlow Lite and PaddlePaddle files can be read directly at runtime, but the guide says that path "offers lower performance and stability, as well as fewer optimization options" and recommends explicit conversion to IR with openvino.convert_model plus openvino.save_model, or the ovc command-line tool. PyTorch and JAX models convert only through the Python API, which takes an example_input for tracing.

When saving IR, compress_to_fp16 is on by default: per the conversion parameters page, "all produced IRs are saved using FP16 data type for weights", which "saves up to 2x storage space" and "in most cases doesn't sacrifice model accuracy". The test below confirms the halving, and a small change in outputs.

Plugins, hints and precision

Each device is a plugin (libopenvino_intel_cpu_plugin.so, the GPU and NPU plugins, plus the virtual AUTO, HETERO and batching plugins). Instead of hand-tuning threads and streams, the docs recommend two performance hints: LATENCY, the runtime default, and THROUGHPUT. The hints page says the device "translates" the hint into concrete settings, such as the number of CPU streams, and that "models may load slower and consume more memory" with THROUGHPUT.

Precision is chosen the same way. The CPU device page says that on hardware with native half-precision support, "the plugin uses the half-precision type (bf16 or f16) automatically instead of f32 to improve performance", under the default PERFORMANCE execution mode. Setting ov::hint::inference_precision to f32, or the execution mode to ACCURACY, forces full precision.

Key features

  • Multi-framework model import. Frontends for PyTorch, TensorFlow, TensorFlow Lite, ONNX, PaddlePaddle and JAX/Flax; the installed 2026.4.1 package reported jax, paddle, onnx, pytorch, tflite, ir and tf as available frontends.
  • OpenVINO IR. A portable .xml graph plus .bin weights file, with FP16 weight compression on by default when saving.
  • Device plugins. CPU (Intel x86-64 and Arm), Intel GPU and Intel NPU, with automatic device selection, heterogeneous execution across devices and automatic batching (the last one GPU-only, per the supported-devices table).
  • Performance hints. LATENCY and THROUGHPUT hints that pick streams, threads and request counts per model and device; results can be read back from the compiled model.
  • Automatic low precision. bf16 or f16 execution on CPUs that support it, plus INT8 and other quantised types for models prepared with NNCF.
  • Dynamic shapes and stateful models. Full dynamic-shape support on CPU (but not dynamically changing rank), partial on GPU; stateful models on all three device families.
  • Bundled tools. The PyPI wheel installs ovc (model conversion), benchmark_app (performance measurement) and opt_in_out (telemetry consent) as console commands.
  • APIs and integrations. C++, Python, C and Node.js APIs, plus README-linked integrations with torch.compile, ONNX Runtime, ExecuTorch, vLLM, Keras 3, LangChain and LlamaIndex.

Getting started

The README's quick install is a single pip command:

pip install -U openvino

and its verification snippet is:

import openvino as ov
print(ov.__version__)

The PyPI install guide recommends a virtual environment first, then lists available devices as the check:

python3 -m venv openvino_env
source openvino_env/bin/activate
python -m pip install openvino
python -c "from openvino import Core; print(Core().available_devices)"

The same page notes that the PyPI distribution "offers the Python API only" and that on macOS it "offers support only for CPU inference". GPU and NPU use need drivers that, per the system requirements, are not included in the package.

Converting a model on the command line, with and without the default FP16 weight compression, per the conversion parameters page:

ovc path_to_your_model
ovc path_to_your_model --compress_to_fp16=False

The Python equivalent:

ov_model = ov.convert_model(original_model)
ov.save_model(ov_model, 'model.xml', compress_to_fp16=False)

Benchmarking a model with each performance hint, from the Benchmark Tool page:

benchmark_app -m model.xml -hint latency
benchmark_app -m model.xml -hint throughput

To turn off telemetry, which is on by default (see the hands-on findings), the README gives:

opt_in_out --opt_out

Hands-on: OpenVINO 2026.4.1 on a 2 vCPU Linux VM

Everything in this section was run on 20 November 2026 in a cloud VM: Ubuntu 24.04.5 LTS, Linux 6.18 x86_64, 2 vCPUs reported as "Intel(R) Xeon(R) Processor @ 2.10GHz" (1 thread per core), about 7.8 GiB RAM, Python 3.13.18 in a fresh venv. The CPU flags include avx512_bf16, amx_bf16, amx_int8 and avx512_fp16, but not amx_fp16. The VM cannot reach Hugging Face, so no pretrained models were used: all models were built in-process with the openvino.opset13 graph API or the onnx package, with random weights. All numbers are single-machine measurements on a small shared VM and only indicative; repeat runs moved some latencies by up to about 1.7 times.

Install footprint and devices

CheckResult
pip install --no-cache-dir openvino into an empty venv5.2 s wall clock; installed openvino-2026.4.1, numpy-2.5.5 and openvino-telemetry-2025.2.0
Wheel usedcp313-manylinux_2_28_x86_64, 59,142,703 bytes; a Python 3.13 wheel exists, so no fallback interpreter was needed
Disk added to the venv236.6 MB in total: 183.5 MB for openvino/ (169.6 MB of it in libs/) and 52.6 MB for NumPy
Largest librariesCPU plugin 63.7 MB, GPU plugin 41.7 MB, core libopenvino.so 20.5 MB, NPU plugin 10.6 MB, ONNX frontend 6.8 MB, PyTorch frontend 3.3 MB, plus bundled oneTBB and hwloc
import openvinoabout 0.16 s cumulative, per python -X importtime
Core().available_devices['CPU'] (no GPU or NPU in the VM, as expected)
CPU OPTIMIZATION_CAPABILITIESBF16, WINOGRAD, FP32, FP16, INT8, BIN, EXPORT_IMPORT
CPU defaults before compilingPERFORMANCE_HINT LATENCY, INFERENCE_PRECISION_HINT bfloat16, EXECUTION_MODE_HINT PERFORMANCE, KV_CACHE_PRECISION uint8_t, CPU pinning on

The default inference precision on this CPU is bfloat16, not f32. That matches the CPU device page (half precision is used automatically where the hardware supports it), but it means the same application produces slightly different numbers on a machine with AMX than on one without it.

Two test models

  • Small CNN: input [1,3,224,224], four stride-2 3x3 convolutions (3 to 32 to 64 to 128 to 256 channels) with bias and ReLU, global average pooling and a 256x1000 MatMul; about 0.64 million weights.
  • MLP block: input [32,1024] (a batch of 32 rows), six MatMul layers alternating 1024x4096 and 4096x1024 with GELU after each expansion; 25.2 million weights, roughly the size of the feed-forward layers in a small transformer.

Each configuration was warmed up with 10 calls, timed over 500 synchronous infer() calls, then given 1,000 requests through an AsyncInferQueue sized to OPTIMAL_NUMBER_OF_INFER_REQUESTS, with a fixed input per model.

Performance hints and inference precision

Model, hintPrecisionStreams / requestsSync p50 / p90 latencyAsync throughput
CNN, LATENCYf321 / 11.66 / 2.24 ms511 inferences/s
CNN, LATENCYbf161 / 10.92 / 1.00 ms1,382 inferences/s
CNN, LATENCYf161 / 12.55 / 3.19 ms352 inferences/s
CNN, LATENCYnot set (became bf16)1 / 10.91 / 0.98 ms1,049 inferences/s
CNN, THROUGHPUTf322 / 23.79 / 4.20 ms463 inferences/s
CNN, THROUGHPUTbf162 / 21.45 / 1.63 ms1,573 inferences/s
CNN, THROUGHPUTnot set (became bf16)2 / 21.01 / 1.70 ms1,465 inferences/s
MLP, LATENCYf321 / 19.64 / 12.53 ms94 batches/s
MLP, LATENCYbf161 / 11.87 / 3.18 ms393 batches/s
MLP, LATENCYf161 / 113.87 / 16.20 ms71 batches/s
MLP, THROUGHPUTf322 / 220.67 / 24.20 ms100 batches/s
MLP, THROUGHPUTbf162 / 25.97 / 7.97 ms450 batches/s
MLP, THROUGHPUTnot set (became bf16)2 / 25.55 / 10.01 ms537 batches/s

Compilation took 0.02 to 0.06 s for the CNN and 0.07 to 0.10 s for the MLP in every configuration. Three things stand out:

  • bf16 on AMX was the big lever. On the 25-million-weight MLP, bf16 cut single-request latency about five times against f32 (1.87 against 9.64 ms) and raised throughput about 4.5 times. Because the default is already bf16 here, the code that wrote nothing got the fast path; code that forced f32 for accuracy paid for it.
  • Explicit f16 was the slowest option. The CPU advertises FP16 and accepted INFERENCE_PRECISION_HINT=f16, but it ran 1.4 to 1.5 times slower than f32. This VM's CPU has avx512_fp16 but no amx_fp16 flag; the docs tie the AMX speed-up to "4th-gen and newer" Xeons without listing which half type each generation accelerates, so measure before forcing f16.
  • THROUGHPUT barely helps on 2 vCPUs. The hint created 2 streams and 2 requests, roughly doubling per-request latency for a throughput gain of 0 to 40 percent, and once a loss (CNN f32: 463 against 511 inferences/s).

What lower precision did to outputs

Modelbf16 against f32 (max absolute difference, relative to largest output)f16 against f32
CNN logits1.08e-3 (0.32 percent)1.39e-4 (0.04 percent)
MLP output2.23e-3 (0.56 percent)2.87e-4 (0.07 percent)

Outputs were identical between LATENCY and THROUGHPUT at the same precision, and the "not set" runs matched bf16 exactly. f16 was about eight times closer to f32 than bf16, as expected from its longer mantissa. Random weights say nothing about real accuracy, but the docs warn that bf16 "may affect inference accuracy in LLM models"; validate on real data.

Saving IR and model caching

Modelcompress_to_fp16.xml.binWeights after reloadMax difference vs in-memory model (both run in f32)
CNNdefault (True)17,102 B1,288,848 Bfloat16, 9 Convert nodes1.0e-4
CNNFalse12,127 B2,577,680 Bfloat320
MLPdefault (True)9,983 B50,331,648 Bfloat16, 6 Convert nodes1.9e-4
MLPFalse7,008 B100,663,296 Bfloat320

The default exactly halves the weights file, and the IR stays readable in milliseconds (read_model took 2 to 5 ms). The cost is that an FP16-compressed IR no longer reproduces the original model bit for bit, even when executed in f32. Model caching with CACHE_DIR wrote a 50,353,690-byte blob for the MLP, but compile times across three runs (0.077, 0.047 and 0.083 s) showed no consistent gain; caching is aimed at large models where compilation takes seconds.

benchmark_app

The wheel installs the Python benchmark_app. Run with only -m on the CNN IR, it printed a warning that the hint "will be set to PerformanceMode.THROUGHPUT", used bfloat16, 2 streams and 2 requests, ran for 60,002.78 ms and reported 98,296 iterations: median latency 1.01 ms, max 71.80 ms, 1,638.19 FPS. Runs of 15 s on the MLP IR (-t 15):

FlagsPrecisionCompile time, RAMFirst inferenceMedian latencyThroughput
-hint latencybfloat1688.23 ms, 216,580 KB9.23 ms1.78 ms424.30 FPS
-hint throughputbfloat1689.02 ms, 355,848 KB7.09 ms3.17 ms481.74 FPS
-hint latency -infer_precision f32float3278.33 ms, 219,652 KB15.86 ms11.97 ms83.54 FPS
-hint throughput -infer_precision f32float32104.81 ms, 358,920 KB24.96 ms19.97 ms95.47 FPS

"FPS" here is inferences per second; the [32,1024] input has no layout, so each counted frame is a 32-row batch. The figures agree with the hand-written loop, and "Compile model ram used" confirms the hints page: THROUGHPUT used about 64 percent more memory at compile time.

ONNX import and ovc

A 256-512-10 MLP with softmax and a dynamic batch dimension was built with onnx 1.23.2 (installed from PyPI in 11.4 s) and saved as a 547,069-byte .onnx file.

StepResult
ovc tiny.onnx --output_model ovc_out0.28 s; wrote ovc_out.xml (4,334 B) and ovc_out.bin (272,384 B) with FP16 weights; input shape kept as [?,256]
Same with --compress_to_fp16=False.bin of 544,768 B
core.read_model('tiny.onnx') directly4.8 ms, no conversion step needed
Accuracy against a NumPy reference, f32ONNX and FP32 IR: max error 1.2e-7; FP16 IR: 7.6e-5
Accuracy, default bf16, batch of 64max error 1.1e-3 to 1.2e-3; top-1 class changed for 1 of 64 rows
Dynamic batchBatches of 1 and 64 both ran on the same compiled model

Two ovc quirks: its FP16 notice says to disable compression "by removing argument "compress_to_fp16"", which had not been passed, and links a docs.openvino.ai/2023.0 page. And although the docs say a directory passed to --output_model receives <model>.xml, that only happened for an existing directory or a trailing slash (outdir/); a new name without one became a file prefix.

Telemetry on import

The README says usage data is collected "directly by OpenVINO or through the use of Google Analytics 4", by default per the telemetry page. The trigger is less obvious. The package's __init__.py imports convert_model from openvino.tools.ovc, and that module initialises telemetry with enable_opt_in_dialog=False and sends an import event. Observed:

  • A bare python -c "import openvino" created ~/intel/openvino_ga_cid (a client ID) and ~/intel/stats ("usage_count": 1). With the HTTPS proxy pointed at a local listener, the process opened a CONNECT www.google-analytics.com:443.
  • With CI=true set, no files were created and no connection was made; the code also checks TF_BUILD and JENKINS_URL.
  • opt_in_out --opt_out wrote 0 to ~/intel/openvino_telemetry, and itself made one connection (the code force-sends a "declined" event). Later imports made no connection.

The openvino-telemetry package on PyPI describes itself as "sending statistics with user's consent", but in this flow a missing consent file counts as consent. For containers, opt out at image build time or set CI=true.

Use cases

  • CPU inference on Intel servers. Vision, audio and embedding models on Xeons, where the automatic bf16 path on AMX gave the test MLP a 4 to 5 times speed-up.
  • Edge and client devices. Laptops and gateways with Intel integrated GPUs or NPUs, using AUTO to pick a device and the IR format to avoid shipping a training framework.
  • Computer vision pipelines. Running detection or classification models next to OpenCV preprocessing; the README's tutorial list includes YOLO optimisation notebooks.
  • Speech recognition. The README links Whisper notebooks, and the 2026.4 release notes add an ASRPipeline to the Node.js GenAI API; for a plain C/C++ Whisper engine without OpenVINO, see whisper.cpp.
  • Accelerating an existing ONNX Runtime app. The README links an OpenVINO execution provider for ONNX Runtime, which keeps ONNX Runtime's API while using OpenVINO underneath.
  • LLM serving on Intel hardware. Through OpenVINO GenAI, OpenVINO Model Server or the vllm-openvino backend linked from the README; the core runtime alone is a building block, not an LLM server.

How OpenVINO compares

Descriptions below come from each project's README, licence file or PyPI metadata as read on 20 November 2026; none of the alternatives were run here, and no performance comparison is implied.

ProjectHow it describes itselfModel formatsHardware focusLicence
OpenVINO"Open-source software toolkit for optimizing and deploying deep learning models"PyTorch, TensorFlow, TFLite, ONNX, PaddlePaddle, JAX (experimental), own IRx86 and Arm CPUs, Intel GPUs, Intel NPUsApache-2.0
ONNX Runtime"a performance-focused scoring engine for Open Neural Network Exchange (ONNX) models" (PyPI description)ONNXPluggable execution providers, one of which is OpenVINOMIT
llama.cppaims "to enable LLM (and MLLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware"GGUFBroad CPU and GPU support, built on ggmlMIT
vLLM"a fast and easy-to-use library for LLM inference and serving"Hugging Face model architecturesNVIDIA, AMD and Intel GPUs, x86/Arm/PowerPC CPUs, hardware pluginsApache-2.0
ggml"Tensor library for machine learning"Library-level; models are built in codeCPU, GPU, NPU and browser backendsMIT

ONNX Runtime is the closest general-purpose peer: both run conventional neural networks across many model types. The main split is format and scope. ONNX Runtime is built around ONNX and swaps hardware back ends through execution providers; OpenVINO imports several frameworks, converts them to its own IR and concentrates on Intel devices. OpenVINO also ships as an ONNX Runtime execution provider, so the two can be combined.

llama.cpp and vLLM are LLM engines rather than general model runtimes. llama.cpp targets minimal-setup local inference on GGUF files; vLLM focuses on high-throughput serving with PagedAttention and continuous batching, and OpenVINO's README links a vLLM backend for Intel hardware. OpenVINO's LLM equivalents are the separate GenAI library and Model Server, not tested here. ggml, the tensor library under llama.cpp, works one level lower: models are assembled in C, much like the opset graph API used in the test above.

Limits, trade-offs and when not to use it

  • Defaults depend on the CPU. On this AMX-capable Xeon the runtime silently used bf16; on a CPU without native half precision it uses f32. Results, accuracy and speed can therefore change when the same container moves between hosts. Pin INFERENCE_PRECISION_HINT or EXECUTION_MODE_HINT if reproducibility matters.
  • FP16 IR and forced precisions. Default weight compression changed outputs by up to 1.9e-4 even in f32; use compress_to_fp16=False when outputs must match the source framework. Forced f16 was slower than f32 here, and THROUGHPUT on 2 vCPUs cost latency and memory for little gain.
  • Telemetry fires on import. Any process importing openvino contacted Google Analytics unless consent was declined or CI=true was set.
  • Intel-centric acceleration. GPU and NPU support covers Intel devices only; other accelerators need different runtimes. Arm CPUs are supported but, per the CPU page, run quantised models "in simulation mode" in floating point, and Windows on ARM64 has no CPU inference.
  • Hardware floor raised. The 2026 deprecation notes say the CPU plugin now requires AVX2 and SSE is no longer supported, which rules out CPUs without AVX2.
  • Not an LLM server by itself. Tokenisation, KV-cache management and serving live in GenAI, Tokenizers and Model Server; the core runtime is lower level.
  • API churn. The 2026 notes list removals such as the openvino.runtime namespace and openvino.Type.undefined, an NNCF 3.0 API break, and Python 3.10 support ending in 2026.5.

Documentation discrepancies found

  • The system requirements page still lists "Intel Atom processor with Intel SSE4.2 support" as supported hardware, while the release notes' deprecation section says the CPU plugin now requires AVX2.
  • The system requirements give Python 3.11 to 3.15. The 2026.4.1 wheels on PyPI cover CPython 3.10 to 3.14 (including a free-threaded 3.14t build), declare Requires-Python >=3.10, and there is no 3.15 wheel.
  • The PyPI project's "Homepage" link points to the docs.openvino.ai/2023.0 documentation, and ovc's FP16 message links to a 2023.0 page as well.
  • The supported-devices page's API coverage table is dated "as of OpenVINO 2024.5, 20 Nov. 2024", and its distribution table is labelled 2024.6.
  • The Benchmark Tool page says benchmark_app "creates 4 and 16 inference requests for CPU and GPU respectively" by default; on this 2 vCPU VM the default run used 2, derived from the THROUGHPUT hint.
  • The wheel ships libopenvino_gguf_frontend.so, but FrontEndManager().get_available_front_ends() did not list a GGUF frontend, and the model preparation page does not mention GGUF.
  • The release notes on master describe 2026.4 (16 September 2026) but have no separate section for the 2026.4.1 patch that PyPI and GitHub published on 1 October 2026.

Releases, security and project activity

Checked 20 November 2026: latest release 2026.4.1, published 1 October 2026 on GitHub and on PyPI (2026.4.0 appeared on 16 September 2026, the date of the 2026.4 release notes), licence Apache-2.0. Releases are frequent: PyPI shows 2026.0.0 (23 February), 2026.1.0 (7 April), 2026.2.0 (28 May), 2026.2.1 (17 June), 2026.3.0 (4 August), 2026.3.1 (26 August), 2026.4.0 (16 September) and 2026.4.1 (1 October). The 2026.4 release notes mention a possible move to C++20 as the default standard from release 27.0, and list known issues including a "slight performance regression" on some convolution-heavy models on Xeon after the oneDNN 3.13 upgrade.

SECURITY.md routes vulnerability reports to the Intel Security Center and shows OpenSSF Best Practices and Coverity badges. The 2026.4 notes also mention hardening against malformed IR files, so a corrupted .xml "now produces a predictable error rather than a crash"; IR from untrusted sources should still be treated as untrusted input.

The openvinotoolkit/openvino repository had about 11,700 stars (11,672 at the time of writing) and 3,631 forks, with the latest push on 20 November 2026. It is written in C++, was created on 15 October 2018 and develops on master. User documentation is at docs.openvino.ai; its sources live under docs/ in the repository, and the README points to GitHub Issues, the Intel DevHub Discord and the openvino Stack Overflow tag for support.

Enjoying this project?

Discover more amazing open-source projects on TechLogHub. We curate the best developer tools and projects.

Project
openvino
Created
November 20
Last Updated
November 20, 2026 at 01:46 PM

Maintain OpenVINO: Open-Source Toolkit for Deep Learning Inference? Add the badge to your README

Show that the project is featured on TechLogHub and send readers to this write-up.

Featured on TechLogHub
[![Featured on TechLogHub](https://techloghub.com/badge-neutral.svg)](https://techloghub.com/open-source/openvino?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)

Find more projects like this

One email a week: new and trending developer tools, fresh comparisons, and what shipped. Unsubscribe in one click.