ONNX Runtime: Cross-Platform ML Inference and Training

What ONNX Runtime is

ONNX Runtime is Microsoft's open-source engine for running machine-learning models stored in the ONNX format. The README describes it as "a cross-platform inference and training machine-learning accelerator", and the documentation site calls it a cross-platform model accelerator "with a flexible interface to integrate hardware-specific libraries". In practice it is the piece of software that sits between a trained model file and the hardware that has to execute it: a CPU in a server, an NVIDIA GPU, a phone's neural accelerator, or a browser tab.

The problem it addresses is the gap between where models are trained and where they run. Teams train in PyTorch, TensorFlow/Keras, scikit-learn, LightGBM or XGBoost, but they deploy into C# services, C++ applications, Java backends, mobile apps and web pages, often on a mix of hardware. The docs list the inferencing use cases directly: improve inference performance for a wide variety of models, run on different hardware and operating systems, "train in Python but deploy into a C#/C++/Java app", and train and run inference with models created in different frameworks. Exporting to ONNX gives a single interchange format; ONNX Runtime is the engine that executes it everywhere.

The project has two halves. ONNX Runtime inference is the part most developers meet: load a model, feed it tensors, get outputs. The docs state that it powers machine-learning models in Microsoft products and services across Office, Azure and Bing, as well as dozens of community projects. ONNX Runtime training covers two separate scenarios: large model training, where the README says it can accelerate training of transformer models on multi-node NVIDIA GPUs "with a one-line addition for existing PyTorch training scripts", and on-device training for edge and mobile targets. The core is written in C++ and released under the MIT licence.

How it works

The basic workflow

The documentation summarises the workflow in three steps. First, get a model, trained in any framework that supports export or conversion to ONNX. Second, load and run it with ONNX Runtime. Third, optionally tune performance with runtime configuration or hardware accelerators. The docs add that "even without step 3, ONNX Runtime will often provide performance improvements compared to the original framework", which is a claim from the project rather than a measured guarantee for any particular model.

When a session is created, ONNX Runtime applies a series of graph optimizations to the model graph and then partitions the graph into subgraphs based on which hardware accelerators are available. Optimized kernels in the core runtime handle what remains, and the assigned subgraphs get further acceleration from the relevant execution provider.

Execution providers

Hardware support is organised around execution providers (EPs). Each EP wraps a hardware acceleration library, and ONNX Runtime asks each one, through a GetCapability() interface, which nodes or subgraphs it can take. The EP library then executes those pieces on its hardware. The docs present this as a way to abstract hardware-specific libraries across CPU, GPU, FPGA and specialised NPUs, while the application keeps calling the same API.

Providers are registered in priority order. The EP documentation gives the example ['CUDAExecutionProvider', 'CPUExecutionProvider'], which means a node runs on CUDA if that provider can handle it, and otherwise falls back to the CPU provider. The same API is used across all EPs, and the options for setting EP configuration are available in Python, C/C++/C#, Java and Node.js. The page notes that API parity across language bindings is still being worked on.

The summary table on the execution providers page groups the supported EPs as follows:

• CPU: the default CPU provider, Intel DNNL, Intel OpenVINO, XNNPACK, TVM (preview) and AMD ROCm (marked deprecated).
• GPU: NVIDIA CUDA, NVIDIA TensorRT, DirectML, AMD MIGraphX, Intel OpenVINO, Qualcomm QNN and WebGPU.
• IoT, edge and mobile: Intel OpenVINO, Android Neural Networks API, XNNPACK, plus Arm Compute Library, Arm NN and CoreML in preview.
• Other: Rockchip NPU, Xilinx Vitis-AI, Huawei CANN and an Azure provider, all in preview.

The page is explicit that maturity varies: some EPs are "in production for live service, while others are released in preview". Vendors can also add their own provider; the docs point hardware developers to an "Add a new execution provider" guide, and the README links a separate plugin EP repository for Qualcomm QNN.

Graph optimizations

The graph optimization documentation splits transformations into three levels, each applied after the previous one, and states that all optimizations are enabled by default:

• Basic optimizations are semantics-preserving rewrites that run before partitioning, so they apply to every EP. They include constant folding, redundant node elimination (Identity, Slice, Unsqueeze and Dropout) and simple fusions such as Conv Add, Conv Mul, Conv BatchNorm, Relu Clip and Reshape fusion.
• Extended optimizations are more complex fusions that run after partitioning and only on nodes assigned to the CPU, CUDA or ROCm providers. The listed fusions include GELU, Layer Normalization, BERT Embedding Layer, Attention, Skip Layer Normalization and Bias GELU. A GELU approximation exists for CUDA and ROCm but is disabled by default.
• Layout optimizations change data layout for CPU-assigned nodes, for example the NCHWc optimizer, which uses an NCHWc layout instead of NCHW.

Optimizations can run online, at session creation, or offline, where ONNX Runtime serializes the optimized graph to disk so later sessions can skip the work and start faster.

Key features

• Framework-neutral model input. The docs name PyTorch, TensorFlow/Keras, TFLite and scikit-learn as sources, and the README adds LightGBM and XGBoost among classical libraries. Converters such as tf2onnx and skl2onnx handle export for non-PyTorch frameworks.
• Many language bindings. The install table covers Python, C#/C/C++, WinML, Java, Android (Java/Kotlin and C/C++), iOS (C/C++ and Objective-C), React Native, Node.js and the web.
• Pluggable hardware acceleration. The execution provider framework covers CPUs, NVIDIA, AMD, Intel and Qualcomm hardware, Apple CoreML, Android NNAPI, DirectML and WebGPU, with prioritised fallback between providers.
• Built-in graph optimizations. Constant folding, node elimination, operator fusions and layout changes are on by default, with a GraphOptimizationLevel setting from ORT_DISABLE_ALL to ORT_ENABLE_ALL.
• Offline optimization. Setting optimized_model_filepath saves the optimized model, so startup cost does not have to be paid on every session.
• Browser inference. ONNX Runtime Web runs models in the browser on a WebAssembly CPU backend, with WebGPU and WebNN marked experimental and WebGL in maintenance mode.
• Size-reduced builds. Custom builds can include only the operator kernels a model needs, and "minimal builds" go further for mobile and web deployments.
• Training support. Large model training integrates with PyTorch through torch-ort, and on-device training packages exist for Windows, Linux, Android, iOS and the web.
• Backwards compatibility. The compatibility page states that newer versions support all models that worked with prior versions, and that every release supports ONNX opset 7 and higher.

Getting started

Install the Python package

The Python guide notes that there are two main Python packages and that "only one of these packages should be installed at a time in any one environment". Use the CPU package on Arm-based CPUs or macOS:

pip install onnxruntime

For NVIDIA GPUs, with the CUDA and TensorRT providers:

pip install onnxruntime-gpu

The GPU package requires CUDA and cuDNN. The CUDA provider page explains that onnxruntime-gpu can reuse the CUDA and cuDNN libraries installed with PyTorch when both are built against the same major versions, either by importing PyTorch first or by preloading the libraries before creating a session:

import onnxruntime

# Preload necessary DLLs
onnxruntime.preload_dlls()

Other packages in the install guide include onnxruntime-directml, onnxruntime-qnn and nightly builds from an Azure DevOps feed. The guide asks you to run pip install --upgrade pip first.

Other languages

For .NET, C and C++ on NuGet:

# CPU
dotnet add package Microsoft.ML.OnnxRuntime

# GPU
dotnet add package Microsoft.ML.OnnxRuntime.Gpu

For JavaScript in the browser, in Node.js and in React Native:

# install latest release version
npm install onnxruntime-web
npm install onnxruntime-node
npm install onnxruntime-react-native

Android projects add com.microsoft.onnxruntime:onnxruntime-android from Maven Central, and iOS projects add the onnxruntime-c or onnxruntime-objc CocoaPod.

Export a model and run it

The Python quickstart walks through PyTorch, TensorFlow and scikit-learn examples. For export, ONNX support is built into PyTorch, and the other frameworks use converters:

pip install tf2onnx
pip install skl2onnx

Once a model is in ONNX format, inference is a few lines. This is the scikit-learn iris example from the guide, run against a logistic regression model converted with skl2onnx:

import numpy
import onnxruntime as rt

sess = rt.InferenceSession("logreg_iris.onnx")
input_name = sess.get_inputs()[0].name
pred_onx = sess.run(None, {input_name: X_test.astype(numpy.float32)})[0]
print(pred_onx)

Choose execution providers

To prefer the GPU and fall back to the CPU, pass an ordered provider list when creating the session, as in the EP documentation:

import onnxruntime as rt

# prefer CUDA Execution Provider over CPU Execution Provider
EP_list = ['CUDAExecutionProvider', 'CPUExecutionProvider']

# initialize the model.onnx
sess = rt.InferenceSession("model.onnx", providers=EP_list)

set_providers changes the list later, and the docs note that the underlying session is re-created when it does.

Save an optimized model

The graph optimization page shows how to pick an optimization level and write the optimized graph to disk for offline use:

sess_options = rt.SessionOptions()

# Set graph optimization level
sess_options.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_EXTENDED

# To enable model serialization after graph optimization set this
sess_options.optimized_model_filepath = "<model_output_path\optimized_model.onnx>"

session = rt.InferenceSession("<model_path>", sess_options)

In the browser

ONNX Runtime Web is imported like any other npm module. WebGPU support, which the docs label experimental, uses a separate entry point:

import * as ort from 'onnxruntime-web';
import * as ort from 'onnxruntime-web/webgpu';

Building from source

The build docs recommend building from source only when you need a feature that is not yet in a released package, and building production deployments only from an official release branch. A basic CPU build needs Python 3.10+ and CMake 3.28 or higher:

git clone --recursive https://github.com/Microsoft/onnxruntime.git
cd onnxruntime
./build.sh --config RelWithDebInfo --build_shared_lib --parallel --compile_no_warning_as_error --skip_submodule_sync

Flags such as --build_wheel, --build_nuget, --build_java and --build_nodejs produce the language packages, and --enable_training adds training support.

Use cases

• Serving models trained in Python from other languages. A model trained in PyTorch or scikit-learn can be exported once and loaded from a C#, C++ or Java service using the same runtime, which is one of the use cases the docs name directly.
• Running one model on mixed hardware. An application can list CUDA or TensorRT first and fall back to CPU, so the same deployment works on GPU and CPU-only machines.
• Windows desktop applications. The install guide recommends WinML (Microsoft.AI.MachineLearning) for new Windows projects, with DirectML still available in sustained engineering.
• On-device and mobile inference. Android and iOS packages, NNAPI and CoreML providers, and size-reduced custom builds target phones and edge devices where binary size matters.
• Inference in the browser. The web getting-started page links examples such as image classification in a Next.js app, Whisper speech recognition and Segment Anything running client-side with WebGPU.
• Classical ML in production. Gradient-boosted trees and scikit-learn pipelines can be converted with ONNXMLTools or skl2onnx and served without a Python dependency in the serving path.
• Accelerating PyTorch training. The torch-ort package targets transformer training on NVIDIA GPUs, and on-device training covers fine-tuning on edge hardware.

How it compares

The ONNX Runtime README and docs do not include a comparison with other inference engines, so there is no project-sourced feature matrix to report. What the docs do make clear is how ONNX Runtime relates to the tools around it. It does not replace training frameworks: PyTorch, TensorFlow and scikit-learn remain where models are built, and the compatibility page recommends the latest stable releases of exporters such as PyTorch, tf2onnx, ONNXMLTools, skl2onnx, WinMLTools and Paddle2ONNX.

It also does not replace vendor libraries. TensorRT, OpenVINO, CoreML, NNAPI, DirectML, MIGraphX and QNN appear in ONNX Runtime as execution providers, so the choice is often between calling a vendor SDK directly or calling it through ONNX Runtime's single API with CPU fallback for unsupported operators. Some vendor builds are managed by the vendor: the install table lists OpenVINO packages as "Intel managed" and Jetson TensorRT builds as "NVIDIA managed". Calling a vendor runtime directly ties the application to that hardware, while going through ONNX Runtime adds a layer but keeps the application code the same across targets. Which trade-off is better depends on how many targets a team has to support.

Things to know before adopting

• Telemetry is on by default in official builds. The README says the project "may collect usage data and send it to Microsoft". The privacy document states that telemetry is turned on by default in official builds. Builds for Windows apps and components such as Windows ML use the Windows ETW system, while other official builds on Windows, supported Linux architectures, macOS, Android and iOS use the cross-platform 1DS telemetry SDK. Targets without a supported provider, including WebAssembly, tvOS, visionOS, Mac Catalyst, AIX and RISC-V, include no telemetry. The document says collection is subject to user consent and that custom model metadata is not uploaded through 1DS.
• How to turn telemetry off. The privacy document lists three options: pass --no_telemetry to the build script, set ORT_DISABLE_TELEMETRY=1 before ONNX Runtime initializes (this does not affect the legacy Windows TraceLogging backend), or suppress non-essential events through the C API and its C#, Python and Java bindings. It notes that a minimal initialization event may already have been sent before the API can be called. Private builds from source also enable telemetry by default on supported native platforms.
• Static Linux builds and curl/mbedTLS. Telemetry-enabled static Linux builds depend on static curl and mbedTLS, and the privacy document warns that applications linking such a build must not link another copy of either library. Build with --no_telemetry if your application needs its own copy.
• Packages are split by accelerator. CPU, CUDA/TensorRT, DirectML, QNN, Azure and training builds ship as separate packages, and the Python guide says only one package should be installed per environment. If several EPs are combined into one custom build, all their dependent libraries must be present at runtime.
• CUDA version matching takes care. The CUDA provider page says builds made with cuDNN 8.x and 9.x are not compatible with each other, and that from version 1.27 the default onnxruntime-gpu and Microsoft.ML.OnnxRuntime.Gpu packages are built with CUDA 13.0, with separate CUDA 12.x packages built with CUDA 12.8. The general install page still describes CUDA 12.x as the PyPI default, so check the CUDA provider requirements table for the exact release you install.
• Some providers are preview or legacy. TVM, CoreML, Arm NN, Arm Compute Library, Rockchip NPU, Vitis-AI, CANN and Azure are marked preview. ROCm is marked deprecated, DirectML is in sustained engineering, and WebGL in ONNX Runtime Web is in maintenance mode.
• Platform testing is limited. The compatibility page says ONNX Runtime is not tested against every environment combination. It lists tested platforms such as Windows 10 and Windows Server 2019, CentOS 7, macOS 10.14, Android API level 28 and iOS 12. The build page sets runtime minimums of Windows 10, CentOS 7 and Ubuntu 16.04, and says the release macOS packages target macOS 13.3 or later.
• Building from source is a substantial job. Source builds need recursive submodules, CMake 3.28+, Python 3.10+ and a supported compiler. On Windows only the Visual Studio 2022 generator is supported, GCC 8.x and below are not supported on Linux, and 32-bit Windows builds are no longer supported. The build script runs all unit tests by default for native builds.
• Minimal builds have trade-offs. A basic minimal build cannot load ONNX-format models, which must first be converted to ORT format. It also performs no runtime optimizations and supports only providers that statically register kernels.
• Validate untrusted models. The docs make users responsible for testing model accuracy and suitability. They warn that a malicious model can consume large amounts of memory or compute, and recommend inspecting models from untrusted sources in a safe environment first.
• Offline optimization is hardware-specific. An optimized model must be produced with the same execution providers, options and hardware as the target. A model with AVX2-specific layouts, for example, needs CPUs that support AVX2.
• System requirements. The install guide says all builds require the English language package with the en_US.UTF-8 locale, and Windows builds require the Visual C++ 2019 runtime.

Project activity

As of November 2026 the repository has about 22,300 stars on GitHub. It was created in November 2018, is written mainly in C++, and develops on the main branch, where the VERSION_NUMBER file currently reads 1.32.0. The CUDA provider documentation lists GPU package requirements for releases up to 1.30.x. Releases are published on the GitHub releases page, and a public roadmap at onnxruntime.ai describes upcoming release dates and features.

The project is licensed under the MIT License, copyright Microsoft Corporation, and follows the Microsoft Open Source Code of Conduct. Bugs and feature requests go to GitHub Issues and general questions to GitHub Discussions. Companion repositories hold inference examples (microsoft/onnxruntime-inference-examples) and training examples (microsoft/onnxruntime-training-examples). The source is at github.com/microsoft/onnxruntime, and documentation and the installation matrix are at onnxruntime.ai.