nvme-cli and libnvme NVM-Express user space tooling for Linux

nvme-cli and libnvme: A Detailed Guide to NVM-Express User Space Tooling for Linux

Introduction

The NVMe (Non-Volatile Memory Express) standard has transformed how fast storage is accessed on modern systems. At the user-space level, nvme-cli and its companion library, libnvme, provide a powerful toolkit for interacting with NVM-Express devices, configuring NVMe over Fabrics, querying device capabilities, running tests, and managing resources. The project combines command-line tooling (nvme-cli) with a supportive library (libnvme) that together form a flexible, extensible interface for Linux environments.

What you’ll find in this project

• A cohesive set of commands and plugins designed to work with a wide range of NVMe devices and subsystems.
• Tight integration between nvme-cli and libnvme starting with nvme-cli 3.x, where libnvme is built directly into the nvme-cli source tree for easier maintenance and deployment.
• Rich tooling designed to support both traditional NVMe devices and NVMe over Fabrics (NVMe-oF) configurations.
• A multi-layer quality assurance approach that includes static analysis, code coverage, and ongoing testing.

Project at a glance: badges and indicators

The project ships with a collection of visual badges that summarize status, licensing, documentation, and packaging availability. These badges provide a quick snapshot of the project’s health and ecosystem:

• MesonBuild badge: a badge indicating build status via the Meson build system MesonBuild

• nvme-cli license and docs: the project’s license and documentation status GitHub license for nvme-cli Docs for nvme-cli

• libnvme license and docs: the companion library’s license and documentation GitHub license for libnvme Docs for libnvme

• Python bindings and builds for libnvme PyBuild badge PyPI for libnvme PyPI - Wheel for libnvme

Quality assurance and testing

• Static analysis and code quality

• clang-analyzer integration

• Coverity Scan build status

• Test coverage

• Codecov badge showing test coverage on the master branch

These indicators appear at the repository level and reflect ongoing commitments to code quality, testing, and reliability.

What nvme-cli and libnvme do

• nvme-cli: A Linux user-space CLI that interacts with NVM-Express devices, enabling administrators and developers to inspect, configure, and manage NVMe hardware and NVMe-oF networks from the command line.
• libnvme: A supporting library that provides the underlying interfaces used by nvme-cli to communicate with NVMe devices, expose analytics, and enable advanced features such as plugins and config discovery.

A key architectural shift in recent versions

Starting with nvme-cli 3.x, libnvme is fully integrated into the nvme-cli source tree. This means:

• No external libnvme repository or separate package is required for building nvme-cli.
• All necessary libnvme and libnvme-mi code is bundled with nvme-cli and built together.
• A streamlined build and packaging experience across most environments.

Dependency and optional feature landscape (a practical view)

The project relies on a core set of components, with optional pieces that unlock additional capabilities. The essential relationships are described below, translated into a practical, non-tabular view:

Core dependencies and integration

• libnvme and libnvme-mi: Integrated directly into the nvme-cli source tree. No external dependency required; everything is built together.
• json-c: Optional but highly recommended. When available, json-c enables richer JSON output formats and broader plugin support. Without json-c, many vendor plugins and the JSON output format are disabled.
• libkmod: Optional. If absent, nvme-cli won’t be able to load the nvme-fabrics module when needed, which can limit fabric-related functionality.
• The project is designed to be flexible: if you lack certain components on your system, nvme-cli gracefully disables the corresponding features rather than failing entirely.

Optional feature dependencies that unlock functionality

• json-c: Default behavior is auto-detection; enabling json-c unlocks:
• JSON output formats
• Access to /etc/nvme/config.json parsing
• All vendor plugins
• openssl: Auto-detects and enables TLS over NVMe-TCP and host authentication features
• keyutils: Auto-detects and unlocks key management for NVMe-oF authentication
• libdbus: Disabled by default; when enabled, it supports end-point discovery for NVMe-MI
• liburing: Disabled by default; when enabled, enables Get-log-page via io_uring passthrough
• python: Auto-detects and enables Python bindings for libnvme
• You can explicitly disable Python bindings as an example: meson setup .build -Dpython=disabled
• The meson options file (meson_options.txt) lists all the available options; you can inspect them to tailor your build to your environment.

Configuring and building with Meson: a practical workflow

• In a modern environment, you can use Meson directly or via a Makefile wrapper. For older environments, you might choose alternative build tools like samurai and muon.
• The build environment typically requires:
• A C/C++ compiler (gcc or clang)
• Ninja
• Meson
• A minimal, modern build setup is straightforward:
• meson setup .build
• meson compile -C .build
• meson test -C .build (to run unit tests)
• meson install -C .build (to install)
• Key installation paths and options
• By default, installations occur under /usr/local (executables in /usr/local/bin, libraries in /usr/local/lib, configuration in /usr/local/etc, etc.)
• You can customize the destination with: meson setup .build --prefix /usr --sysconfdir /etc
• For a production-ready build with optimizations and reduced debugging, you can specify: meson setup .build --prefix /usr --sysconfdir /etc --buildtype release
• Static vs shared libraries
• To generate a static library instead of a shared one: meson setup --default-library=static .build
• This can be useful for environments that favor static binaries for portability.
• Debugging and sanitizers
• For debugging builds: meson setup .build --buildtype=debug
• To enable address sanitizer: meson setup .build -Db_sanitize=address
• If you enable sanitizers, you may need to preload libasan for runtime checks: LD_PRELOAD=/lib64/libasan.so.6 ninja -C .build test
• Undefined behavior sanitizer is also supported (e.g., -Db_sanitize=address,undefined)
• Build wrappers and alternative approaches
• The project includes a scripts/build.sh wrapper, used by CI but also available for local builds.
• Example usage: scripts/build.sh To use clang instead of the default compiler: scripts/build.sh -c clang To enable fallback options: scripts/build.sh fallback
• Minimal static build with muon scripts/build.sh -m muon This approach downloads and builds samurai and muon instead of relying on meson directly. The result is a static binary with minimal external dependencies.
• Makefile wrapper for backwards compatibility
• There is a Makefile wrapper that delegates to Meson so users familiar with make can continue to work as they did in older iterations.
• Typical usage: make make install
• There is also a make rpm target for RPM-based distributions and a make static target for static binary builds.

Plugins and plugin selection

• By default, nvme-cli builds all vendor plugins.
• If you want to limit the plugins, you can specify a specific set: meson setup .build -Dplugins=intel,wdc,ocp
• You can also build with the Makefile wrapper, selecting plugins via: make PLUGINS="intel,wdc,ocp"
• If you want to exclude plugins, you can set PLUGINS="" to build with no vendor plugins.
• The default for plugin selection is "all," which means all plugins are included.

Distro support and packaging

• nvme-cli is designed to be available across a broad range of Linux distributions:
• Alpine, Arch, Debian, Fedora, Ubuntu, openSUSE, and more
• Typical package name across distributions is nvme-cli
• OpenEmbedded/Yocto: An nvme-cli recipe exists as part of the meta-openembedded layer
• Buildroot: nvme-cli is available as a buildroot package named nvme Kernel and runtime requirements
• libnvme relies on the Linux kernel's NVMe subsystem interface at /sys/class/nvme-subsystem
• This interface was introduced in Linux kernel version 4.15
• Therefore, nvme-cli requires kernel version 4.15 or later to operate correctly Contributing and AI-assisted development resources
• The project encourages community contributions, including new commands, plugin integration, and API naming conventions
• Guidelines and workflow details are described in the CONTRIBUTING.md file
• There is a companion repository for AI-assisted development resources: nvme-cli-ai (https://github.com/linux-nvme/nvme-cli-ai) Persistent versus volatile configuration: coordinating with test tools
• Persistent configuration
• Two main locations exist for persistent, long-lived configuration:
  • /etc/nvme/discovery.conf (old style)
  • /etc/nvme/config.json (new style)
• Volatile configuration
• Volatile configuration is used to coordinate access to global resources and can be created by third-party tools like nvme-stats or blktests
• These configurations live in /run/nvme
• The key purpose of volatile configuration is to prevent automated actions from triggering in test contexts (for example, during a blktests run)
• When tools like blktests spawn nvme-cli, they may generate a JSON file in /run/nvme to inform nvme-cli to refrain from performing certain actions in the udev context
• Important: updates to volatile configuration should be atomic to avoid partial reads by nvme-cli; the recommended procedure is to write to a temporary file in /run/nvme (without the .json extension), then rename the file to finalize the JSON file name Example volatile configuration shape { "hostnqn": "nqn.2014-08.org.nvmexpress:uuid:242d4a24-2484-4a80-8234-d0169409c5e8", "hostid": "242d4a24-2484-4a80-8234-d0169409c5e8", "subsystems": [ { "application": "blktests", "nqn": "blktests-subsystem-1", "ports": [ { "transport": "fc", "traddr": "nn-0x10001100aa000001:pn-0x20001100aa000001", "host_traddr": "nn-0x10001100aa000002:pn-0x20001100aa000002" } ] } ] }
• Practical note: when updating the volatile configuration at runtime, perform the update atomically, using a temporary file and a rename operation to finalize the new configuration. This ensures nvme-cli always sees a complete and valid configuration file.

Testing, CI, and reproducibility

• For pre-built binaries, CI build reproducibility, and container-based debugging, the project maintains TESTING.md as a centralized reference
• The TESTING.md document provides guidance on reproducing builds, running tests locally, and debugging containerized environments
• This structured approach helps maintainers and users alike verify behavior across different environments and toolchains

Running through a representative build and test flow (a concise recap)

• Ensure you have a modern Linux environment with a supported kernel (4.15+)
• Install or prepare necessary build tools:
• gcc or clang
• ninja
• meson
• Optionally install json-c, openssl, keyutils, libdbus, liburing, and python to unlock related features
• Configure the build:
• meson setup .build
• If you want to customize, add flags like: -Dplugins=intel,wdc,ocp -Dpython=disabled --prefix /usr --sysconfdir /etc
• Build and install:
• meson compile -C .build
• meson install -C .build
• Run tests:
• meson test -C .build
• If you prefer a static binary:
• meson setup --default-library=static .build
• If you need a minimal static build with muon:
• scripts/build.sh -m muon
• For a quick CI-aligned workflow, you can use:
• scripts/build.sh
• Or run clang explicitly: scripts/build.sh -c clang

Open-world usage and exploration

nvme-cli and libnvme are designed to be extensible and compatible with a broad ecosystem of NVMe devices, NVMe over Fabrics configurations, and related tooling. The project emphasizes:

• Compatibility with a wide range of Linux distributions via conventional packaging names
• Ease of integration into custom enterprise environments through modular plugin architecture
• Clear configuration strategies that balance persistent settings with the need for ephemeral, test-driven volatility

Conclusion: why this toolset matters

NVMe has reshaped storage with speed and scalability, and nvme-cli paired with libnvme gives administrators and developers a robust, flexible interface to explore, configure, and test NVMe devices in Linux environments. The integration of libnvme into the nvme-cli source tree since 3.x simplifies building and distribution, while optional dependencies and plugins unlock a broad spectrum of capabilities without forcing unnecessary complexity on every user. With careful configuration, thorough testing, and an emphasis on clean, atomic configuration management, nvme-cli remains a cornerstone tool for anyone working with NVMe hardware or NVMe over Fabrics in Linux environments.

Images and visuals referenced in the article

• MesonBuild badge: https://github.com/linux-nvme/nvme-cli/actions/workflows/build.yml/badge.svg
• nvme-cli license badge: https://img.shields.io/github/license/linux-nvme/nvme-cli
• nvme-cli docs badge: https://img.shields.io/readthedocs/nvme-cli
• libnvme license badge: https://img.shields.io/github/license/linux-nvme/libnvme
• libnvme docs badge: https://img.shields.io/readthedocs/libnvme
• PyBuild badge: https://github.com/linux-nvme/nvme-cli/actions/workflows/libnvme-release-python.yml/badge.svg
• PyPI badge for libnvme: https://img.shields.io/pypi/v/libnvme
• PyPI Wheel badge for libnvme: https://img.shields.io/pypi/wheel/libnvme
• Quality badges for static analysis and test coverage (references to clang-analyzer, Coverity, and codecov status)

Note: All imagery is embedded or linked as provided in the input and serves to illustrate build status, licensing, and documentation readiness for nvme-cli and libnvme.