HAMi: GPU Sharing and Accelerator Scheduling on Kubernetes

What HAMi is

HAMi stands for Heterogeneous AI Computing Virtualization Middleware. It is a Kubernetes-native layer that lets platform teams share GPUs and other AI accelerators across workloads, isolate device memory and compute between them, and schedule pods with device-aware policies, without changing application code. The project was formerly known as k8s-vGPU-scheduler, and it is a CNCF Incubating project listed in both the CNCF Landscape and the Cloud Native AI (CNAI) Landscape.

The problems it addresses come up in almost every shared AI cluster. Stock Kubernetes hands out accelerators as whole devices, so a small notebook or inference job can occupy an entire expensive GPU. Teams compete for scarce devices, different hardware vendors expose different operational models, and the default scheduler has little device context to place workloads well. HAMi adds fractional allocation, isolation and smarter placement on top of the standard Kubernetes resource model.

The audience is platform and infrastructure teams running Kubernetes for training, inference and notebooks, including private AI clouds and clusters that mix accelerators from several vendors.

How it works

HAMi is made of four kinds of components: a mutating webhook, a scheduler extender, device plugins, and device-specific in-container virtualization components. The README traces the path of a pod through them:

Pod submission
  -> HAMi mutating webhook
  -> HAMi scheduler filter / score / bind
  -> device allocation written to pod annotations
  -> device plugin Allocate()
  -> container runtime environment
  -> HAMi monitor and metrics

The webhook intercepts pods that request accelerator resources, the scheduler extender filters and scores nodes and devices, and the chosen allocation is recorded in pod annotations. The device plugin then hands the allocation to the container runtime, and the in-container component enforces the limits where the device backend supports it. A monitor exposes metrics about what was allocated and used.

Workloads keep using ordinary resource requests and limits. For example, this pod asks for one physical NVIDIA GPU with 3 GiB of GPU memory, expressed as nvidia.com/gpumem: 3000:

resources:
  limits:
    nvidia.com/gpu: 1
    nvidia.com/gpumem: 3000

One detail worth knowing: after HAMi is installed, the nvidia.com/gpu value registered on a node defaults to the number of vGPUs, while in a pod request nvidia.com/gpu still means the number of physical GPUs the pod needs.

Key features

• Device sharing: allocate a fraction of a physical accelerator by memory, by core, or by device count.
• Resource isolation: enforce per-workload accelerator memory and compute limits where the backend supports it.
• Scheduling policies: binpack to consolidate, spread to reduce contention, topology-aware placement, mutex to allocate only idle GPUs, and a NUMA sort key that chains with binpack or spread.
• Per-pod overrides: cluster defaults can be overridden with the hami.io/node-scheduler-policy and hami.io/gpu-scheduler-policy annotations, including combinations such as binpack,numa.
• Dynamic MIG: create, cache, reuse and reclaim NVIDIA MIG instances dynamically on supported cards.
• Many vendors: backends for NVIDIA (including MIG), AMD, Huawei Ascend, Cambricon, Hygon, Iluvatar, Kunlunxin, MetaX, Moore Threads, Enflame, Biren, VastAI, AWS Neuron and remote NVIDIA GPUs via Lupine.
• Observability: a metrics endpoint, Grafana dashboards, the separate HAMi-WebUI project, and agent skills for debugging GPU pods.
• Deployment options: scheduler high availability with leader election, OpenShift and SELinux settings, a NetworkPolicy option and Prometheus Operator ServiceMonitors.

Getting started

For the NVIDIA path the prerequisites are an NVIDIA driver of version 440 or newer, the NVIDIA Container Toolkit configured as the default runtime for containerd, Docker or CRI-O, Kubernetes 1.23 or newer, glibc 2.17 or newer, Linux kernel 3.10 or newer, and Helm 3. Label the GPU nodes HAMi should manage:

kubectl label nodes <node-name> gpu=on

Add the Helm repository and install the chart:

helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update
helm install hami hami-charts/hami -n kube-system

Check that the scheduler and device plugin are running:

kubectl get pods -n kube-system

Once hami-device-plugin and hami-scheduler are both Running, submit the example workload:

kubectl apply -f examples/nvidia/default_use.yaml

Vendor settings live under devices.<vendor> in the chart's values.yaml, and pod examples for each vendor are in the repository's examples/ directory. Metrics are served at the scheduler monitor endpoint, on port 31993 by default. For production, setting scheduler.leaderElect=true with more than one scheduler replica runs several replicas behind one leader, and the chart creates a PodDisruptionBudget for them.

Use cases

• Raising GPU utilisation: pack many small inference services or notebooks onto shared GPUs instead of giving each a whole device.
• Multi-tenant AI platforms: run notebooks, training and inference for several teams on one accelerator pool with fair allocation and quotas.
• Serving several models per GPU: the README lists a vLLM integration where GPU memory caps let multiple inference servers share one GPU.
• Heterogeneous clusters: schedule across NVIDIA, Ascend, Cambricon, Hygon and other vendors through a single workflow.
• Batch AI scheduling: combine HAMi with Volcano for gang and queue-based scheduling, or with Kueue through ResourceTransformation.

How it compares

The natural baseline is the standard Kubernetes device plugin model, where accelerators are allocated as whole devices; HAMi's reason to exist is adding fractional allocation, isolation and device-aware scoring on top. The README positions HAMi as complementary to other tools rather than a replacement: it can coexist with the NVIDIA GPU Operator, with HAMi handling scheduling and the Operator handling drivers, and it works with kube-scheduler as well as Volcano and Kueue for batch workloads. Its distinguishing trait among GPU-sharing approaches is breadth of vendor support, which matters most for clusters that are not NVIDIA-only.

Things to know before adopting

• Licence: Apache-2.0, with the project established as HAMi a Series of LF Projects, LLC.
• Capabilities vary by device: the README notes that isolation and features depend on the vendor, model, driver and hardware generation, so check the maintained support matrix on the project site for your hardware.
• Webhook failure mode: with the NetworkPolicy option enabled, some CNIs do not match host-networked kube-apiserver traffic, which can make the webhook unreachable; with failurePolicy=Ignore, pods are then admitted without HAMi's changes.
• Node resource semantics change: node-level nvidia.com/gpu counts become vGPU counts after installation, which can surprise dashboards or tooling that read node capacity.
• Runs in kube-system: the default install places the scheduler and device plugin in the kube-system namespace, next to other cluster system components, and the webhook sits in the admission path of every pod that requests accelerator resources.

Project activity

As of October 2026 the repository has about 4,800 stars. It was created in September 2021, is written in Go and is licensed under Apache-2.0, with commits landing in October 2026. The project is governed by its maintainers under a published governance document, holds an OpenSSF Best Practices badge, and runs weekly Chinese and biweekly English community meetings, with discussion on Discord and the CNCF Slack. Talks about the project have been given at KubeCon Europe 2024, KubeDay Japan 2024 and KubeCon + AI_dev Open Source GenAI & ML Summit China 2024, and container images are published on Docker Hub as projecthami/hami. The source is at github.com/Project-HAMi/HAMi and the documentation at project-hami.io.