AX: Declarative Orchestrator for AI Agent Workloads

What AX is

AX is a high-throughput, declarative orchestrator for running autonomous agent workloads in a cluster. The README sums up the workflow in one line: declare an agentic task with workspaces and model specifications, and AX sandboxes it, wires up its workspace, and helps run it at scale. It runs on top of Agent Substrate for sandboxed execution, and the project states it is built to run billions of tasks per cluster.

The motivation is that agents are a new kind of workload. The README argues they are neither stateless microservices nor run-to-completion batch jobs: they accumulate state, need strict isolation, call out to model APIs and tool servers, and can burn money in a loop if nobody is watching. Existing schedulers were designed for the first two shapes. AX gives agents their own small set of primitives instead.

The audience is platform teams who need to run many agents for many users or jobs, such as coding agents working against repositories, and who are already comfortable with Kubernetes. The README says that if you have used Kubernetes, ax will feel similar, and the CLI is deliberately shaped like kubectl.

Core concepts

Every AX resource lives in an atespace, with default as the default. There are three resource kinds, all expressed as ax.io/v1alpha1 manifests.

Task

The smallest unit of isolated execution. A Task declares a container image and command, compute requests and limits, environment variables, and references to one or more Workspaces. Each workspace is mounted at its own path and the first becomes the working directory. The concepts doc is explicit that the unit is deliberately small: an agent plans, delegates, retries and fans out work, and rather than modelling that shape, AX provides a primitive that is cheap to create, isolate, suspend and throw away. A single task can be the whole job or the root of a tree of tasks the agent spawns as it breaks a problem down.

Task status has a one-word phase (Running, Suspended, Failed, Terminating) plus conditions. WorkspaceReady turns true once every workspace is set up, and Ready is the condition to wait on. Suspending sets Ready to false with reason TaskSuspended; resuming restores it.

Workspace

A Workspace makes agent setup declarative and done once. It can clone Git repositories into subdirectories, register MCP servers and registries the agent can call, and point at skill registries. Declare it once, bind it from many tasks, and the runner materializes it inside each sandbox before the command starts. A binding can also carry a goal, a plain-language description of the environment needed; on first boot the runner hands that goal to an agent that finishes setup, such as installing a toolchain.

Model

A Model resource configures a provider, model identifier, generation parameters and a reference to the Kubernetes secret holding credentials. Rotating a key or pinning a new model version becomes one ax apply. AX's own components read Model resources too, for example when planning a workspace from a goal.

How AX works

The design doc explains the main architectural choice. Storing millions of short-lived tasks as Kubernetes custom resources would push etcd past its comfort zone, so AX keeps its state in Redis and reconciles directly with Agent Substrate under fine-grained distributed locks. There are three binaries:

• ax: the developer CLI, which applies manifests, inspects and watches resources, and tunnels to the cluster.
• ax-server: a gRPC API on port 8080 that validates manifests, manages locks, reconciles with Agent Substrate and persists state to Redis. GET /healthz serves health checks.
• ax-task-runner: the entrypoint inside every task container. It bootstraps the workspace, serves metadata and runs the agent command, and custom runner images can embed its package directly.

Agent Substrate handles atespace provisioning, actor creation and activation, and worker assignment. Each task runs as a sandboxed actor there, which is what makes suspend and resume possible: the task's actor state is checkpointed and later restored.

Key features

• Sandboxed execution: run untrusted agent code in isolation with CPU and memory limits.
• Warm workspaces: Git repos, MCP servers and skill packages are pre-wired so every agent starts ready.
• Goal-driven setup: describe the environment in plain language and let a setup agent finish it on first boot.
• Suspend and resume: ax suspend checkpoints an idle agent; ax resume picks up where it left off.
• Shell access: ax ssh opens a shell or runs a command inside a running sandbox when the task sets debug: true.
• Live status streaming: ax watch streams phase and condition changes over a server-streaming gRPC call.
• Context-aware CLI: ax follows the active Kubernetes context and tunnels to that cluster's control plane in the background.
• Centralized model config: credentials stay in Kubernetes secrets referenced by Model resources.

Getting started

AX needs a Kubernetes cluster with Agent Substrate already installed, plus Go, kubectl, ko and a container registry your cluster can pull from. Substrate lands in the ate-system namespace; verify it before continuing:

kubectl get svc api -n ate-system

Install the CLI:

go install github.com/google/ax/cmd/ax@latest

Deploy the control plane, which installs Redis and builds the images with ko into the ax-system namespace:

make deploy AX_IMAGE_REPO=<your-registry>

Then run the example task and inspect it:

ax apply -f examples/task.yaml       # Task + Workspace + Model in one file
ax get tasks
# NAME      ATESPACE   PHASE     ACTOR           WORKER-IP    AGE
# task123   default    Running   task123         10.20.3.67   1m

ax watch task task123                # stream phase and condition changes live
ax ssh task123 -- ls -la /workspace  # poke around inside the sandbox
ax suspend task task123              # checkpoint and pause
ax resume task task123               # pick up where it left off

The repository also ships a demo.sh that applies a custom workspace, waits for readiness, runs commands over ax ssh and suspends the task.

Use cases

• Coding agents at scale: the README's own example binds a workspace that clones the Go repository and gives the task a goal of building the toolchain from source.
• Multi-tenant agent platforms: give each user or job its own isolated sandbox with resource limits instead of sharing processes.
• Long-lived, bursty agents: suspend agents waiting on humans or external events and resume them later rather than holding compute.
• Fan-out workflows: an agent can spawn trees of small tasks, each with the same sandbox and lifecycle.
• Debugging agent behaviour: shell into a live sandbox to see what an agent is doing.

How it compares

The natural reference point is Kubernetes itself, which the CLI deliberately imitates. The difference is in the workload model and storage: AX treats each agent as a suspendable sandboxed actor rather than a Pod, Deployment or Job, and keeps state in Redis rather than etcd to handle very large numbers of short-lived tasks. In practice AX runs on Kubernetes rather than replacing it. Agent frameworks, by contrast, define what an agent does; AX is concerned with where and how that code runs, and it is agnostic to what the task's command is.

Things to know before adopting

• Early and unstable: the README carries a warning that AX and several features are in heavy development and will likely introduce major breaking changes before a stable release. The API version is v1alpha1.
• Hard dependency on Agent Substrate: AX schedules every task on Agent Substrate, which must be running in the cluster first.
• Build-your-own install: deployment currently means building images with ko and pushing to your own registry.
• Redis is part of the control plane: plan for its availability and persistence like any other stateful dependency.
• Young project: the repository was created in March 2026. A roadmap in the docs covers planned work on core specs, actor architecture, agentic environments and governance.

Project activity

As of October 2026 the repository has roughly 12,800 stars. It was created on 30 March 2026, is written primarily in Go, and is licensed under Apache-2.0. The source is at github.com/google/ax and the project homepage is agentexecutor.io. Concepts, manifests, sandbox, runner, networking and development guides are in the repository's docs/ directory.