TokenSpeed: LLM Inference Engine for Agentic Workloads

What TokenSpeed is

TokenSpeed is an open-source LLM inference engine built for agentic workloads. Its README describes the goal as "TensorRT-LLM-level performance and vLLM-level usability" and positions it as an engine for production agent traffic: long contexts, many tool-calling turns and latency-sensitive decoding. It exposes an OpenAI-compatible HTTP API, so existing clients can talk to it by changing a base URL.

The project is developed by the LightSeek Foundation, which the README describes as a non-profit, and it says the engine receives hardware and engineering support from vendors including NVIDIA and AMD. According to the README, TokenSpeed joined the PyTorch Ecosystem Landscape in August 2026. The intended audience is teams that self-host large open-weight models, from single-GPU deployments of small models to multi-GPU servers running large mixture-of-experts models.

The code is MIT-licensed. The repository is mostly Python, with a C++ scheduler and a separate kernel package.

How it works

TokenSpeed's main architectural claim is a split between a control plane and an execution plane. The README lists three core components:

• Scheduler: a C++ control plane with a Python execution plane. Request lifecycle, KV cache ownership and overlap timing are encoded as a finite-state machine, and safe KV resource reuse is enforced by the type system at compile time rather than checked at runtime.
• Kernels: a pluggable, layered kernel system with a portable public API and a centralized registry. The README highlights its MLA (Multi-head Latent Attention) implementation on NVIDIA Blackwell, and the repository includes a tokenspeed-mla component.
• Entrypoint: an SMG-integrated AsyncLLM for low-overhead request handling on the CPU side.

The README quotes the PyTorch Foundation's summary of the design: the C++ state machine gives strong correctness guarantees for scheduling and KV cache state, while keeping the execution layer in Python lowers the cognitive load for researchers and engineers changing model code. Kernels are treated as a first-class subsystem, separated from the core engine through a public API, a selection model and a plugin mechanism intended for heterogeneous accelerators.

In practice the engine is installed as three Python packages: the runtime (tokenspeed), the kernels (tokenspeed-kernel) and the scheduler (tokenspeed-scheduler). Nightly wheels pin each TokenSpeed build to the same-date kernel build.

Key features

• OpenAI-compatible server: tokenspeed serve starts an HTTP server exposing the API under /v1, usable from the standard OpenAI client.
• Familiar parameters: the docs say TokenSpeed keeps familiar serving parameter names where the meaning matches, such as --max-model-len, --kv-cache-dtype, --gpu-memory-utilization, --max-num-seqs and --tensor-parallel-size, so existing launch recipes port with fewer changes. TokenSpeed-specific knobs are documented separately.
• Parallelism options: tensor, data and expert parallelism are set from the command line to match the hardware topology.
• Quantised serving: launch examples use FP8 KV caches and NVFP4 weights, and the news section mentions FP4 and MXFP4 deployments.
• Selectable backends: attention and MoE backends are chosen explicitly, for example trtllm_mla or triton attention, which the docs recommend for benchmark and production runs.
• Reasoning and tool-call parsers: model-family parsers can be set for chat formats that need them.
• NVIDIA and AMD support: CUDA nightly wheels, separate ROCm 7.2 nightly wheels, and an initial dense BF16 path for AMD RDNA4 (gfx1201) cards using portable Triton kernels.

Getting started

The getting-started guide uses a GPU runner container. Pull and start it:

docker pull lightseekorg/tokenspeed-runner:latest

On a CUDA 13 host, in a Python 3.11–3.13 environment, install the nightly wheels:

python -m pip install --upgrade tokenspeed --extra-index-url https://lightseek.org/whl/nightly

Check the installation:

tokenspeed env
tokenspeed serve --help

Launch a minimal server:

tokenspeed serve openai/gpt-oss-20b \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1

Then call it from the OpenAI Python client, as in the launching guide:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

The docs also cover installing from source (editable installs of the runtime, kernel and scheduler packages), a Hopper / CUDA 12.9 recipe, ROCm installation and model-specific recipes for Kimi and GPT-OSS deployments.

Launch checklist from the docs

The launching guide recommends explicit parameters in scripts so deployments are reproducible. Its checklist: put the model path directly after tokenspeed serve; set --host, --port and --served-model-name for the API surface; set --max-model-len, --kv-cache-dtype and --gpu-memory-utilization before tuning concurrency; match --tensor-parallel-size, --data-parallel-size and --enable-expert-parallel to the hardware topology; set --reasoning-parser and --tool-call-parser when the chat format needs them; and choose backends explicitly for benchmark or production runs. The docs index adds a simple workflow: pick a recipe close to your model family and hardware, then validate correctness and throughput together before changing more than one tuning dimension at a time.

Use cases

• Serving coding agents: hosting a large open-weight model behind an OpenAI-compatible endpoint for agent frameworks that make many long-context calls.
• Large MoE models on multi-GPU nodes: the production launch skeleton serves an NVFP4 Kimi K2.5 model with tensor and expert parallelism, a 262,144-token context and an FP8 KV cache.
• Day-0 model enablement: the README's news section lists day-0 support announcements for several 2026 model releases, which matters for teams that want to serve new open models quickly.
• AMD deployments: running on Instinct GPUs via the ROCm wheels, or on Radeon RDNA4 cards for smaller dense models.
• Kernel and scheduler research: the Python execution plane and pluggable kernel registry are designed for engineers experimenting with new kernels or model code.

How it compares

The README frames TokenSpeed against two established engines: it aims for the performance of NVIDIA's TensorRT-LLM and the usability of vLLM. It publishes a Pareto-curve chart comparing TokenSpeed with TensorRT-LLM on agentic workloads with Kimi K2.5 on B200, and the news section claims figures such as 580 tokens per second on Qwen3.5-397B-A17B; these are the project's own benchmarks and should be reproduced on your own hardware and traffic before relying on them. Compared with vLLM and SGLang, which have larger communities and longer track records, TokenSpeed is much newer. Its parameter compatibility lowers the cost of trying it alongside an existing engine.

Things to know before adopting

• Maturity: the repository was created in May 2026 and installs are distributed as dated nightly wheels (for example 0.1.0.post20260930). Pin a specific nightly for reproducible deployments.
• Hardware: the runner container targets NVIDIA GPU hosts, the CUDA nightlies expect CUDA 13, and the compatible-parameters doc says --device currently serves CUDA. ROCm uses a separate index, and the docs warn not to configure the CUDA and ROCm indexes at the same time.
• Runner privileges: the example docker run uses --privileged, host networking and host PID namespaces; review that before using it outside a dedicated machine.
• Benchmarks are vendor-reported: performance numbers in the README and blog are the project's claims.
• Licence: MIT, which permits commercial use and modification with attribution.

Project activity

As of October 2026 the repository has about 2,700 stars. It was created in May 2026, is written mainly in Python and is licensed under MIT, with pushes as recent as October 18, 2026. Documentation lives in the repository's docs/ folder and on the LightSeek site. Source is at github.com/lightseekorg/tokenspeed and the announcement post at lightseek.org.

[![Featured on TechLogHub](https://techloghub.com/badge-neutral.svg)](https://techloghub.com/open-source/tokenspeed?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)