vLLM: High-Throughput LLM Inference and Serving Engine

What vLLM is

vLLM is a Python library for large language model inference and serving. Its tagline is "Easy, fast, and cheap LLM serving for everyone", and the README describes it as a fast and easy-to-use library for LLM inference and serving. It was originally developed in the Sky Computing Lab at UC Berkeley and, according to the README, is now built and maintained by a community spanning many dozens of academic institutions and companies, with more than 2,000 contributors.

The problem vLLM addresses is the gap between loading a model in a research framework and serving it efficiently to many concurrent users. Serving a transformer naively wastes GPU memory on the key-value (KV) cache that attention needs for every in-flight request, which limits how many requests fit on a card and pushes up the cost of every generated token. vLLM was built around a memory-management technique called PagedAttention, described in the project's SOSP 2023 paper "Efficient Memory Management for Large Language Model Serving with PagedAttention", and has since grown into a general-purpose inference engine with an OpenAI-compatible HTTP server on top.

It is aimed at ML platform teams, backend engineers who need to self-host open-weight models behind an API, researchers running large offline batch jobs, and hardware vendors who want their accelerators to run the same serving stack as everyone else.

How it works

vLLM has two main entry points. The first is the LLM class for offline batched inference inside a Python process: you construct it with a Hugging Face model name, pass a list of prompts and a SamplingParams object, and call generate. The engine queues the prompts and executes them with high throughput, returning a list of RequestOutput objects. The second is the vllm serve command, which starts an HTTP server implementing the OpenAI API protocol, so existing OpenAI clients can talk to it by changing the base URL. The README adds that the server also supports the Anthropic Messages API and gRPC.

The README lists the techniques behind its throughput:

• PagedAttention for efficient management of attention key and value memory.
• Continuous batching of incoming requests, together with chunked prefill and prefix caching.
• CUDA/HIP graphs, both piecewise and full, for fast model execution.
• Optimized kernels: attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA and Triton, and GEMM/MoE kernels for various precisions built with CUTLASS, TRTLLM-GEN and CuTeDSL.
• torch.compile for automatic kernel generation and graph-level transformations.
• Disaggregated prefill, decode and encode.

vLLM automatically selects the most performant attention backend compatible with the system and model. The quickstart shows how to override it with --attention-backend, for example FLASH_ATTN or FLASHINFER on NVIDIA CUDA, with separate ROCm and Intel XPU options for AMD and Intel hardware.

Key features

• Broad model coverage: the README says vLLM supports 200+ model architectures on Hugging Face, including decoder-only LLMs (Llama, Qwen, Gemma), mixture-of-experts models (Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS), hybrid attention and state-space models (Mamba, Qwen3.5), multi-modal models (LLaVA, Qwen-VL, Pixtral), embedding and retrieval models (E5-Mistral, GTE, ColBERT) and reward and classification models.
• Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt and TorchAO, among others.
• Speculative decoding using n-gram, suffix, EAGLE and DFlash methods.
• Distributed inference with tensor, pipeline, data, expert and context parallelism.
• Decoding algorithms including parallel sampling and beam search, with streaming outputs.
• Structured outputs generated with xgrammar or guidance, plus tool-calling and reasoning parsers for agent-style workloads.
• Multi-LoRA: efficient support for multiple LoRA adapters on dense and MoE layers.
• Hardware breadth: NVIDIA, AMD and Intel GPUs and x86, ARM and PowerPC CPUs, with hardware plugins for Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU and more.
• API keys on the server: pass --api-key or set VLLM_API_KEY to require a key; several keys can be passed at once, which the docs note is useful for key rotation.

Getting started

The quickstart lists Linux and Python 3.11 to 3.14 as prerequisites. On NVIDIA GPUs it recommends creating an environment with uv and letting uv pick the right PyTorch build for the installed CUDA driver:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

vllm download-kernels installs precompiled kernels that vLLM would otherwise download and compile at startup, which the docs say can add several minutes on Hopper and newer GPUs. AMD ROCm users install from an extra wheel index, Google TPU users install the separate vllm-tpu package, Ascend NPUs go through the community-maintained vLLM Ascend plugin, and Apple Silicon Macs use the vLLM-Metal project.

Offline batched inference takes a few lines of Python. The quickstart example uses the small OPT-125M model, where prompts is a plain list of strings:

from vllm import LLM, SamplingParams
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="facebook/opt-125m")
outputs = llm.generate(prompts, sampling_params)

To serve a model over HTTP, start the server and query it the same way you would query the OpenAI API. By default it listens on http://localhost:8000:

vllm serve Qwen/Qwen2.5-1.5B-Instruct
curl http://localhost:8000/v1/models

A chat completion request then looks like this:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Who won the world series in 2020?"}
        ]
    }'

Two defaults are worth knowing early. The server applies the model repository's generation_config.json when one exists, so some sampling defaults come from the model creator rather than from vLLM; pass --generation-config vllm to turn that off. And llm.generate does not apply a chat template, so for instruct or chat models use llm.chat or apply the tokenizer's template yourself.

Use cases

• A self-hosted OpenAI-compatible endpoint: run an open-weight model behind vllm serve and point existing OpenAI SDK code at it by swapping the base URL and key.
• Offline batch jobs: summarize, classify or label a large prompt set with the LLM class, where continuous batching keeps the GPU busy without an HTTP layer in the way.
• Serving many fine-tunes from one base model: multi-LoRA support lets a single deployment host several adapters instead of one full model copy per customer or task.
• Large mixture-of-experts models: tensor, pipeline, data, expert and context parallelism split models that do not fit on one accelerator.
• Retrieval back ends: the supported embedding and retrieval models make vLLM usable for the embedding side of a RAG pipeline as well as for generation.
• Agent back ends: structured output, tool-calling and reasoning parsers cover the features agent frameworks expect from an inference API.

How it compares

The README does not position vLLM against named competitors. In the same category, teams evaluating it usually also look at Hugging Face Text Generation Inference, NVIDIA TensorRT-LLM and SGLang for server-side serving, and at llama.cpp or Ollama for single-machine use. Based on vLLM's own documentation, its distinguishing points are breadth (200+ architectures, a long list of quantization formats and many hardware back ends through plugins), the OpenAI-compatible server that also speaks the Anthropic Messages API, and the PagedAttention memory model the project originated.

The quickstart is clearly written for Linux servers with accelerators, while desktop-oriented tools focus on one-command local use. Throughput depends heavily on the model, quantization, request shape and GPU, so the only comparison that matters is one run on your own workload.

Things to know before adopting

• Licence: Apache-2.0, a permissive licence with an explicit patent grant, so embedding vLLM in commercial services is straightforward from a licensing standpoint.
• Platform: Linux is the documented primary platform. macOS support comes through vLLM-Metal, a separate project that uses MLX instead of PyTorch and requires MLX-optimized models from the mlx-community organization on Hugging Face.
• One model per server: the quickstart states that the OpenAI-compatible server currently hosts one model at a time, so multi-model setups need several processes or a router in front.
• Authentication is opt-in: without --api-key or VLLM_API_KEY, anyone who can reach the port can use the server. Bind it carefully or put it behind a gateway.
• Startup time: kernel downloads and compilation can slow the first start; run vllm download-kernels after installing or upgrading.
• Model sources: models download from Hugging Face by default; setting VLLM_USE_MODELSCOPE switches to ModelScope.
• Fast-moving project: backends, kernel options and hardware tabs change between releases, and some hardware support (for example Ascend) lives in separately versioned plugins. Pin versions in production and read release notes before upgrading.

Project activity

As of October 2026 the repository has about 94,700 stars and more than 23,000 forks. It was created in February 2023, is written primarily in Python, is licensed under Apache-2.0, and was receiving commits on the day of writing. Technical questions go to GitHub issues, user discussion to the vLLM Forum, contributor coordination to the project Slack, and security reports to GitHub Security Advisories.

Source code is at github.com/vllm-project/vllm, the project site is vllm.ai, documentation lives at docs.vllm.ai, and the original PagedAttention paper is on arXiv.

[![Featured on TechLogHub](https://techloghub.com/badge-neutral.svg)](https://techloghub.com/open-source/vllm?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)