Kestra: Event-Driven Workflow Orchestration

What Kestra is

Kestra is an open-source, event-driven orchestration platform for data, AI and infrastructure workflows. It puts scheduled jobs and event-driven automation behind one declarative, language-agnostic interface: a workflow, called a flow, is a short YAML document that lists tasks, inputs, outputs and triggers. The README frames the goal as bringing Infrastructure as Code practices to data, process and microservice pipelines, so the orchestration logic always ends up as versionable code even when people build it in a browser.

That dual nature is the main idea. Engineers can write flows in an editor and keep them in Git, while less code-oriented colleagues can build the same flow with a no-code editor or ask a built-in AI Copilot to draft it. All three edit the same YAML and stay in sync, and the README notes that the YAML definition is adjusted automatically whenever a flow is changed from the UI, the API, Terraform or CI/CD. The audience spans data engineers running ETL, platform teams automating infrastructure, and teams wiring LLM calls into production pipelines.

How Kestra works

A flow belongs to a namespace, which groups and isolates related flows. It contains tasks, the individual units of work such as running a script, moving data or calling an API, and triggers, which start executions on a schedule or in response to events. Inputs and variables parameterize a run, and outputs are values and files passed downstream and shown as artifacts in the UI. Kestra 2.0 adds checks, preconditions evaluated before an execution starts so a flow fails fast instead of half-running, and quotas, which cap how many executions a flow can create in a time window.

Functionality comes from plugins: the README says hundreds are built in, covering databases, cloud storage, APIs and scripts in Python, Node.js, R, Go, Shell and other languages. Task runners decide where work executes, from the local machine or remote servers over SSH to Docker containers, Kubernetes jobs, serverless containers and cloud VMs; 2.0 adds AWS EC2, Azure Virtual Machine, Google Compute Engine and Huawei Cloud CCI runners.

Architecturally, 2.0 moved worker communication from a JDBC queue to a gRPC worker-controller, separating the control plane from the data plane so workers can run in another region or inside a restricted network. Task run outputs now live in dedicated storage instead of inline in the execution record, which the README says shrinks the database and speeds up the execution list.

Key features

• Three editors, one flow: YAML editor, no-code block editor and AI Copilot all edit the same definition.
• Scheduled and real-time triggers: schedules plus event triggers for file arrivals and message buses including Kafka, Redis, Pulsar, AMQP, MQTT, NATS, AWS SQS, Google Pub/Sub and Azure Event Hubs.
• Resilience controls: retries, timeouts, error handling, subflows, conditional branching, backfills, dynamic tasks, and sequential or parallel execution.
• Git integration and drafts: push flows to a Git branch from Kestra, and save a flow as a draft while triggers keep running the last published revision.
• AI-native tasks: AI agents and RAG workflows as first-class task types, model-agnostic calls to OpenAI, Anthropic, Google Gemini, Amazon Bedrock and Ollama, and token usage reported per execution.
• MCP server and Agent Skills: a built-in MCP server exposes flows as tools to MCP-compatible agents, and Agent Skills let coding assistants such as Claude Code, Cursor and Windsurf generate flows.
• Single Loop task: 2.0 replaces ForEach and ForEachItem, and adds date helpers such as isWeekend() and isPublicHoliday() for trigger conditions.
• Tooling: an official Terraform provider, the kestractl CLI, Python and JavaScript SDKs, and a VS Code extension that mounts a namespace as a live folder.
• Notifications: plugins for Slack, email and PagerDuty alerts on workflow status.

Getting started with Kestra

The README's local quick start runs the whole server in a single Docker container. With Docker running:

docker run --pull=always --rm -it -p 8080:8080 --user=root \
  --name kestra \
  -v kestra_data:/app/storage \
  -v kestra_db:/app/data \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp:/tmp \
  -e KESTRA_PLUGINS_AUTO_INSTALL_ENABLED=true \
  kestra/kestra:latest-slim server local

The README also gives PowerShell, Command Prompt and WSL variants. The -slim image ships without bundled plugins and, with KESTRA_PLUGINS_AUTO_INSTALL_ENABLED=true, fetches each plugin from Maven Central the first time a flow needs it; kestra/kestra:latest bundles everything up front. Open the UI at http://localhost:8080 and create a first flow:

id: hello_world
namespace: dev
tasks:
  - id: say_hello
    type: io.kestra.plugin.core.log.Log
    message: "Hello, World!"

The installation guide covers other targets, including Docker Compose, Podman, Kubernetes, AWS, GCP and Azure, and the README links a CloudFormation template for AWS and a Terraform module for Google Cloud Infrastructure Manager.

Use cases

• Data pipelines: extract from databases, cloud storage and APIs, run SQL and scripts, and push results to analytics platforms such as BigQuery or Spark jobs.
• Event-driven automation: start a flow when a file lands in a bucket or a message arrives on Kafka or a cloud queue.
• Infrastructure operations: run scripts over SSH, launch Kubernetes jobs or spin up cloud VMs as task runners, with Terraform managing Kestra itself.
• LLM workflows: chunk documents into a vector store, run RAG and agent tasks, and track token usage per execution.
• Agent-callable automations: expose existing flows to AI assistants through the MCP server or the MCP Tool Trigger.
• Self-service runs: FORM inputs group parameters into a multi-step wizard so non-engineers can trigger flows safely.

How Kestra compares

Kestra sits in the workflow orchestration category alongside tools such as Apache Airflow, Dagster and Prefect, where pipelines are typically defined in Python code. Its differentiators, as the README presents them, are language-agnostic YAML definitions that can still be edited visually, first-class event triggers rather than schedule-only runs, and a plugin model in which tasks can be scripts in any language. Teams whose pipelines are deeply tied to Python abstractions may prefer a Python-native orchestrator, while mixed teams or those orchestrating infrastructure and AI tasks alongside data may find the declarative model easier to share and review.

Things to know before adopting Kestra

• Open core: the platform is Apache-2.0, but the README lists features that belong to the Enterprise Edition, including action-based RBAC, Policies, Cases, Promote, Reusable Inputs, Worker Groups 2.0 and an external log data store.
• Kestra 2.0 is a breaking release: ForEach and ForEachItem are replaced by Loop, trigger conditions are now written as when, and the -no-plugins image suffix became -slim. A migration guide covers upgrades from 1.x.
• Quick-start privileges: the local command runs as root and mounts the Docker socket so tasks can start containers; review that before using it outside a development machine.
• Plugins from Maven Central: the slim image downloads plugins on demand, which matters for air-gapped or tightly firewalled environments; the full image avoids that.
• Outbound HTTP control: operators can allow-list or deny-list the URLs HTTP tasks may reach, per the security hardening docs linked from the README.
• Storage housekeeping: PurgeStorage cleans internal storage by last-modified date, including files whose execution records are gone.

Project activity

As of October 2026 the kestra-io/kestra repository has about 29,500 stars. It was created on August 24, 2019, is written in Java and is licensed under Apache-2.0 by Kestra Technologies, with development happening on the develop branch. Documentation, the plugin catalogue and the Kestra 2.0 release notes are at kestra.io, and the community runs a Slack workspace.