Apache Kafka: Distributed Event Streaming Platform

What Apache Kafka is

Apache Kafka is an open-source distributed event streaming platform. Its README describes it as a system used by thousands of companies for high-performance data pipelines, streaming analytics, data integration and mission-critical applications. In practical terms, Kafka is a cluster of servers that accepts streams of records from producers, stores them durably in an append-only, partitioned log, and lets any number of consumers read those records at their own pace, as many times as they need.

The project documentation frames event streaming as capturing data in real time from sources such as databases, sensors, mobile devices, cloud services and applications; storing those streams durably for later retrieval; processing and reacting to them as they happen or retrospectively; and routing them to other systems. Kafka bundles three capabilities to do this end to end: publishing and subscribing to streams of events (including continuous import and export from other systems), storing streams reliably for as long as you want, and processing streams as they occur.

Kafka is written in Java and Scala, licensed under Apache-2.0 and developed under the Apache Software Foundation. It runs on bare metal, virtual machines and containers, on-premises or in the cloud, and the introduction notes that you can either self-manage it or use fully managed services offered by a variety of vendors. The audience is broad: backend engineers building event-driven services, data engineers wiring pipelines between databases and warehouses, platform teams running shared messaging infrastructure, and anyone who needs a durable buffer between systems that produce and consume data at different rates.

How Kafka works

Servers, clients and the protocol

Kafka is a distributed system of servers and clients that talk over a high-performance TCP protocol. A cluster can be one server or many, and can span datacenters or cloud regions. Some servers form the storage layer and are called brokers. Others run Kafka Connect to continuously import and export data between Kafka and external systems such as relational databases or other Kafka clusters. If a server fails, the others take over its work so the cluster keeps operating without data loss, according to the introduction.

Clients are libraries for writing applications that read, write and process event streams in parallel and at scale. Kafka itself ships Java and Scala clients, including the higher-level Kafka Streams library, and the community maintains clients for Go, Python, C/C++ and many other languages, as well as REST APIs.

Events, topics and partitions

The unit of data is an event (the docs also say record or message). Conceptually an event has a key, a value, a timestamp and optional metadata headers. Producers publish events, consumers subscribe to them, and the two are fully decoupled: producers never wait for consumers. The documentation calls this decoupling a key design element behind Kafka's scalability.

Events are organized into topics, which the docs compare to folders in a filesystem. Topics are multi-producer and multi-subscriber, and unlike traditional message queues, events are not deleted when they are consumed. Instead, each topic has a retention setting that controls how long events are kept, after which old events are discarded. The docs state that Kafka's performance is effectively constant with respect to data size, so storing data for a long time is fine.

Topics are split into partitions spread across brokers, which lets clients read and write from many brokers at once. Every new event is appended to one partition. Events with the same key, such as a customer or vehicle ID, always go to the same partition, and Kafka guarantees that a consumer of a given partition reads events in the exact order they were written. For fault tolerance, each topic can be replicated across brokers and even across regions; the docs call a replication factor of 3 a common production setting. Replication happens at the partition level.

KRaft: metadata without ZooKeeper

Current Kafka runs in KRaft mode, where cluster metadata is managed by Kafka itself using a Raft-based quorum rather than an external ZooKeeper ensemble. The documentation's ZooKeeper-to-KRaft page lists the configurations that were removed, including all the zookeeper.* settings. In KRaft mode each server's process.roles is set to broker, controller or both. Controllers form the metadata quorum, with one active controller and the rest as hot standbys. The docs say admins typically pick 3 or 5 controllers: with 3 the cluster tolerates one controller failure, with 5 it tolerates two, because a majority must stay alive.

Servers that act as both broker and controller are called combined servers. The KRaft documentation says they are simpler for small use cases like development, but that controllers are less isolated in that layout and cannot be rolled or scaled separately from brokers, so combined mode is not recommended for critical deployments. Since Kafka 4.1, clusters can move from a static controller configuration to a dynamic one that supports adding and removing controllers, by upgrading the kraft.version feature with bin/kafka-features.sh.

Key features

• Five core Java/Scala APIs. The Admin API manages topics, brokers and other objects; the Producer API writes events; the Consumer API reads them; the Kafka Streams API builds stream-processing applications; and the Kafka Connect API builds reusable import/export connectors.
• Durable, replayable storage. Events stay in a topic for the configured retention period regardless of whether they were consumed, so new consumers can read history and existing ones can re-read it.
• Ordering per partition. Events with the same key land in the same partition and are read in write order, which is what makes per-entity processing such as per-account ledgers practical.
• Replication and failover. Partition-level replication across brokers, optionally across datacenters, keeps data available through broker failures and maintenance.
• Kafka Streams. A client library for Java and Scala with transformations, stateful operations such as aggregations and joins, windowing and event-time processing, where input and output live in Kafka topics.
• Kafka Connect. A framework that runs connectors to continuously move data between Kafka and external systems; the quickstart demonstrates file source and sink connectors in standalone mode.
• Exactly-once processing guarantees. The introduction lists the ability to process events exactly once among Kafka's guarantees.
• Tiered storage. Brokers can offload completed log segments to a remote tier such as HDFS or S3 while keeping recent data on local disks, per KIP-405.
• Pluggable security. SSL or SASL authentication (GSSAPI/Kerberos, PLAIN, SCRAM-SHA-256/512, OAUTHBEARER), TLS encryption, and pluggable authorization with support for external authorization services.
• Official Docker images. A JVM-based apache/kafka image (from 3.7.0) and an experimental GraalVM-based apache/kafka-native image (from 3.8.0).

Getting started

The quickstart requires Java 17 or newer in the local environment. The fastest route is the official container image from the README:

docker run -p 9092:9092 apache/kafka:latest

To run a broker from a downloaded release or from compiled sources instead, the README and quickstart use the same three steps: generate a cluster ID, format the log directories in standalone KRaft mode, and start the server:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
bin/kafka-server-start.sh config/server.properties

With the broker running, open another terminal and create a topic, then inspect it:

bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092

Write a few events with the console producer (each line becomes one event, Ctrl-C exits):

bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092

And read them back from the beginning in another terminal:

bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092

Because events are stored durably, the quickstart points out that you can re-run the consumer as many times as you like, or open several consumers, and they will all see the same events. The quickstart then walks through Kafka Connect with the bundled file connectors, starting a standalone worker with one worker config and two connector configs:

bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties

A source connector reads lines from test.txt into the connect-test topic and a sink connector writes them to test.sink.txt. The final step shows a Kafka Streams WordCount topology in Java. To build Kafka from source, the README uses Gradle; ./gradlew jar builds the JARs and ./gradlew clean releaseTarGz produces a binary release tarball under ./core/build/distributions/.

To tear down the local environment and delete its data, the quickstart uses:

rm -rf /tmp/kafka-logs /tmp/kraft-combined-logs

Use cases

The project's use-cases page lists several patterns, each with its own trade-offs:

• Messaging. Kafka works as a replacement for a traditional message broker, decoupling producers from consumers and buffering unprocessed messages. The docs say it is comparable to ActiveMQ or RabbitMQ in this role, with better throughput, built-in partitioning, replication and fault tolerance.
• Website activity tracking. This was Kafka's original use case: page views, searches and other user actions published to one topic per activity type, consumed for real-time processing, monitoring or loading into offline warehouses. The docs note activity tracking is often very high volume.
• Metrics. Aggregating statistics from distributed applications into centralized feeds of operational data.
• Log aggregation. Replacing file-based log collection with a stream abstraction, which the docs say allows lower-latency processing and easier support for multiple data sources and distributed consumption.
• Stream processing. Multi-stage pipelines where raw data is consumed, enriched or aggregated, and written to new topics for downstream consumers, using Kafka Streams or other processors.
• Event sourcing. Applications that log state changes as a time-ordered sequence of records, backed by Kafka's support for very large stored logs.
• Commit log. An external commit log for distributed systems, replicating data between nodes and re-syncing failed nodes, with log compaction helping this pattern.

The introduction adds industry examples: processing payments in real time at banks and exchanges, tracking fleets and shipments, capturing IoT sensor data in factories, reacting to retail orders, and serving as the foundation for data platforms, event-driven architectures and microservices.

How it compares

The project itself draws two comparisons. For messaging, it positions Kafka alongside traditional brokers such as ActiveMQ and RabbitMQ, arguing that Kafka offers better throughput and built-in partitioning and replication, while noting that messaging workloads are often lower-throughput and more latency-sensitive. For the commit-log pattern, the docs say Kafka is similar to Apache BookKeeper.

The practical difference from a classic queue is the storage model. In a queue, a message is typically gone once it is acknowledged. In Kafka, a topic is a retained log, so many independent consumer groups can read the same data, replay it after a bug fix, or bootstrap a new service from history. The cost is operational: you plan partitions, retention, replication and disk capacity, and consumers track their own position in the log. Teams that only need simple work queues with per-message routing may find a lighter broker easier to run; teams that need a shared, durable backbone for many producers and consumers are the audience Kafka is designed for.

Compared with running Kafka yourself, the introduction acknowledges that fully managed Kafka services from various vendors exist. The open-source project is the same codebase either way; the decision is whether your team wants to own broker operations, upgrades and capacity planning.

Things to know before adopting

• Java versions. The docs say Java 17, 21 and 25 are fully supported for running Kafka, with Java 11 supported only for a subset of modules (clients, streams and related). The project recommends the most recent LTS release, Java 25 at the time of writing, and the latest patch version for security. The README says Kafka is built and tested with Java 17 and 25, and that Scala 2.13 is the only supported Scala version.
• ZooKeeper is gone. Kafka now runs in KRaft mode. Existing ZooKeeper-based clusters need the documented ZooKeeper-to-KRaft migration, and a range of ZooKeeper-era settings were removed. Some former dynamic configs, such as advertised.listeners, now require a broker or controller restart to change.
• Sensitive config isn't encrypted in KRaft. The KRaft differences page states that the password encoder settings were removed and that in KRaft mode Kafka stores sensitive data in records and the data is not encrypted in Kafka. Plan access to metadata accordingly.
• Security is opt-in. The security overview says non-secured clusters are supported, as is a mix of authenticated, unauthenticated, encrypted and unencrypted clients. The quickstart runs plaintext on localhost:9092. It also notes there is a performance cost when SSL is enabled, depending on CPU and JVM.
• Controller topology. Combined broker/controller servers are fine for development but not recommended for critical deployments; production clusters typically dedicate 3 or 5 controllers.
• Tiered storage has limits. It is off by default, Kafka ships no production RemoteStorageManager implementation (you must supply one), and the docs list no support for compacted topics, a requirement to disable it on all topics before disabling it on the broker, admin actions only from 3.0+ clients, and no support for segments missing producer snapshot files (topics created before 2.8.0).
• Native image is experimental. The GraalVM-based apache/kafka-native image is labelled experimental and intended for local development and testing only. The Docker docs also require Docker 20.10.4 or newer to avoid permission errors.
• Platform and OS tuning. The hardware page says Kafka has been tested on Linux and Solaris, and that Windows is not currently a well-supported platform. It recommends at least 100,000 allowed file descriptors for broker processes, watching vm.max_map_count as partition counts grow, and keeping Kafka data off disks shared with application logs or OS activity.
• Combined mode and local defaults. The quickstart stores data under /tmp, which is convenient for experiments but not where production data should live.

Project activity

As of November 2026 the apache/kafka repository has about 34,200 stars and around 15,660 forks on GitHub. It was created in August 2011, is written mainly in Java (with Scala), and is licensed under Apache-2.0. The default branch is trunk, which received commits on the day of writing, and the repository's CI and flaky-test report badges are linked from the README. Kafka's documentation now lives as Markdown under docs/ in the main repository.

The source is at github.com/apache/kafka, and downloads, the quickstart and full documentation are at kafka.apache.org.