Apache ZooKeeper: Distributed Coordination Service
What Apache ZooKeeper is
Apache ZooKeeper is a coordination service for distributed systems. Its overview document describes it as "a distributed, open-source coordination service for distributed applications" that "exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming." The motivation, in the project's own words, is that coordination services "are notoriously hard to get right" and prone to race conditions and deadlock, so applications should not have to build them from scratch.
In practice ZooKeeper is the small, strongly ordered store that sits underneath larger systems: a place to keep cluster membership, leader locks, configuration flags and service locations. It is not a general database. The overview says the data at each node "is usually small, in the byte to kilobyte range", and the administrator's guide calls the roughly 1 MB per-request limit "really a sanity check". Clients get a file-system-like tree, change notifications and ephemeral entries tied to a session, and build locks, barriers, queues and leader election on top.
The project lives at apache/zookeeper on GitHub, a repository created on 27 March 2009. It is written in Java, licensed under Apache-2.0, and the NOTICE.txt file reads "Copyright 2009-2026 The Apache Software Foundation". The current stable line is 3.9; this page tests 3.9.7, the newest version on the Apache download mirror when checked.
How it works
Data model: znodes, versions, ephemerals and sequences
The namespace is a tree of paths separated by slashes. Each node, a znode, can hold data and children at the same time; the overview compares it to "a file-system that allows a file to also be a directory". Every znode carries a stat structure with version numbers for data, children and ACL changes, plus creation and modification transaction IDs (czxid, mzxid). Reads return the whole value, writes replace it, and a write can be made conditional on an expected version.
Three node flavours matter most. Ephemeral znodes exist only while the session that created them is alive. Sequential znodes get a counter appended to the name; the programmer's guide says the counter is formatted as %010d, is "unique to the parent znode", and is a signed 32-bit integer that overflows past 2147483647. Container and TTL nodes, added in 3.5.x, are cleaned up by the server, but TTL nodes "must be enabled via System property as they are disabled by default".
Watches and sessions
A client can set a watch while reading. The classic watch "will be triggered and removed when the znode changes", so it fires once. Since 3.6.0 there are also persistent, recursive watches that are not removed when triggered and cover a whole subtree; the 3.9.7 CLI exposes them as addWatch. Sessions are kept alive by heartbeats. The guide says the server bounds the requested session timeout between 2 and 20 times tickTime, which with the default 2,000 ms tick is 4 to 40 seconds; the server logged exactly that range on startup.
Ensemble, leader and ZAB
Production deployments run as a replicated ensemble. Every server keeps the full tree in memory and persists a transaction log plus periodic snapshots. "As long as a majority of the servers are available, the ZooKeeper service will be available." Clients connect to any one server. Reads are answered from that server's local replica. Writes are forwarded to a single elected leader, which turns them into transactions that followers acknowledge through ZooKeeper's own atomic broadcast protocol. The ClickHouse Keeper documentation names this protocol ZooKeeper Atomic Broadcast (ZAB) and points out the consequence: reads are not linearizable by default, because each node serves them locally. ZooKeeper's guarantees section lists sequential consistency, atomicity, a single system image, reliability and timeliness; a client that needs the latest value can call sync before reading.
Each server in a replicated config needs two extra ports: one for followers to talk to the leader (2888 in the docs' examples) and one for leader election (3888). A myid file in the data directory tells each server which server.N line is its own. Optional observers receive the replicated log without voting, and dynamic reconfiguration can change membership at runtime, but the administrator's guide says reconfigEnabled defaults to false.
Key features in 3.9.7
- Hierarchical, versioned key space with conditional updates, ACLs per znode and quotas on node count or bytes.
- Coordination primitives: ephemeral, sequential, container and (opt-in) TTL nodes; one-shot and persistent recursive watches. The 3.9.7 release notes add limits on batched multi-read requests (
zookeeper.multiRead.maxOps, default 1000, andzookeeper.multiRead.maxResponseBytes, default 64 MB). - Recipes documented in the repo: the recipes guide covers barriers and double barriers, queues and priority queues, locks (including shared and revocable shared locks), two-phase commit and leader election.
- Client bindings: the getting-started guide lists Java and C bindings as "functionally equivalent", with single- and multi-threaded C variants. Third-party clients such as kazoo (Python) speak the same wire protocol.
- Operations tooling:
zkServer.sh,zkCli.sh(28 commands in 3.9.7),zkCleanup.shfor retention, and snapshot and transaction-log toolkits for offline inspection. - Monitoring: four-letter-word commands over the client port, an embedded Jetty AdminServer with JSON endpoints, JMX, and an optional Prometheus metrics provider (port 7000 by default) with a Grafana dashboard template referenced in the monitoring guide.
- Security options: TLS for client and quorum traffic, SASL/Kerberos, digest and X.509 authentication. The 3.9.6 and 3.9.7 release notes include FIPS-mode fixes, TLS cipher changes (JDK defaults since 3.9.6) and dependency upgrades for Netty and Jackson CVEs.
- High availability features: observers, read-only mode for partitioned servers (off by default), and dynamic reconfiguration (off by default).
Getting started
The project publishes source and binary tarballs through the Apache mirrors; the README says apache-zookeeper-[version]-bin.tar.gz "Contains all the jar files required to run ZooKeeper", and that Maven Central artifacts are deployed after each release vote. After unpacking, the getting-started guide asks for a config file at conf/zoo.cfg:
tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
Then the server starts with:
bin/zkServer.sh start
And the CLI connects with:
$ bin/zkCli.sh -server 127.0.0.1:2181
The guide walks through a first znode:
[zkshell: 9] create /zk_test my_data
Created /zk_test
For a replicated ensemble, each server gets the same config with the member list, plus a myid file in its data directory:
tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888
The guide notes that for several servers on one machine, each needs a distinct client port and distinct quorum and election ports (for example 2888:3888, 2889:3889, 2890:3890). Four-letter words other than srvr must be allowed explicitly, for example:
4lw.commands.whitelist=stat, ruok, conf, isro
After which the administrator's guide shows a health check like:
$ echo ruok | nc 127.0.0.1 5111
imok
The same guide recommends binding the AdminServer to localhost, or turning it off if unused:
admin.serverAddress=127.0.0.1
admin.enableServer=false
Hands-on: Apache ZooKeeper 3.9.7 on a Linux VM
Everything in this section was run on 21 November 2026 in a cloud VM: Ubuntu 24.04.5 LTS, Linux 6.18, x86_64, 2 vCPUs, about 7.8 GiB RAM, no swap, no Docker, and no internet access other than PyPI and raw GitHub files. Java was OpenJDK 21.0.13. The input was apache-zookeeper-3.9.7-bin.tar.gz from dlcdn.apache.org, downloaded on another machine and checked against the official .sha512 file. Clients were the bundled zkCli.sh and kazoo 2.11.0, a third-party Python client installed from PyPI. All servers ran on one machine over loopback, so the timings are only indicative and say nothing about cross-host latency or real disks.
The distribution
| Check | Result |
|---|---|
| Archive | 22,946,610 bytes, 1,725 entries; 53 MB unpacked (lib/ 21 MB, docs/ 33 MB, mostly Javadoc) |
| Server jars | zookeeper-3.9.7.jar (1,373,194 bytes), zookeeper-jute-3.9.7.jar, zookeeper-prometheus-metrics-3.9.7.jar |
| Bundled dependencies | Jetty 9.4.58, Netty 4.1.138, logback 1.3.15, slf4j-api 2.0.13, Jackson 2.22.3, JLine 3.30.14, snappy-java 1.1.10.5, Prometheus simpleclient 0.9.0; netty-tcnative native jars for five platforms add 6.1 MB |
| Version string | 3.9.7-0cc8919035e246f6a6e3ebb4ef1df15c4243bce7, built on 2026-10-22 14:12 UTC |
Default JVM flags from zkEnv.sh | -Xmx1000m for the server, -Xmx256m for the CLI; JMX enabled; -XX:OnOutOfMemoryError=kill |
conf/zoo_sample.cfg | tickTime=2000, initLimit=10, syncLimit=5, dataDir=/tmp/zookeeper (with a comment not to use /tmp), autopurge lines commented out |
Standalone start-up, memory and defaults
With the three-line config from the guide, zkServer.sh start returned in 1.06 s. Measured from launching start-foreground to the first successful srvr reply, four starts took 0.82, 0.94, 0.96 and 1.46 s. The log showed about 0.37 s from reading the config to binding port 2181. Resident memory five seconds after start was 99,696 to 108,800 KiB (about 100 MB); after the throughput runs below it had grown to 230,192 KiB with the default 1,000 MB heap ceiling.
| Observed default | Value |
|---|---|
| Session timeout bounds | Log: "minSessionTimeout 4000 ms maxSessionTimeout 40000 ms" |
| Client port binding | 0.0.0.0:2181 |
| AdminServer | "Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands" |
| Autopurge | snapRetainCount 3, purgeInterval 0, "Purge task is not scheduled." |
| Transaction log file | Preallocated to 67,108,880 bytes (64 MiB) as soon as the first write arrived |
| Snapshot compression | zookeeper.snapshot.compression.method = CHECKED |
Four-letter words versus the AdminServer
The four-letter-word whitelist behaved as documented: srvr answered, while ruok, stat and mntr on port 2181 each returned "… is not executed because it is not in the whitelist." zkServer.sh status still worked because it uses srvr. The AdminServer on port 8080, however, answered the same commands over HTTP with no authentication: /commands/ruok, /commands/stat, /commands/mntr, /commands/environment and /commands/system_properties all returned 200, the last two exposing the Java version and the OS user (root here). The index listed 28 commands. Only snapshot and restore refused, with HTTP 401 and "Auth info is missing for the command". In practice, the 4lw whitelist does not restrict access while the AdminServer is running with its defaults; use the admin.* settings shown above to lock it down. Setting admin.enableServer=false removed the listener completely (connection refused on 8080).
zkCli and znode behaviour
| Action | Result |
|---|---|
One zkCli.sh command per invocation | About 1.05 s each, almost all JVM start-up |
create -e /app/eph, then create -s /app/seq- twice | seq-0000000001 and seq-0000000002: numbering started at 1, because the parent's counter had already moved when the ephemeral child was created |
Ephemeral znode after a one-shot zkCli.sh command exits | Still listed by other clients for 31.4 s, until the log showed "Expiring session …, timeout of 30000ms exceeded" |
Ephemeral znode after kazoo stop() | Gone 2 ms later (session closed explicitly) |
create -t 5000 /app/ttl with default config | KeeperErrorCode = Unimplemented for /app/ttl, as the programmer's guide warns |
One-shot data watch, then two set calls | Fired once (CHANGED, 7.7 ms after the write), not again on the second write |
zkCli.sh help | 28 commands, including addWatch, version and whoami |
The 1 MB limit
The administrator's guide gives the default jute.maxbuffer as 1,048,575 bytes, "just under 1M", and says it is the "maximum size of the data that can be stored in a znode". In practice the limit applies to the whole request packet. With an 11-character path, a binary search found the largest value that could be created was 1,048,517 bytes; 1,048,518 bytes failed. The failure is not a clean error. The server logged java.io.IOException: Len error ("advertised length of 1048576 … greater than jute.maxbuffer=1048575") and dropped the connection, so the client saw ConnectionLoss and reconnected. Code that retries on connection loss will therefore retry an oversized write that can never succeed. A value just under the limit took 19 ms to create and 3.7 ms to read back.
Throughput with a small Python client
kazoo ran a single client: 2,000 synchronous creates, sets and gets of 100-byte values, then 20,000 pipelined asynchronous sets and gets. Two standalone runs and three ensemble runs (one per server) give an idea of the spread. With one Python process on a 2-vCPU machine, the client is likely the bottleneck for the asynchronous numbers.
| Target | Sync create | Sync set (p50) | Sync get (p50) | Async set | Async get |
|---|---|---|---|---|---|
| Standalone, run 1 | 563 ops/s | 1,184 ops/s (0.61 ms) | 3,775 ops/s (0.21 ms) | 4,373 ops/s | 6,040 ops/s |
| Standalone, run 2 | 1,128 ops/s | 906 ops/s (0.52 ms) | 5,445 ops/s (0.18 ms) | 5,908 ops/s | 4,774 ops/s |
| Ensemble, via follower 1 | 325 ops/s | 484 ops/s (1.24 ms) | 2,656 ops/s (0.25 ms) | 2,685 ops/s | 4,570 ops/s |
| Ensemble, via leader | 527 ops/s | 671 ops/s (0.98 ms) | 2,290 ops/s (0.21 ms) | 2,488 ops/s | 4,599 ops/s |
| Ensemble, via follower 3 | 545 ops/s | 623 ops/s (1.10 ms) | 1,422 ops/s (0.32 ms) | 3,108 ops/s | 4,503 ops/s |
The pattern matches the design: reads cost about the same anywhere, while writes in the three-node ensemble roughly halve throughput and double median latency compared with standalone, even over loopback, because every write needs a quorum acknowledgement and a log append on each server. Synchronous p99 latencies for writes ranged from 5.7 to 15.7 ms.
Three-node ensemble, leader failover and quorum loss
Three servers ran on one host with client ports 2181 to 2183, quorum and election ports 2888:3888, 2889:3889 and 2890:3890, AdminServer ports 8081 to 8083 and 4lw.commands.whitelist=*. Started about a second apart, server 2 became leader ("LEADING - LEADER ELECTION TOOK - 297 MS"); server 1, which started first and had to wait for peers, logged 1,938 ms. Each idle server used about 102 MB RSS. mntr on the leader reported zk_synced_followers 2 and zk_quorum_size 3.
For failover, a kazoo client with all three addresses wrote in a tight loop while the leader was killed with kill -9.
| Trial | New leader visible via srvr | Longest gap between successful writes | Client errors |
|---|---|---|---|
| 1 | 741 ms | not recorded | 1 ConnectionLoss |
| 2 | 431 ms | 960 ms | 1 ConnectionLoss |
| 3 | 467 ms | 980 ms | 1 ConnectionLoss |
| 4 | 446 ms | 650 ms | 1 ConnectionLoss |
Election itself was logged at 238 to 316 ms in the trials whose logs survived (zkServer.sh start overwrites the .out log on restart). The overview's figure of "less than 200ms to elect a new leader" comes from a 7-machine benchmark of release 3.2. From the application's side the write pause was 0.65 to 1 s, plus one connection-loss exception to handle.
Killing two of the three servers left the survivor answering srvr with "This ZooKeeper instance is not currently serving requests". Both a write and a read through it timed out after 15 s, with the client in the SUSPENDED state. Reads stop too unless read-only mode is enabled. After one server was restarted, the first write succeeded 3.61 s after the start command, JVM start-up included.
Snapshots, transaction logs and cleanup
A standalone server with snapCount=1000 received 500 creates and 10,000 sets. It produced 14 snapshots of 91,993 bytes and 14 transaction logs. Each log reported 67,108,880 bytes, so du --apparent-size showed 898 MB, while actual disk use was 3.1 MB because the preallocated files are sparse on this file system. Monitoring that counts apparent sizes will overstate usage. zkCleanup.sh -n 3 kept three snapshots and four logs. -n 2 was rejected, because the count "should be greater than or equal to 3". Autopurge was not timed, since its interval is set in whole hours. After the benchmarks, ensemble member 1 held a 379,631-byte snapshot and two 64 MiB logs (15 MB on disk), plus acceptedEpoch and currentEpoch files.
Enabling the Prometheus provider with two lines in zoo.cfg started a second Jetty listener on 0.0.0.0:7000 that served 764 lines and 180 metric families, including znode_count and JVM heap use.
Not tested
TLS, SASL and Kerberos; dynamic reconfiguration; observers; read-only mode; rolling upgrades; the C client; Apache Curator (Maven Central was unreachable); real multi-host networking and disk fsync latency; and the 3.10.0 candidate. No container images were tried because the VM has no Docker.
Use cases
- Leader election and locks for a pool of workers, using ephemeral sequential znodes as the recipes guide describes, so a crashed holder releases the lock when its session expires.
- Service discovery and group membership: each instance registers an ephemeral znode, and consumers watch the parent for changes. The programmer's guide mentions
getEphemerals()for this pattern. - Shared configuration and feature flags that many processes read often and few write, the "read-dominant" workload the overview says ZooKeeper suits best, at ratios "around 10:1".
- Coordination for existing systems that still use a ZooKeeper-compatible store, for example ClickHouse replication, where ClickHouse Keeper is the documented drop-in alternative.
- Development and integration tests using a standalone server, which starts in about a second and needs around 100 MB of RAM.
How Apache ZooKeeper compares
Descriptions come from each project's own documentation read on 21 November 2026; none of the alternatives were run here.
| Project | Consensus and language | Client interface | Positioning in its docs | Licence |
|---|---|---|---|---|
| Apache ZooKeeper 3.9.7 | Own atomic broadcast protocol (ZAB); Java | Own binary protocol; Java and C bindings, third-party clients | General coordination service: znodes, watches, ephemerals, recipes | Apache-2.0 |
| etcd | Raft; Go | gRPC API, etcdctl | "distributed reliable key-value store for the most critical data of a distributed system"; often used with Kubernetes; README claims "benchmarked 10,000 writes/sec" | Apache-2.0 |
| ClickHouse Keeper (part of ClickHouse) | Raft via NuRaft; C++ | ZooKeeper-compatible client protocol, so "any standard ZooKeeper client can be used" | Coordination for ClickHouse replication and distributed DDL; can run standalone or inside the server | Apache-2.0 (ClickHouse) |
| KRaft in Apache Kafka | Kafka controller quorum | Internal to Kafka | Kafka's metadata quorum; the docs say ZooKeeper-to-KRaft migration needs a bridge release, "The last bridge release is Kafka 3.9" | Apache-2.0 |
etcd is the most direct general-purpose alternative: a replicated key-value store with Raft consensus, a gRPC API and automatic TLS, per its README. Its README says it is frequently paired with Kubernetes. ZooKeeper's distinguishing features are its tree model, ephemeral and sequential znodes, watches and the recipes in its own docs. Porting between the two means mapping each primitive to the other's API, so check every one you rely on before switching.
ClickHouse Keeper is explicitly a ZooKeeper replacement for ClickHouse. Its docs say it gives "the same guarantees as ZooKeeper: linearizable writes and non-linearizable reads" by default, with an option for quorum reads. The same docs warn that snapshots, logs and the inter-server protocol are incompatible, so a mixed cluster "is impossible" and migration requires stopping ZooKeeper and converting data with clickhouse-keeper-converter.
Kafka is not an alternative for general coordination, but it matters for ZooKeeper's install base. Current Kafka documentation describes KRaft controllers and treats ZooKeeper only as a migration source, so new Kafka clusters do not need a ZooKeeper ensemble. For metrics on any of these, the ZooKeeper monitoring guide documents scraping with Prometheus and importing a Grafana dashboard.
Limits, trade-offs and when not to use it
- Not a data store. Values are capped at roughly 1 MB per request (1,048,517 bytes in the test), the whole tree lives in memory on every server, and the guide warns that large znodes cause "unwarranted latency spikes" and unstable quorums. Put payloads elsewhere and keep pointers in ZooKeeper.
- Writes do not scale with servers. Each write goes through the leader and a quorum; the test showed ensemble writes at about half of standalone throughput even over loopback. Adding voters increases fault tolerance, not write capacity. Observers help with read fan-out only.
- Reads can be stale. A follower serves reads from its local replica. Call
syncfirst when a read must reflect the latest committed write. - Quorum loss stops everything by default. With two of three servers down, reads and writes both stalled. Two-server ensembles are worse than one, as the getting-started guide explains; use three or five.
- Insecure defaults on the HTTP side. The AdminServer listens on
0.0.0.0:8080without TLS or authentication and exposes commands that the 4lw whitelist blocks on the client port. Port 8080 also clashes with many other services. The Prometheus provider adds another listener on all interfaces. - Operators must manage disk. Autopurge is off by default, transaction logs are preallocated in 64 MiB files, and the docs recommend a dedicated log device for production. Set
autopurge.purgeIntervalor schedulezkCleanup.sh. - Sessions shape behaviour. Ephemeral znodes from a client that exits without closing its session linger for the full session timeout (31 s observed with
zkCli.sh). Oversized writes surface asConnectionLoss, not as a size error. - JVM footprint. About 100 MB RSS idle per server and a 1,000 MB default heap ceiling. That is small for a JVM service but larger than single-binary alternatives.
- Documentation drift observed. The 3.9.7 administrator's guide lists Java "1.8 or greater (JDK 8 LTS, JDK 11 LTS, JDK 12 - Java 9 and 10 are not supported)", yet 3.9.7 ran on JDK 21 without problems. Its
PurgeTxnLogexample namesslf4j-api-1.7.30.jarandlogback-classic-1.2.10.jar, while the 3.9.7 tarball ships slf4j-api 2.0.13 and logback 1.3.15. The getting-started guide'shelplisting omitsaddWatch,versionandwhoami. The 3.9.6 release notes list ZOOKEEPER-5050, "Disable AdminServer and enhance documentation to highlight security considerations", but the 3.9.7 guide and the test both show the AdminServer enabled by default. The overview's performance figures date from release 3.2. - Upcoming Java baseline change. The 3.9.7 build targets Java 8 (
maven.compiler.release8). Thepom.xmlat therelease-3.10.0-0tag sets version 3.10.0 withmaven.compiler.release17. That tag looks like a release candidate: 3.10.0 was not on the download mirror on 21 November 2026, so plan for a Java 17 minimum once it ships, but do not treat it as released.
Release lines and documentation sources
The Apache download mirror listed three lines on 21 November 2026: 3.9.7, 3.8.8 and 3.7.2. The 3.9.7 release notes cover client thread shutdown fixes after authentication failures, stream and resource leak fixes in the transaction log code, iterative snapshot serialization, the new multi-read limits, and upgrades of Netty to 4.1.138 and Jackson to 2.22.3 for CVE fixes. The 3.9.6 notes add TLS and FIPS fixes, AdminServer TLS settings, removal of hard-coded TLS cipher lists, and a fix for Prometheus provider passwords being logged in clear text.
On master, the root pom.xml declares version 3.11.0-SNAPSHOT and replaces the zookeeper-docs module with a zookeeper-website module; the markdown paths used by the 3.9.7 docs return 404 there. Use the documentation from the release tag that matches the deployed version, which is what this page does. The project also points to its website, zookeeper.apache.org, and a Confluence wiki for contribution guidelines.
Project activity
Checked 21 November 2026: latest release 3.9.7 (tag release-3.9.7, commit dated 22 October 2026; the binary reports a build time of 22 October 2026), licence Apache-2.0, per the Apache download mirror, the release tag and the repository's licence file. A release-3.10.0-0 tag (commit dated 10 November 2026) exists but 3.10.0 was not yet on the mirror. The apache/zookeeper repository had about 12,800 stars (12,833 at the time of writing) and 7,330 forks. It is written in Java, was created on 27 March 2009, develops on master (last pushed 20 November 2026) and is not archived. Documentation and downloads are at zookeeper.apache.org.
Enjoying this project?
Discover more amazing open-source projects on TechLogHub. We curate the best developer tools and projects.
Repository:https://github.com/apache/zookeeper
GitHub - apache/zookeeper: Apache ZooKeeper: Distributed Coordination Service
Apache ZooKeeper is an Apache-2.0 coordination service for distributed systems. Version 3.9.7 tested on a Linux VM: znodes, watches, the 1 MB limit, a 3-node en...
github - apache/zookeeper
Related Projects
Uptime Kuma: Self-Hosted Uptime Monitoring Tool
Self-hosted uptime monitor with 90+ notification services, status pages and a WebSocket-driven web dashboard.
Spring Boot: Production-Ready Spring Applications
Opinionated Spring framework: starters, auto-configuration, embedded servers, Actuator and native images.
Caddy: Web Server With Automatic HTTPS
Go web server and reverse proxy with automatic HTTPS, a simple Caddyfile and a live JSON config API.
Maintain Apache ZooKeeper: Distributed Coordination Service? Add the badge to your README
Show that the project is featured on TechLogHub and send readers to this write-up.
[](https://techloghub.com/open-source/apache-zookeeper?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)