DuckLake: SQL-Based Open Lakehouse Format for DuckDB

What DuckLake is

DuckLake is an open lakehouse format built on SQL and Parquet. A DuckLake keeps its metadata, such as tables, schemas, snapshots and file statistics, in an ordinary catalog database and keeps the data itself in Parquet files on local disk or object storage. The duckdb/ducklake repository contains the DuckDB ducklake extension, which lets DuckDB read and write DuckLake directly and serves as the reference implementation of the format.

The DuckDB team explains the motivation in the DuckLake v1.0 announcement, which recaps its May 2025 manifesto: in their view it makes much more sense to store all of a lakehouse's metadata in a database than in scattered files in object storage. The announcement places DuckLake alongside setups such as Delta Lake with Unity Catalog and Iceberg with Lakekeeper, with the key difference that the catalog is simply any system that speaks SQL, supports primary keys and persists data in tables.

DuckLake v1.0 was announced in April 2026 as a production-ready release with a stable specification and guaranteed backward compatibility, and its reference implementation ships with DuckDB v1.5.2. It suits data engineers and analysts who want lakehouse features, such as ACID changes, time travel and schema evolution on Parquet, without running a separate catalog service.

How DuckLake works

The specification defines two building blocks. The catalog database must support transactions and primary key constraints as defined by SQL-92; DuckLake defines its metadata as SQL tables and the queries used to read and update them. The data storage holds Parquet files, which can live in object storage, block storage or file storage. DuckLake never changes existing files, neither by rewriting their content nor by appending to them, which the docs say greatly reduces the consistency requirements on file systems and simplifies caching.

Changes are recorded as snapshots. The introduction walks through an example: creating a table writes a Parquet data file, and an UPDATE is modelled as a delete followed by an insert, which produces a small Parquet delete file marking the old row plus a new data file with the new values. Each change creates a snapshot, and queries can read any earlier one. For workloads with many tiny writes, data inlining writes small inserts and deletes into tables inside the catalog database instead of creating new Parquet files; it is enabled by default with a row limit of 10.

The choice of catalog decides how many clients can work at once. The docs recommend a DuckDB file for local warehousing with a single client, SQLite for several local clients, and PostgreSQL 12 or newer for a multi-user lakehouse with remote clients. MySQL 8 or newer also works, but the docs list known issues with the DuckDB MySQL connector and do not recommend it. For storage, the extension works with any file system DuckDB supports: local folders, AWS S3 and compatible stores such as Cloudflare R2 and Hetzner Object Storage, Google Cloud Storage, Azure Blob Storage, and network or virtual file systems such as NFS, SMB, FUSE and Python fsspec.

Key features

• Plain SQL interface: after ATTACH, a DuckLake behaves like any other DuckDB database for creating schemas and tables and inserting, updating and deleting data.
• Time travel: query a table as of an earlier snapshot with AT (VERSION => n), and list snapshots with the snapshots() function.
• Schema evolution: ALTER TABLE adds columns without rewriting existing data.
• Change data feed: table_changes() returns the inserts, updates and deletes between snapshots.
• Encryption: with the ENCRYPTED flag, every Parquet file is encrypted with its own automatically generated key, stored in the catalog, so data can sit in an untrusted location.
• Data inlining: small changes go into the catalog database instead of tiny Parquet files, which the v1.0 post links to streaming data into the lake.
• Upserts: supported through MERGE INTO, since the format has no primary keys.
• Interoperability: according to the v1.0 announcement, releases since 2025 added registering existing Parquet files without a deep copy, Iceberg compatibility, and geometry and variant types.
• Maintenance tooling: the documentation covers checkpointing, expiring snapshots, merging adjacent files, rewriting data files and cleaning up old files.
• Growing client ecosystem: the announcement lists DuckLake clients for Apache DataFusion, Apache Spark, Trino and pandas dataframes, and a hosted DuckLake service from MotherDuck.

Getting started with DuckLake

Install a recent DuckDB release, then install the extension from SQL:

INSTALL ducklake;

The README's example keeps metadata in a DuckDB file called metadata.ducklake and writes data as Parquet into the file_path directory. Attaching creates the DuckLake if it does not exist yet:

ATTACH 'ducklake:metadata.ducklake' AS my_ducklake (DATA_PATH 'file_path/');
USE my_ducklake;
CREATE TABLE my_ducklake.my_table(id INTEGER, val VARCHAR);
INSERT INTO my_ducklake.my_table VALUES (1, 'Hello'), (2, 'World');
FROM my_ducklake.my_table;

Updates, time travel and schema changes are standard SQL:

UPDATE my_ducklake.my_table SET val='DuckLake' WHERE id=2;
FROM my_ducklake.my_table AT (VERSION => 2);
ALTER TABLE my_ducklake.my_table ADD COLUMN new_column VARCHAR;

For a shared, multi-user setup, the catalog guide points DuckLake at PostgreSQL. The ducklake_catalog database must already exist:

INSTALL postgres;
ATTACH 'ducklake:postgres:dbname=ducklake_catalog host=localhost' AS my_ducklake
    (DATA_PATH 'data_files/');

The DuckDB command-line client can also open a DuckLake directly:

duckdb ducklake:my_ducklake.ducklake

To build the extension from source, initialise the submodules and run make; the submodules are pinned to the DuckDB version CI builds against, and the bundled shell lands in ./build/release/duckdb:

git submodule init
git submodule update --recursive

Use cases

• Local data warehousing: a single DuckDB catalog file plus a folder of Parquet gives an analyst ACID tables, history and schema changes on a laptop.
• Multi-user lakehouse: several DuckDB instances share one lake by coordinating through a central PostgreSQL catalog, which the announcement describes as a "multiplayer" setup.
• Streaming small writes: data inlining absorbs frequent small inserts in the catalog instead of producing thousands of tiny files.
• Public read-only datasets: the v1.0 post notes that storage plus a public HTTPS endpoint is enough to serve a read-only lakehouse without authentication.
• Encrypted data on third-party storage: per-file Parquet encryption keeps data unreadable to the storage provider while the keys stay in your catalog.
• Migrating from DuckDB files: the docs include migration scripts from DuckDB databases to DuckLake.

How DuckLake compares

The natural comparisons are Apache Iceberg and Delta Lake, both named in the DuckLake announcement. Those formats keep much of their table metadata as files in object storage and typically pair with a catalog service. DuckLake's argument is that a SQL database already solves transactional metadata well, so it puts all metadata there and keeps only Parquet data in storage. The trade-off is that you operate, back up and secure that catalog database. Iceberg compatibility features suggest the DuckDB team expects DuckLake to coexist with Iceberg rather than replace it outright.

Compared with plain DuckDB database files, DuckLake gives up some features, listed below, in exchange for open Parquet storage, multi-client access through a shared catalog and snapshot history.

Things to know before adopting DuckLake

• License: the extension is MIT-licensed.
• Not every DuckDB feature is available: like other lakehouse formats, DuckLake does not support indexes, primary keys, foreign keys, or UNIQUE and CHECK constraints. The unsupported-features page also lists user-defined types, fixed-size arrays, ENUM, sequences and some other types.
• Catalog choice matters: a DuckDB catalog limits you to one client; use SQLite or PostgreSQL for concurrency, and avoid MySQL for now.
• Protect the catalog: with encryption enabled, the file keys are stored in the catalog, so catalog access is effectively data access.
• Version requirements: DuckLake v1.0 needs DuckDB v1.5.2 or newer, and the PostgreSQL catalog needs PostgreSQL 12 or newer.
• Paths: the docs ask for relative DATA_PATH and database paths, and the directory must already exist when creating a new DuckLake.
• Repository layout: the extension's README lives in docs/README.md; the full user guide and specification are on the DuckLake website.

Project activity

As of October 2026 the repository has about 3,300 stars. It was created in March 2025, is written in C++, is licensed under MIT and was receiving commits in October 2026. The v1.0 announcement says the ducklake extension ranks among DuckDB's top 10 core extensions by downloads and is used in production at dozens of companies. The source is at github.com/duckdb/ducklake, and the documentation and specification are at ducklake.select.