DuckDB: In-Process Analytical SQL Database

What DuckDB is

DuckDB is a high-performance analytical database system that runs in-process inside your application. There is no separate server to manage and no external dependencies: the README describes it as designed to be fast, reliable, portable and easy to use, and the CLI documentation describes the command line client as a single, dependency-free executable precompiled for Windows, Mac and Linux. DuckDB is written in C++, released under the MIT licence, and ships clients for Python, R, Java, Rust, Go, Wasm and more, with deep integrations with packages such as pandas.

The audience is anyone who needs analytical SQL (aggregations, joins and window functions over large tables) close to where the data lives: data engineers working with Parquet and CSV files, analysts in Python notebooks, application developers who want an embedded OLAP engine, and increasingly coding agents that drive the CLI on a user's behalf. The README says its SQL dialect goes far beyond basic SQL, with arbitrary and nested correlated subqueries, window functions, collations and complex types such as arrays, structs and maps, plus "friendly SQL" extensions that make common queries shorter to write.

How it works

By default DuckDB is embedded: the engine lives inside the host process, whether that is the CLI, a Python interpreter or a Java application. The concurrency documentation explains the trade-off. In in-process mode, one process can both read and write a database file, or many processes can open it read-only. Inside that single writer process DuckDB supports multiple writer threads using MVCC combined with optimistic concurrency control. Appends never conflict, even on the same table, and when two threads update or delete the same row at the same time the second one fails with a transaction conflict error. The docs say this model was chosen so data, function pointers and the catalog can be cached in RAM for faster analytical queries instead of going back and forth to disk.

For writes from multiple processes there are now two routes. The first is the Quack remote protocol, first released in DuckDB 1.5.3 and shipped in polished form in 2.0, which turns a DuckDB instance into a server that other DuckDB clients attach to over HTTP. Quack encodes requests and responses with DuckDB's own serialization primitives (the same code path used by its write-ahead log), so nested types, decimals and intervals survive the wire losslessly. After the connection handshake a query needs a single request-response pair, large results stream back in chunks, and the default port is 9494. The second route is DuckLake, a lakehouse format that stores table metadata in a SQL database such as PostgreSQL, so several DuckDB instances can coordinate concurrent read-writes through a central catalog. DuckLake 1.1, which the docs describe as intended for production use, was released alongside DuckDB 2.0.

Functionality is modular. Core extensions are bundled with DuckDB, while community and third-party extensions can be installed and loaded at runtime to add file formats, database connectors, functions and types.

Key features

• Query files directly: CSV, Parquet and JSON can be queried by naming the file in the FROM clause, from local disk or from remote endpoints such as S3, HTTP(S), Azure and Google Cloud Storage. Excel files are supported through an extension.
• Lakehouse formats: DuckLake, Apache Iceberg and Delta Lake are supported. The 2.0 release notes say the Delta extension can now create Delta tables with CREATE TABLE and supports time travel, and the Iceberg extension gains REST catalog ATTACH options.
• Query other databases in place: DuckDB speaks the native protocols of PostgreSQL, MySQL and SQLite, so you can read, join and move data across engines without a separate ETL step.
• Client-server mode with CONNECT: 2.0 adds a CONNECT statement that attaches and switches to a remote database in one command. For PostgreSQL and MySQL a new remote pushdown optimizer ships queries to the remote server instead of pulling tables over the wire.
• Asynchronous I/O: 2.0 issues reads for Parquet, CSV and its native storage format in the background with multiple requests in flight. The release post says this gives far more parallelism for remote reads and dramatically faster queries on object storage such as s3://.
• First-class VARIANT type: semi-structured data is stored in a binary columnar form rather than as text, and DuckDB "shreds" it when it detects common structure so it compresses well, with a family of variant_* functions.
• Agent mode in the CLI: when the 2.0 CLI detects it is being run by a coding agent, it prints compact Markdown tables, says clearly when a result was truncated, stops runaway queries early and reports errors as JSON. The team says agent mode cut the DuckDB output a model had to read by 59% in its TPC-H experiments. The -agent and -no-agent flags force it on or off.
• New PEG-based parser: the default in 2.0, replacing the PostgreSQL-derived YACC/Bison parser. Extensions can hook into the grammar itself, and error messages now come with precise source locations.

Getting started

The installation page offers a one-line installer for the CLI on macOS and Linux:

curl https://install.duckdb.org | bash

The Python client installs from pip. The Python docs say DuckDB requires Python 3.11 or newer and that free-threaded CPython builds are not supported:

pip install duckdb

A first query from Python runs against an in-memory database stored globally inside the module:

import duckdb

duckdb.sql("SELECT 42").show()

Files can be queried by referencing them in the FROM clause:

SELECT * FROM 'myfile.csv';
SELECT * FROM 'myfile.parquet';

To try client-server mode, the 2.0 CLI can start a Quack server in one terminal and connect to it from another:

duckdb -serve
duckdb -connect

From inside an existing session, the equivalent server call is CALL quack_serve('quack:localhost');.

Use cases

• Ad hoc analysis of data files: point the CLI or the Python client at Parquet or CSV files in a local folder or an S3 bucket and run SQL without loading anything into a server first.
• Embedded analytics in applications: ship an analytical engine inside a Python, Java, Go or Rust service with no extra infrastructure to operate.
• Cross-database joins and migrations: attach PostgreSQL, MySQL or SQLite and join their tables with local files, or copy data between engines.
• Lightweight lakehouse: use DuckLake with a PostgreSQL catalog, or read Iceberg and Delta tables, to get snapshots and time travel without a separate catalog server.
• Shared analytical database: run a Quack server so several DuckDB clients can query and write to one database, behind a TLS-terminating reverse proxy.

How it compares

The nearest point of reference is SQLite. The DuckDB CLI is based on the SQLite command line shell, and both are embedded databases that store data in a single file. The difference is workload: DuckDB is built for analytical queries that scan many rows, while SQLite is a general-purpose transactional store. Because DuckDB can attach a SQLite file and query it in place, the two are often used side by side rather than as substitutes.

Compared with client-server databases such as PostgreSQL and MySQL, DuckDB historically ran only in-process. Quack in 2.0 narrows that gap, but the docs still recommend keeping the server on localhost behind a reverse proxy rather than exposing it directly.

Things to know before adopting

• Licence: MIT, permissive, with no network-use clauses.
• 2.0 storage is not readable by older versions: DuckDB versions before 2.0 cannot read database files created by 2.0, although 2.0 still reads and writes existing files and those remain readable by older versions. The team recommends testing 2.0.0 on a shadow system and, for mission-critical workloads, possibly waiting for the first patch release planned for November.
• Support windows: 2.0 is a long-term support release supported until at least October 2027. The previous LTS, 1.4, is supported until November 17, 2026, and support for 1.5 was extended to March 2027.
• SQL changes: the old single-arrow lambda syntax (x -> x + 1) is disabled by default in favour of lambda x: x + 1, and both the old syntax and its compatibility flag will be removed in 2.1. The new parser has a few deliberate differences listed in the migration guide, and SET enable_legacy_parser = true; switches back to the old one for now.
• Single-writer model: without Quack or DuckLake, only one process can write to a database file at a time.
• Quack exposure: the server binds only to a local hostname by default. The allow_other_hostname option exposes it directly, and the docs say that is only meant for testing.
• Platform changes: macOS x86_64 is being phased out. Community extensions are no longer built for it, and DuckDB 2.1 will stop supporting it. Building from source now requires a C++17 compiler, and the amalgamation build is deprecated.

Project activity

As of October 2026 the duckdb/duckdb repository has about 42,500 stars. It was created in June 2018, is written in C++ and is licensed under MIT. DuckDB 2.0.0 LTS, codenamed "Cinnamon Teal", was released on October 21, 2026, and the team says it was built from close to 20,000 commits since DuckDB 1.5. For development the README lists CMake, Python 3 and a C++17 compiler, with make, make debug and make unit as the main targets. Source is at github.com/duckdb/duckdb, documentation at duckdb.org/docs, and the full 2.0 release notes are on the GitHub release page.

[![Featured on TechLogHub](https://techloghub.com/badge-neutral.svg)](https://techloghub.com/open-source/duckdb?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)