Neon

Neon: A Deep Dive into Serverless PostgreSQL

Neon is an open-source serverless Postgres database platform that reimagines how storage and compute work together in a modern cloud-native environment. At its core, Neon separates storage from compute and replaces the traditional PostgreSQL storage layer with a scalable storage engine that redistributes data across a cluster of nodes. This architectural choice unlocks elastic resources, on-demand scalability, and a flexible, pay-as-you-go model while preserving the familiarity and power of PostgreSQL.

In this guide, we explore Neon’s foundations, how to get started quickly, the architecture that powers Neon’s serverless model, how to run Neon locally for development, and some of the deeper development and testing workflows that drive this project. The aim is to provide a detailed, readable overview that both developers and curious readers can use to understand what Neon offers and how to work with it effectively.

Quick Start: A Path to a Serverless PostgreSQL Instance

Neon invites you to try a Free Tier to spin up a serverless PostgreSQL instance with minimal friction. The Free Tier provides a serverless Postgres environment you can connect to with your favorite PostgreSQL client, such as psql or DBeaver, or interact through an online SQL Editor. The quick-start path emphasizes simplicity and speed, so you can experiment with real data, run queries, and observe how Neon handles workloads without the overhead of provisioning and managing servers.

If you prefer connecting from applications, Neon offers straightforward connection instructions that guide you to connect from any environment. The emphasis is on seamless integration, ensuring developers can drop Neon into their stacks with familiar PostgreSQL tooling. For those who want to customize their setup further or run tests locally, Neon also provides a pathway to build and operate the project on a local workstation, which we cover in detail later in this guide.

Architecture Overview: Compute, Storage, and Redundancy

A Neon installation presents a distributed ecosystem built around two primary anchors: compute nodes and the Neon storage engine. The compute layer is composed of stateless PostgreSQL nodes. Each compute node is designed to perform standard PostgreSQL operations, while the heavy lifting and data management responsibilities sit with the Neon storage engine, which orchestrates data movement, persistence, and retrieval across the cluster.

The Neon storage engine is comprised of two critical components:

• Pageserver: This is the scalable storage backend that serves the compute nodes. It is responsible for the actual storage and retrieval of data blocks, acting as the primary interface between compute and persistent data.

• Safekeepers: The safekeepers provide a redundant Write-Ahead Logging (WAL) service. They receive WAL records from compute nodes and store them durably until the pageserver has processed and uploaded them to cloud storage. This layer adds resilience and durability to the system by guarding WAL against node failures and ensuring recoverability.

These components work together to deliver serverless Postgres with elastic compute scaling while maintaining data integrity and durability through a robust storage backbone. The separation of concerns—stateless compute nodes and a persistent, distributed storage layer—enables Neon to scale out by adding more compute resources as needed, while the storage engine handles long-term durability and consistency across the cluster.

Running a Local Development Environment: A Developer’s Path

For developers who want to experiment locally, Neon can be run on a workstation for small-scale experiments and to test code changes. The local development workflow is designed to be approachable yet feature-rich, offering a realistic glimpse into how Neon behaves in production while remaining accessible for development tasks.

Installing dependencies on Linux

A successful local build begins with installing the right dependencies. On Debian-based distributions like Ubuntu, a curated set of packages is typically sufficient to build the code. The packages cover essential tooling, building, and PostgreSQL client interfaces. A representative list includes compilers, development libraries, cryptography libraries, protobuf tooling, Python tooling, and other utilities that Neon’s build system relies on. The aim is to provide a complete environment so that the Neon sources can be compiled and linked without missing pieces.

On Fedora, a different package set is used, tailored to Fedora’s packaging conventions and the specific libraries it ships. Arch-based systems have their own minimal yet compatible package requirements. The common theme across all distributions is ensuring you have a modern toolchain, Rust tooling, protobuf compiler, OpenSSL, and the PostgreSQL client libraries.

One critical note is that Neon’s build process requires a protobuf compiler version of 3.15 or newer. If your distribution provides an older version, you’ll need to install a newer protobuf release from its official sources. This requirement ensures compatibility with the project’s protocol buffer definitions and code generation steps.

Installing Rust

Rust is central to Neon’s build pipeline. The recommended approach is to install Rust using the official Rust toolchain installer. After setting up Rust, you’re positioned to compile Neon alongside a patched Postgres distribution that Neon ships with or works against. The exact toolchain pinned in the repository is captured in a toolchain file, which helps ensure CI consistency. If you’re an advanced user who wants to experiment with a different Rust toolchain, you can use rustup’s directory overrides to enforce a specific version for the project directory.

Building Neon on Linux

The typical Linux build flow is to clone Neon’s repository with all submodules, then initiate a development build. The default is often a debug build, which prioritizes fast iteration and easier debugging, even though it can be slower than a release build. When you’re ready to produce a release, the build can be invoked with a release flag and parallel build jobs. It’s common to avoid spaces in the path to Neon sources to prevent path-related build issues. Once the code is compiled, you’ll have the Neon binaries and the patched version of PostgreSQL ready for a local run.

Building Neon on macOS

macOS users follow a similar pattern, with macOS-specific steps for dependency installation. Xcode is required for the compilation toolchain, and additional libraries such as protobuf, OpenSSL, and ICU are installed through a package manager like Homebrew. Similar to Linux, the build can be configured for a debug or release output, with parallelization to speed up compilation. In addition to Neon, you’ll need a PostgreSQL client to interact with the local database, which can be obtained through system-specific package managers while leaving the client accessible via the system’s PATH.

Rustc Version and Tooling

The project adheres to a specific Rust toolchain. The rust-toolchain.toml file guides rustup in selecting the correct compiler version for CI testing and local builds. If you prefer a different setup, rustup’s override command can constrain the toolchain to the project’s directory. While newer Rust compilers may generally work, older versions could lack features used by some crates or internal code paths, so it’s best to align with the pinned version for compatibility.

Running Neon Database Locally: A Step-by-Step Preview

Starting the Neon components locally is a multi-step process designed to mirror the production workflow while keeping the steps approachable for development. The general flow includes initializing a local repository structure, launching the storage and control plane components, creating a default tenant, and then spinning up a PostgreSQL compute endpoint that behaves like a standard PostgreSQL instance.

1) Initialize and start the core components

• Create a local repository to hold binaries and data.
• Start the storage plane together with the broker that coordinates interactions between components.
• Launch the pageserver, which serves as the storage layer for compute nodes.
• Bring up safekeepers that act as a redundant WAL service to ensure durability.

2) Create a default tenant and timeline

• Create an initial tenant and designate it as the default for subsequent Neon local invocations. The tenant acts as a logical container for data and timelines represent different branches or versions of the data.

3) Create and start a PostgreSQL compute endpoint

• Define a compute endpoint that matches the PostgreSQL version you want to test (for example, PostgreSQL v14).
• Start the endpoint, which runs PostgreSQL locally, bound to a local address and port.
• Connect to the endpoint using a PostgreSQL client and confirm that basic DDL and DML work as expected.

4) Explore branches and branching workflows

• Neon supports creating branches (timelines) to experiment with changes without affecting the main data set.
• You can create a branch named migration_check, which spawns a new timeline and a corresponding PostgreSQL endpoint.
• The new branch inherits data from the main timeline, but changes in the branch do not affect the primary instance until you deliberately merge or propagate changes.
• This branching model provides isolation for experiments, tests, and migrations while preserving the integrity of the main data.

5) Clean up when finished

• When you’re ready to stop the local setup, you can terminate all running components with a centralized stop command.
• If you encounter build or setup failures, stopping all processes and removing the .neon directory can help reset the environment and let you reattempt the setup from a clean slate.

Accessing and Interacting with Neon

Once Neon compute endpoints are running, you can connect using standard PostgreSQL tools. For instance, you might use a psql client to connect to a local endpoint and execute normal SQL statements such as creating tables, inserting data, and querying results. The Neon approach makes it feel familiar to Postgres users, while under the covers the data is managed by Neon’s distributed storage system.

Branching and Data Isolation in Practice

Among Neon’s powerful capabilities is the ability to create and operate on separate branches. The branching system is akin to timelines. Each branch has its own timeline, and Neon will show a hierarchical view of the main branch and its descendants. When you create a branch, you designate a branch name and receive a new timeline that inherits from the parent. You can then create a dedicated endpoint on that branch to run PostgreSQL independently of the main branch. This isolation is particularly valuable for testing migrations, validating new features, or performing experiments that should not affect production-like data.

Tests and Flamegraphs: Quality and Observability

Neon emphasizes robust testing and performance analysis. The project encourages Rust unit tests and integration tests, with a modern testing workflow that uses tools like cargo-nextest for more reliable and scalable test runs. Integration tests can be run by cloning the Neon repository, setting up dependencies, and invoking test scripts that exercise both the serverless storage layer and the Postgres compute nodes in a coordinated fashion.

For performance analysis, Neon supports generating flamegraphs to visualize where time is spent within the software stack. This can involve using flamegraph-rs or the traditional FlameGraph tooling. When using certain linkers such as lld or mold, there are linker arguments to consider to avoid segmentation and resegment issues during flamegraph generation. The project provides guidance on these details to help developers obtain meaningful performance visualizations.

Cleanup, Maintenance, and Documentation

Maintaining a local Neon environment includes keeping the working tree clean of build artifacts. Commands exist to remove build and configuration artifacts, with an emphasis on understanding that removing the Neon data directory (.neon) will erase all stored data. This is a critical reminder for developers who experiment locally: ensure you’ve backed up any important data before a cleanup.

Neon’s documentation hub holds a broad collection of resources, including a top-level overview of available Markdown documentation and a practical guide to the repository structure. The documentation also points readers to a sourcetree overview, recommended Rust docs-building commands, and a host of external resources that situate Neon within the broader PostgreSQL and database-architecture conversation. The project maintains cross-references to relevant glossary terms, PostgreSQL glossaries, and external learning resources, ensuring readers can connect Neon’s design choices to well-known database concepts.

Postgres-Specific Terms and Glossaries

Because Neon maintains a close relationship with PostgreSQL internals, several domain-specific terms appear frequently. Neon users benefit from consulting dedicated glossaries to align terminology with PostgreSQL’s established usage. This helps bridge the conceptual gap between Neon’s storage engine innovations and classic PostgreSQL concepts like databases, schemas, WAL, and timelines. The glossaries also help clarify measurement units (such as MB vs MiB and other terminology), which is a useful guardrail for developers who work with performance metrics and data sizing.

Join the Development: Contributing and Collaboration

For developers who wish to contribute, Neon provides a path to participation through its contribution guidelines. The project emphasizes learning through exploring the codebase, reading the documentation, and following the established code style and practices. It’s recommended to study the repository’s directory layout and Rust-based internals to gain familiarity with how components interact. The development path is complemented by blog posts and talks that discuss Neon’s architecture decisions and the evolution of its serverless Postgres design.

A Quick Recap of What Neon Brings to PostgreSQL

• Serverless Postgres with elastic compute scaling and a robust distributed storage engine.
• A clear separation of storage and compute, enabling flexible resource provisioning.
• Pageserver as the scalable storage backend and safekeepers as a redundant WAL service for durability.
• Local development pathways that simulate production environments with a local repository, tenants, timelines, and endpoints.
• Branching and timelines for experiments and migrations without impacting main data.
• Strong emphasis on testing, from unit tests to integration tests, with modern tooling like cargo-nextest.
• Observability tools and guidance for flamegraphs to diagnose performance characteristics.
• Comprehensive documentation and glossary resources to aid in understanding PostgreSQL terms in Neon’s context.

Images and Visual Context

The Neon brand image at the top of this guide provides a visual anchor for understanding Neon’s identity and purpose. While Neon’s architecture is described in words and diagrams, the visual image helps readers connect with the brand as a symbol of the serverless Postgres approach. If you’re building your own blog post or documentation page, placing the Neon logo image near the introductory paragraphs helps set expectations and invites readers into the detailed sections that follow.

Final Thoughts: Embracing Serverless PostgreSQL with Neon

Neon represents a thoughtful convergence of modern cloud-native design and the enduring strengths of PostgreSQL. By decoupling storage from compute and distributing data across a cluster with a resilient WAL service, Neon provides a scalable, durable, and flexible platform for database workloads. It supports developers who want fast iteration, data isolation for experiments, and a PostgreSQL-compatible interface that reduces friction for teams already familiar with PostgreSQL. The local development workflow, the branching model, and the tested integration and unit testing approach all contribute to a robust ecosystem for building and testing applications that rely on a serverless Postgres experience.

If you’re curious about where Neon fits in the broader landscape of cloud databases, think of Neon as a bridge between the simplicity and familiarity of PostgreSQL and the distributed, elastic realities of modern cloud infrastructures. It invites experimentation, learning, and practical deployment patterns without sacrificing the reliability and feature set that PostgreSQL users expect. As you explore Neon, you’ll encounter a coherent model for scaling compute independently from durable storage, a workflow that supports rapid experimentation through branches, and a development story that emphasizes testing and performance analysis as central to delivering a robust serverless Postgres experience.