Fjall: A Log-Structured, Embeddable Rust Key-Value Storage Engine

Fjall: A Nordic-Mountain KV Store for Rust

In a world of data-driven applications, Fjall stands out as a practical, embeddable storage engine designed to be fast, safe, and flexible. Born from the Nordic spirit of resilience and efficiency, Fjall is a log-structured, embeddable key-value storage engine written in Rust. It aims to give developers a robust foundation for building high-performance applications without forcing you into a database server or a rigid schema.

Introduction and Philosophy

Fjall is designed around a few core ideas:

A thread-safe, BTreeMap-like API that feels familiar to Rust developers.
100% safe and stable Rust, with a focus on correctness and zero-cost abstractions.
An LSM-tree-based storage layer that provides high write throughput and good read performance, akin to RocksDB, but in a more embeddable and Rust-friendly package.
Rich querying capabilities via range and prefix searches, with support for forward and reverse iteration.
Support for multiple keyspaces (also known as column families) with cross-keyspace atomic semantics, enabling clean separation of concerns within a single database instance.
Built-in data management features like compression (default = LZ4) and optional serializable transactions for stronger consistency guarantees.
An option for key-value separation to optimize storage for large blob use cases and flexible data layouts.
The ability to customize compaction through user-defined filters to run custom logic during compactions.
Automatic background maintenance to keep the storage structures healthy without manual intervention.

What Fjall is Not

As exciting as Fjall sounds, it’s important to set expectations correctly. Fjall is not:

A standalone database server. It is an embeddable library, designed to be integrated into your application.
A relational or wide-column database. Fjall focuses on a key-value paradigm and does not come with a built-in query language or columnar abstractions.

Acknowledgments and Sponsors

Fjall benefits from a collaborative ecosystem. The project welcomes sponsors and community contributions, and you can see community logos and links in the header:

Orbitinghail, sponsor and partner, with a link to sqlsync.dev.

Basic usage and getting started

To begin using Fjall, you’ll typically create one or more keyspaces within a single database instance. Each keyspace is its own physical LSM-tree, which means data is isolated per keyspace, giving you clean separation of concerns and predictable performance characteristics.

Key steps (high level):

Create a database instance with a builder, pointing to a path on disk.
Create or open one or more keyspaces within that database, using KeyspaceCreateOptions to tailor behavior per keyspace.
Insert, retrieve, and delete items with the familiar insert, get, and remove operations.
Use prefix and range queries to scan over related keys efficiently.
Persist data to disk as needed to control durability (details below).
When done, drop or persist the database to ensure data durability.

A practical look at basic usage (snippets)

Basic usage example (Rust):

cargo add fjall

use fjall::{Database, KeyspaceCreateOptions, PersistMode};

fn main() -> fjall::Result<()> {
    // A database may contain multiple keyspaces
    // You should probably only use a single database for your application
    let db = Database::builder(path).open()?;

    // TxDatabase::builder for transactional semantics
    // Each keyspace is its own physical LSM-tree, and thus isolated from other keyspaces
    let items = db.keyspace("my_items", KeyspaceCreateOptions::default)?;

    // Write some data items
    items.insert("a", "hello")?;

    // And retrieve it
    let bytes = items.get("a")?;

    // Or remove it again
    items.remove("a")?;

    // Search by prefix
    for kv in items.prefix("prefix") {
        // ...
    }

    // Search by range
    for kv in items.range("a" ..= "z") {
        // ...
    }

    // Iterators implement DoubleEndedIterator, so you can search backwards, too!
    for kv in items.prefix("prefix").rev() {
        // ...
    }

    // Sync the journal to disk to make sure data is definitely durable
    // When the database is dropped, it will try to persist with `PersistMode::SyncAll` automatically
    db.persist(PersistMode::SyncAll)
}

Tip:

Like any typical key-value store, keys are stored in lexicographic order. If you are storing integer keys (e.g., timeseries data), you should use the big endian form to have predictable ordering.

Durability and persistence

Durability is a central concern for applications that rely on Fjall to protect data against failures. Fjall is designed to be flexible about durability to accommodate a wide range of workloads. After you write data—whether through insert, remove, or a committed write batch or transaction—you can choose how aggressively you persist to disk.

Persist to OS buffers by default: By default, operations flush to OS buffers but are not immediately written to disk. This mirrors RocksDB’s default durability guarantees: fast writes with a chance to be flushed to stable storage in a controlled manner.
Explicit persistence: If you need stronger guarantees, you can call Database::persist with a PersistMode parameter to flush in a specified manner.
Synchronous journal persistence on drop: When the database is dropped, Fjall will attempt to persist the journal to disk synchronously. This provides a safety net to reduce data loss on unexpected shutdowns.

Multithreading, Async, and Multiprocessing

Fjall is designed with practical concurrency in mind. A single database instance is internally synchronized for multi-threaded access, and you can clone Database and Keyspace handles as needed without requiring your own mutexes.

Safe from cross-thread data races: You can freely move handles between threads within your application.
Async examples: For asynchronous usage, there are examples in the repository (notably the tokio example) to illustrate how to integrate Fjall into an async runtime.
Process isolation caution: A single database may not be loaded in parallel from separate processes. Fjall is optimized for multi-threaded access within a single process, and cross-process parallel loading would require a different architectural approach or a separate process boundary.

Memory usage and tuning

Fjall’s memory footprint is shaped by how data, indexes, and blocks are cached. The system uses block-based memory management, and returned values may retain their backing buffers if you hold onto a Slice.

Block cache capacity: This controls how much of the dataset can be kept in memory for fast access. A larger block cache means more of your active dataset can be served from memory, but it also consumes more RAM.
Copying long-lived values: If you know you will keep a value around for a long time, consider copying it into a new buffer (Vec, Box<[u8]>, Arc<[u8]>, or a new Slice). This helps avoid keeping large blocks pinned in memory unnecessarily.
Recommended memory strategy: Configure the block cache to be roughly 20-25% of available memory, or more if your dataset fits entirely in memory. This is a practical starting point for balancing performance with resource usage.
Per-keyspace memtable sizing: Each keyspace has its own memtable (write buffer). The size of these memtables determines how much in-flight data you can accumulate before it is flushed back into the index structures.

Error handling

Fjall exposes a dedicated error enum that covers the common failure modes you’ll encounter while using the library. In practice, most errors are best treated as transient or recoverable by retrying or restarting the application, and may be used primarily for debugging and tracing.

The recommended approach is to allow the application to fail fast and restart in the face of unexpected errors, which can be a pragmatic strategy for durable systems.
This philosophy aligns with many robust storage strategies that favor simple recovery semantics over complex, granular error handling at every call site.

Transactional modes: serializable options

Fjall supports transactional semantics to meet the needs of workloads requiring stronger consistency. The design includes two transactional models:

Single Writer: Opens a transactional database that supports a single writer at a time. This model is trivially serializable because it serializes all write transactions, removing the risk of write-write conflicts.
Optimistic: Opens a transactional database that supports multi-writer, serializable transactions with optimistic concurrency control. Conflicts may occur, in which case a transaction must be retried.

These options let you balance performance and consistency according to your application’s needs. If your workload is dominated by writes from a single thread, Single Writer is straightforward and predictable. If you have multiple writers and require full serializability, Optimistic mode provides stronger guarantees at the cost of potential retries.

Feature flags and build-time choices

Fjall provides a few important feature flags that you can enable or disable at compile time:

lz4: Enables LZ4 compression (powered by lz4_flex). This is enabled by default and helps reduce storage usage while maintaining decent throughput.
bytes_1: Uses the tokio-bytes 1.x type as the underlying Slice. If disabled, Fjall will use the alternative byteview-based Slice type. This flag is disabled by default, allowing you to opt into the variant that best matches your ecosystem.

Stable disk format and future-proofing

Fjall’s disk format is designed to be stable across minor versions, with backward-compatible upgrades where feasible. The project emphasizes that future breaking changes will result in a major version bump and a migration path. For deeper insights into the underlying LSM-tree implementation, you can explore related crates such as lsm-tree on crates.io.

Expectation management: If you rely on Fjall in production, be aware that significant format changes will require a major version update and a migration strategy.
Practical stability: The emphasis on a stable surface for everyday use makes Fjall a reliable choice for embedded storage in Rust applications.

Examples and practical usage

The Fjall repository includes a collection of practical examples that demonstrate how to integrate Fjall into real-world projects. These examples show how to set up the database, create keyspaces, perform inserts and reads, and execute more advanced operations such as prefix and range queries, all while respecting durability and concurrency considerations.

Contributing and community

Fjall invites collaboration from developers who want to contribute code, benchmarks, and new ideas. If you’re curious or enthusiastic about performance-minded Rust projects, consider engaging through:

Asking questions or starting discussions on the project’s GitHub discussions channel.
Joining the community Discord for quick questions and collaboration.
Contributing by opening or reviewing PRs, sharing benchmarks, or posting show-and-tell demonstrations.
Browsing open issues to find tasks that you might help with, or opening new issues for bugs or feature ideas.

Contributing channels and contact points include:

Discord invite: https://discord.com/invite/HvYGp4NFFk
Discussions for questions, show-and-tell, and more
Open PRs and issue trackers on the Fjall GitHub repository

License and open-source philosophy

All Fjall source code is licensed under MIT OR Apache-2.0. Contributions are accepted under the same license terms, ensuring openness and broad usability across different projects and ecosystems.

A permissive licensing posture helps teams adopt Fjall without licensing friction.
The MIT/Apache-2.0 dual license is a common and flexible choice for Rust projects and libraries.

Practical recommendations for developers

Start with a single database instance and a small number of keyspaces to learn the API and performance characteristics.
Carefully configure block cache and memtable sizes based on available RAM and your workload characteristics.
Use range and prefix queries to efficiently scan subsets of keys, rather than scanning entire datasets.
When durability is critical, use the persist mechanism to force synchrony with disk storage, especially during important checkpoints or after critical writes.
If you’re building an application with multiple threads that perform writes, consider the transactional model that best suits your concurrency patterns—Single Writer for serialized writes or Optimistic for multi-writer workloads.
Keep an eye on compression gains with LZ4 and balance them against CPU usage in your deployment environment.

Images and visual cues

The header image /kawaii.png introduces Fjall with a visually appealing motif that evokes the Nordic mountain heritage of the project.
The project’s badges at the top of the page provide a quick glance at CI status, documentation, crate version availability, MSRV, dependency health, and community channels.
The Orbitinghail logo serves as a visual anchor for the project’s sponsorship and community partnerships.

Examples of how Fjall fits into real-world projects

Lightweight embedded services: Fjall shines as the local, embedded storage layer for mobile or desktop applications that require fast, local data access with predictable latency.
Edge computing and IoT: Fjall’s low-level control over durability and concurrency makes it suitable for edge devices that need robust, local data persistence without a centralized server.
Real-time analytics: The MVCC semantics and range/prefix queries enable efficient time-series-like workloads, where ordering and fast scans by key prefixes can accelerate dashboards and alerts.

Durability and recovery in practice

Default durability is OS-buffered writes, which are fast but do not necessarily guarantee immediate on-disk persistence.
When you are ready to ensure that data is durable across unexpected shutdowns, you can invoke explicit persistence with a chosen PersistMode.
On shutdown, Fjall’s approach to synchronously persisting the journal minimizes risk of data loss by ensuring the journal is reflected on disk before the application fully exits.

Stable development expectations

Fjall emphasizes a conservative approach to API stability and feature evolution. Changes that could break existing users are handled with a major version bump and migration guidance.
The project encourages contributions that improve reliability, benchmarking, and real-world usage patterns, with documentation and examples to help new users get started quickly.

Sponsors and community engagement (revisited)

The project maintains an active sponsorship channel to support ongoing development and maintenance.
Community channels include a Discord server for real-time collaboration and support, and a GitHub ecosystem for code contributions, issues, and discussions.

Examples and further reading

For developers who want practical guidance, consult the repository’s examples directory, which contains practical patterns for building with Fjall.
The official documentation on docs.rs provides API references and usage patterns to deepen your understanding of the available features and semantics.

Conclusion: Fjall as a reliable, embeddable KV store

Fjall is more than just a storage engine; it is a thoughtfully crafted Rust library that blends the strengths of LSM-tree storage with a safe, ergonomic Rust API. Its key features—range and prefix searches, multiple keyspaces with cross-keyspace semantics, optional serializable transactions, and built-in compression—offer a versatile foundation for a wide array of applications. It provides a pragmatic balance between speed, safety, and configurability, making it a compelling choice for developers who want a robust local storage solution without resorting to a full-fledged database server.

If you’re evaluating a new storage layer for your Rust project, Fjall’s design goals and feature set make it worth a close look. The project’s emphasis on performance, safety, and practical durability controls aligns well with modern software development demands, and its embeddable nature makes it a good fit for services, microservices, and edge-oriented deployments where local data access matters most. Explore Fjall, try the examples, and consider how a multikeyspace, MVCC-enabled engine could simplify your data layer while delivering predictable performance and robust resilience.