FlashDB: Ultra-lightweight Embedded Key-Value and Time Series Database

GitHub Action license docs-perfect-blue EN | 中文

FlashDB: An Ultra-Low Footprint Embedded Database for IoT

Introduction

FlashDB is an ultra-lightweight embedded database designed to provide robust data storage solutions for embedded products. It stands out by combining the characteristics of Flash memory with database capabilities to deliver strong performance and reliability while keeping resource usage to an absolute minimum. The goal is to maximize the service life of Flash by minimizing memory footprint and wear, making FlashDB a practical choice for devices with constrained resources.

Key ideas behind FlashDB include a careful separation of concerns: data organization is tailored for flash storage, and the database operations are optimized for embedded environments. The result is a storage engine that behaves like a database but behaves within the tight confines of microcontrollers and single-board computers often used in IoT devices. FlashDB provides two distinct database modes to cover a wide range of use cases:

• Key-Value Database (KVDB)

• A non-relational store that maintains data as a collection of key-value pairs.

• The key serves as a unique identifier, enabling straightforward retrieval, updates, and deletion.

• Designed for simple operations and scalable growth, KVDB is ideal for storing configuration data, parameters, and small files with predictable access patterns.

• Time Series Database (TSDB)

• Stores data in a time sequence, where each record carries a timestamp.

• Optimized for high-volume insertions and efficient time-based queries, making it well-suited for sensor data, status logs, and event histories.

• The TSDB model excels at capturing environmental metrics, device health information, and operational logs in real time.

Usage Scenarios

IoT devices and embedded systems generate an increasingly diverse and large volume of data. FlashDB addresses these realities with flexible storage options and resilient behavior even in harsh conditions. The main usage scenarios span two broad categories:

Key-Value Database (KVDB) scenarios

• Product parameter storage: configuration values, calibration settings, and feature flags that must persist across power cycles.
• User configuration information: preferences and user-related data that devices need to persist locally.
• Small file management: small, frequently accessed files that benefit from quick lookups by key.

Time Series Database (TSDB) scenarios

• Storing dynamically generated structured data: environmental readings (temperature, humidity, air quality), and other sensor data collected by IoT sensors and smart devices.
• Real-time health and status information: data streams from wearables or health-monitoring devices recorded as they occur.
• Operation logs and alarms: recording product history, events, anomalies, and alerts for later analysis or auditing.

Key Features

FlashDB is designed to deliver robust data management without overwhelming system resources. Core features include:

• Ultra-small footprint: RAM usage is effectively near zero, enabling deployment on devices with tiny memory budgets.
• Multi-partition and multi-instance support: The database can be partitioned to handle large datasets, and multiple instances can run concurrently. Partitioning helps reduce retrieval times as data grows, by narrowing search ranges and isolating workloads.
• Wear leveling (wear balance): This feature helps distribute write operations evenly across flash memory, extending the life of the flash medium in write-intensive scenarios.
• Power-off protection: The system is designed to maintain data integrity even in the event of unexpected power loss, a critical requirement for embedded devices that may experience brownouts or power glitches.
• Two KV types supported: FlashDB supports both string and blob (binary large object) value types, giving users flexible choices for how data is stored and retrieved.
• Incremental upgrade for KVDB: After a firmware upgrade, KVDB content can automatically upgrade, simplifying maintenance and ensuring compatibility with newer firmware versions.
• TSDB record status management: You can modify the status of each TSDB record to help with maintenance, lifecycle management, and user workflows.

Performance and Footprint

FlashDB’s performance metrics underscore its suitability for resource-constrained environments. The project presents measured results that illustrate both speed and footprint in real devices and configurations.

TSDB performance test 1 (nor flash W25Q64)

• Test scenario: benchmarking with a W25Q64 NOR flash device.
• Command excerpt:
• Insertions: Append 1250 TSL in 5 seconds, average ~250.00 TSL/s.
• Queries: Average ~4.00 ms per operation for total 1251 TSL, with a total query time of 2218 ms; minimum ~1 ms per query, maximum ~2 ms, and an average of ~1.77 ms per query.
• Takeaway: TSDB on NOR flash can sustain high insertion rates with low-latency queries, making it suitable for streaming sensor data and high-frequency event logging.

TSDB performance test 2 (stm32f2 on-chip flash)

• Test scenario: benchmarking on microcontroller on-chip flash (stm32f2).
• Command excerpt:
• Insertions: Append 13421 TSL in 5 seconds, average ~2684.20 TSL/s, ~0.37 ms per operation.
• Queries: Total spent 1475 ms for 13422 TSL, minimum 0 ms, maximum 1 ms, average ~0.11 ms per query.
• Takeaway: On-chip flash can deliver very high throughput for time-series data with remarkably low latency, enabling near real-time data capture on resource-limited devices.

Footprint (stm32f4 IAR8.20)

• Footprint details (as shown by the map file on IAR):
• fdb.o: ROM 276, RAM 232
• fdb_kvdb.o: ROM 4, RAM 584, RW 356
• fdb_tsdb.o: ROM 1, RAM 160, RW 236
• fdb_utils.o: ROM 418, RAM 1,024
• Interpretation: The FlashDB footprint remains extremely small, reinforcing its suitability for devices with limited flash and RAM resources. The map file demonstrates a compact codebase with lean data structures and tight memory usage.

How to Use FlashDB

FlashDB provides comprehensive documentation to help developers get started quickly and safely. The official documentation is hosted at armink.github.io and covers a range of topics from quick-start guidance to detailed API references.

Quick access to essential guides:

• Quick Start Document
• Porting Document
• Configuration Document
• API Document

These resources offer step-by-step instructions, integration tips, and best practices to maximize the reliability and performance of FlashDB in real-world applications.

License

FlashDB is released under the Apache-2.0 open source license. For detailed terms, users are encouraged to read the LICENSE file included in the project repository. The Apache-2.0 license provides permissive usage, modification, and distribution rights, making FlashDB a flexible option for both personal projects and commercial products.

In-Depth Reflection: Why FlashDB Fits Embedded Needs

• Minimal resource demand: The near-zero RAM footprint means that even devices with a few kilobytes of RAM can benefit from a robust persistent storage solution. This characteristic is especially valuable for microcontrollers with strict budgets and for energy-constrained devices where memory efficiency translates into longer battery life.
• Reliability and data integrity: Power-off protection helps ensure that data remains consistent and recoverable after an unexpected power loss. For devices deployed in remote locations or without reliable power infrastructure, this trait is non-negotiable.
• Flexible data modeling: The two KV types (string and blob) provide practical options for a range of data shapes. KVDB is ideal for small, structured metadata or settings, while TSDB is tailored to time-stamped data streams that require efficient appends and time-based querying.
• Lifecycle-conscious design: Incremental upgrade support for KVDB content means firmware updates do not trap users in a migration nightmare. Automatic upgrades reduce maintenance overhead and help avoid data compatibility issues after updates.
• Management and observability: The ability to modify TSDB record status and to partition data sets enables developers to optimize maintenance workflows, perform selective cleanups, and manage long-running data archives with clarity and precision.

A Practical Roadmap for Developers

If you are evaluating FlashDB for a project, consider the following steps to integrate it smoothly:

1) Define data models

• Decide whether the data you are handling fits KVDB or TSDB semantics.
• For static configuration, parameters, and small files, KVDB is often the most straightforward option.
• For sensor streams, event logs, and health metrics, TSDB provides performance advantages and scalable throughput.

2) Plan partitions and instances

• Determine how large your dataset may become and partition accordingly to minimize retrieval times.
• If needed, set up multiple KVDB/TSDB instances to isolate workloads, parallelize operations, and distribute wear across flash regions.

3) Enable wear leveling and power-off protection

• Confirm that wear leveling is activated in your configuration to maximize flash longevity.
• Ensure that your power management strategy leverages the built-in power-off protection to safeguard data integrity.

4) Integrate with firmware updates

• Leverage KVDB's incremental upgrade capability to keep data structures compatible with new firmware versions without manual migrations.

5) Consult documentation and examples

• Use the Quick Start, Porting, Configuration, and API documents to implement core functionality, adapt to your hardware, and follow recommended practices for robust operation.

6) Measure and optimize

• Use the supplied performance tests to benchmark your target hardware.
• Analyze memory usage and storage footprint through map files and profiling tools, aiming to keep the footprint as small as possible.

A Note on Documentation and Community

The project maintains a set of online resources that cover everything from conceptual overviews to executable examples. If you need help or want to explore advanced features, the documentation hub is a reliable starting point. The community and core contributors frequently update guides, examples, and compatibility notes to align with evolving hardware platforms and toolchains.

Conclusion

FlashDB represents a thoughtful approach to embedded data storage by combining the strengths of Flash memory with two well-defined database paradigms: Key-Value and Time Series databases. Its design emphasizes minimal resource consumption, reliability, and extensibility, making it a compelling option for IoT devices, embedded systems, and microcontroller-based products. Whether your application requires simple parameter storage or high-throughput time-stamped data logging, FlashDB provides a coherent, scalable, and maintainable solution that can adapt as your device ecosystem grows.

Images and Visual Aids

• The FlashDB overview image at the top illustrates the branding and a quick sense of the project’s scope.
• Documentation and build-related badges below the introduction convey ongoing CI status, licensing information, and documentation availability, signaling an active, well-supported project.
• The performance and footprint sections rely on real-world benchmarks and map-file data, helping developers understand the practical implications of deploying FlashDB on various hardware platforms.

Appendix: Source Snippet Highlights Tsdb bench (nor flash W25Q64)

• Insertion: Append 1250 TSL in 5 seconds, average 250.00 TSL/s, 4.00 ms per operation
• Query: Total 2218 ms for 1251 TSL, min 1 ms, max 2 ms, average 1.77 ms per query

Tsdb bench (stm32f2 on-chip flash)

• Insertion: Append 13421 TSL in 5 seconds, average 2684.20 TSL/s, 0.37 ms per operation
• Query: Total 1475 ms for 13422 TSL, min 0 ms, max 1 ms, average 0.11 ms per query

Footprint (stm32f4 IAR8.20)

• Map file shows a very small footprint across modules:
• fdb.o: ROM 276, RAM 232
• fdb_kvdb.o: ROM 4, RAM 584, RW 356
• fdb_tsdb.o: ROM 1, RAM 160, RW 236
• fdb_utils.o: ROM 418, RAM 1,024

License and Quick Access Links

• Apache-2.0 license
• Quick Start Document
• Porting Document
• Configuration Document
• API Document

If you’re exploring embedded data storage today, FlashDB offers a carefully designed balance between capability and footprint. Its dual-mode approach aligns with common IoT use cases, enabling you to store configuration data and time-series measurements efficiently, all while maintaining resilience against power interruptions and hardware constraints. With ongoing documentation, community support, and clear upgrade paths, FlashDB stands as a robust option for developers seeking dependable, lightweight embedded databases.