-
Notifications
You must be signed in to change notification settings - Fork 174
ForestDB 1.0 Beta
We are pleased to announce ForestDB 1.0 Beta availability. The ForestDB project was launched a year ago as part of developing the new storage engine for both Couchbase NoSQL database system and Couchbase Lite, the native JSON database for mobile.
ForestDB is a fast and space-efficient key-value storage engine that can be used as an embedded storage layer for applications running on high-end servers or mobile devices. Its main index structure is built from Hierarchical B+-Tree based Trie, or HB+-Trie, to address the main drawbacks of typical B+-Tree index structure that has been widely used in traditional database systems. HB+-Trie was originally presented at ACM SIGMOD 2011 Programming Contest.
From our benchmark results, we observed that ForestDB especially shows much better I/O performance on Solid-State Drives (SSDs) and flash memory.
The ForestDB main features are summarized as follows:
- Keys and values are treated as an arbitrary binary.
- Applications can supply a custom compare function to support a customized key order.
- A value can be retrieved by its sequence number or disk offset in addition to a key.
- Write-Ahead Logging (WAL) and its in-memory index are used to reduce the main index lookup / update overhead.
- Multi-Version Concurrency Control (MVCC) support and append-only storage model.
- Multiple snapshot instances can be created from a given ForestDB instance to provide different views of database.
- Rollback is supported to revert the database to a specific point.
- Ranged iteration by keys or sequence numbers is supported for a partial or full range lookup operation.
- Manual or auto compaction can be configured per ForestDB database file.
- Transactional support with read_committed or read_uncommitted isolation level.
- Configurable buffer cache that provides much better cache efficiency than the OS page cache.
For more details, please visit the Overview page.
We have measured the ForestDB performance in various cases (e.g., initial load, read-heavy, write-heavy, mixed load, etc.) and compared them with LevelDB and RocksDB on server OS environments (Centos and Ubuntu). The performance results show that ForestDB outperforms both LevelDB and RocksDB in all the benchmark cases with less storage overhead.
In addition, we conducted the performance comparisons between Sqlite and ForestDB on a mobile device, and observed that ForestDB shows much better (up to 8x) read / write performance than Sqlite.
Please visit our performance page for more details.
We have tested ForestDB on various server OS environments (Centos, Ubuntu, Mac OS x, Windows) and mobile OSs (iOS, Android).
From our benchmark results, we found that the compaction overhead is significant in ForestDB because all the valid pages should be copied from an old file to a new compacted file. To reduce this overhead, we plan to investigate the new compaction scheme that can avoid this copy overhead by implementing the dedicated volume manager.
As another optimization, we will extend the ForestDB I/O module to leverage async IO library (e.g., libaio) for exploiting the parallel IO capabilities provided by SSD drives. This optimization would provide much better disk utilization in traversing secondary indexes when items satisfying a query predicate are located in multiple blocks.
A single ForestDB file can only maintain a single key-value instance at this moment. In some use cases, applications need to create lots of key-value instances and read/write items from/to those instances. For this, we plan to support multiple key-value instances in a single ForestDB file, so that applications can avoid maintaining a lot of individual files and unnecessary overhead.
We also plan lots of other optimizations and new features. Please refer to our future work page.
ForestDB is the open source project under Apache 2.0 license. We always welcome any feedback and contributions from the community. Please read the Couchbase Contribution Guideline for instructions. The ForestDB source code is available in the [Github repository] (https://github.com/couchbaselabs/forestdb)