FFindex is a very simple index/database for huge amounts of small files. The files are stored concatenated in one big
data file, seperated by \0. A second file contains a plain text index, giving name, offset and length of the small
files.
Performance was measured on a collection of 190k BinaryCIF files (stored as .bcif.gz). Multi-threaded operations could
use up to 12 cores.
| Approach | read 1,000 random files [ms] |
|---|---|
| File System | 48 |
| FFindex-java | 30 |
| Approach | bundle 190,000 files [s] |
|---|---|
| tar | 57 |
| tar + gzip | 166 |
| tar + pigz (multi-threaded) | 62 |
| FFindex-java (single-threaded) | 80 |
| FFindex-java (multi-threaded) | 12 |
| Approach | size [GB] |
|---|---|
| plain directory | 3.8 |
| tar | 3.6 |
| tar + gzip | 3.5 |
| FFindex-java | 3.4 GB (data), 6.0 MB (index) |
FFindex-java provides access to a "file bundle", i.e. a pair of a FFindex data file and an index file.
Open a file bundle using:
ReadableFileBundle readable = FileBundleIO.openBundle(dataPath, indexPath).inReadOnlyMode();Bundles provide some information about the content of a bundle (like the number of files and the names of registered files).
Read individual files by invoking #readFile(String filename).
All data read is returned as ByteBuffer.
WritableFileBundle writeable = FileBundleIO.openBundle(dataPath, indexPath).inWriteOnlyMode();Writable bundles can write data using e.g. #writeFile(String filename, ByteBuffer content).
AppendableFileBundle appendable = FileBundleIO.openBundle(dataPath, indexPath).inReadWriteMode();Appendable bundles support all functionality of read-only and write-only bundles. However, they require more memory to track metadata of written files, so it's a good idea to use read-/write-only bundles if possible.
No guarantees are made that files produced by this project are interoperable with the original FFindex files or implementations. This implementation is motivated by FFindex and produces identical files in simple cases. However, the original FFindex anticipates index entries to be sorted (to perform binary search on it), this implementation writes entries in their insertion order and implements access by a map.
The maximum size of individual files to store is ~2 GB. The maximum number of files is capped by the size of Java arrays as well.
FFindex was written by Andreas Hauser [email protected].
FFindex is registered trademark of the Ludwig-Maximilians-Universität, Munich (LMU). FFindex is provided under the Create Commons license "Attribution-ShareAlike 4.0".
The reference implementation can be found at: https://github.com/ahcm/ffindex
Check out the ffindex command-line tool for useful functionality that isn't part of this project.