State of DVID for External Use

As of the 2016 DVID Hackathon v0.8.1 release, we consider DVID suitable for use by our collaborators as well as other research groups. DVID is used for production research at Janelia as well as collaborators at Dalhousie and Harvard Universities using the base basho-tuned embedded leveldb (combined storage of ~25 TB).

Notes:

• We do not guarantee support for outside groups given the limited manpower allocated to DVID at this time. Open source contributions are very welcome.

• We might release a RocksDB driver in the future and, depending on the results of benchmarking, migrate our production databases from basho-tuned leveldb to RocksDB. Migration can be done using the dvid "push" command but will take time for multi-TB databases. A RocksDB store will allow use of DVID natively on Windows platform.

• We continue to explore cloud-based data storage for DVID. As of late 2018, Google Cloud Storage is being used in production for large-scale grayscale retrieval and given its immutable nature, we can use many DVIDs in parallel to retrieve the imagery. A long-term goal is to have simple DVID deployments for outside collaborators using Google and/or Amazon cloud services.

• Backups should be made regularly. This currently requires shutdown of DVID for less than 10 seconds while a snapshot is made and an rsync process is run in background. Once the base storage backend is switched to RocksDB, we will provide an online backup solution.

• Authentication and Authorization are not currently available in DVID. This is slightly mitigated by the versioned nature of DVID and the fact that committed data cannot be changed via the HTTP API. Repos and data instances can be deleted but only through the command line, and an optional repo passcode can be used to authenticate delete commands. Lightweight authentication and authorization support is planned using something like password-less tokens. We will only choose a solution that minimally impacts performance of DVID queries.