Distributed Data Management System

[omar-rifai]

This thread's purpose is to gather details on the different data management solutions available and consequently decide on the best for our purposes. Please edit the following posts on the "use-cases" and "candidate solutions" with updated information as deemed necessary.

[omar-rifai]

Use Cases

• Version control non-regression data_ci directory.
• Download (fetch) data_ci files on a per-use basis.
• Minimize the storage space used on client side (avoid duplicates and deprecated version).
• Track ML/DL models.

[mdiazmel]

I added "track ML/DL models" as "use cases". This doesn't concern Clinica itself but ClinicaDL. It should be nice to use the same tool for both projects.

[omar-rifai]

One question I had in mind was whether we wanted to track our data along our public repositories (i.e Clinica and ClinidaDL)? My initial thought was that for the non-regression tests data, we needed a separate (private?) repository.

[mdiazmel]

The short answer is a separated repository! (at least for Clinica's non-regression data).

[omar-rifai]

Candidate Solution: DataLad

• Wrapper around git-annex (Stores only symlinks in a repo and the data is stored in a seperate backend)
• DataLad "datasets" are fully git-annex compatible
• Allows easy integration of nested  ”Datasets”
• Ability to act on symlinks (the required data are fetched on a per-use basis)

[omar-rifai]

Candidate Solution: DVC

• Independent of SCM for the core features (but can be used along side git for storing metadata)
• Relies on human-readable .dvc files for storing pipeline stages
• DVC you can specify external outputs to be used in a pipeline