Programmatic utilisation of inheritance principle

[Lestropie]

This is a proposal for development of a new piece of software that would reside within the BIDS ecosystem.

---

There is debate about the utility of the inheritance principle. Some are pushing to make the principle even more generalised and powerful (eg. bids-standard#1003, #5), others think that the entire principle should be removed from the specification. And it seems that many users struggle to understand the concept itself adequately in order to take advantage of it.

Further, it could be argued that existing software tools fail to properly make use of it, therefore relying on manual interaction:

• For BIDS Raw data (using MRI as an example since it's familiar to me), one typically uses DICOM to NIfTI conversion software that produces one sidecar JSON alongside each NIfTI image, but each of these conversions is essentially independent of all others. So if there is common content across the DICOM data of multiple series, these data will be duplicated across multiple sidecars.

• For BIDS Derivatives data, such data would typically be generated by processing each participant individually. There may be metadata geenrated by the BIDS App that is consistent across all participants, but that content is duplicated across all participants.

  (You do not want a BIDS App to be writing metadata from a participant-level analysis at a higher level in the filesystem hierarchy than the participant itself in order to be inherited by data from all participants, since this could lead to race conditions in the scenario where multiple participants are processed in parallel.)

---

What I propose here is that in both of these scenarios, optimal exploitation of the inheritance principle could be achieved programmatically. A new software tool would be executed upon completion of generation of a dataset (eg. conversion of all DICOM data for all participants in the case of BIDS Raw, processing of all participants in the case of BIDS Derivatives). For each unique metadata key-value pair, it would find the highest possible level in the filesystem hierarchy and the minimal set of entities by which that metadata key would still be associated with all data files to which it is applicable, and still not be associated with all data files to which it is not applicable. The metadata files of the dataset would then be re-written in such a way that no metadata key-value pair is unnecessarily duplicated, and interpretation of the dataset is unaffected as long as the inheritance principle is obeyed (and this would likely depend on software packages making use of BIDS APIs that themselves properly load metadata in a manner faithful to the inheritance principle).

This wouldn't be a BEP, but a piece of software that is intended for utilisation within experiments that themselves utilise BIDS. So similar to BEP015, which became the File Mapper utility.

I can potentially create a toy example demonstrating what such a tool would do to an exemplar dataset if there is adequate interest in the concept. I'll hold off on investing that effort for now, in part because the scope of the inheritance principle is right now under debate as part of the BIDS Connectivity project.

[Lestropie]

Pinging @yarikoptic.

Also, if there's merit to the idea, could re-post this as an issue in the upstream repository rather than my fork.

[Lestropie]

Moved to: https://github.com/Lestropie/IP-freely.