[SCHEMA] Add metadata term files

[tsalo]

References #699.

This is a draft that might be useful for determining how to structure metadata YAML files for the schema.

This approach does not incorporate any interactions between metadata terms, or requirement levels, which are both dependent on the suffix or data type to which the metadata term is being applied. This fits well with #603.

Changes proposed:

• Take the metadata JSON files from the bids-validator and convert them to YAML format.
• Add "description" and "key_name" fields to the metadata YAML files.
• Add functions to render the metadata information in tables within the schema. These functions require the specification text to include the requirement levels.
• Create files for a number of metadata terms that aren't present in the validator, but are present in specification tables.

TODO:

• Different units dependent on different data types?
  • Yes
• Include "unit" for string/integer data types?
  • No
• Limits, like minimum and maximum values
  • Taken from validator
• Should we retain references to other info (e.g., lists of valid coordinate spaces for iEEG, EEG, and MEG)?
  • Currently, I have the references still in the files, with the referenced info in "private" yaml files (i.e., files with leading underscore).
• Links that have to be broken up aren't looking great.
  • This is because the flattening step adds a space between separate lines.
• Would be great to replace relative link paths with absolute ones.
• Type determination for objects.
  • additionalProperties defines type.
  • properties should create additional rows. How should individual sub-object-level requirement levels be annotated then?? Take Genetics.Dataset, Genetics.Database, and Genetics.Descriptors as an example.
    • SOLUTION: Instead of trying to expand nested dictionaries into new rows, I've replaced the internal definitions with $refs and added new files for the properties that can be referenced on their own. So now Genetics' properties fields is a dict with three $refs and there are separate files for Genetics.Dataset.yaml etc.
• Make type determination function recursive to deal with nested objects
  • See SoftwareFilters as an example.
• Document new structure once [MISC] move schema documentation into the schema directory #740 is merged.
• Add "relative" and "absolute"(?) path formats for strings?
  • I decided to go with dataset_relative and participant_relative for now.
• Add "unit" format for strings?
  • With PET now merged in, we have a bunch of new metadata fields that just define units of other fields instead of committing to a standard like we have for most other fields with units. Maybe we could define a "unit" format that will let the validator check for valid units?
• Identify and incorporate section-specific definition modifications.

[effigies]

Is there an advantage to a single file per term? Grouping terms feels more natural to me.

[tsalo]

I don't want to group terms into multiple files, since relationships between terms (including their categories) probably vary across the specification, along with any logic that needs to be applied. I figured one file per term would be easier to search and edit than one file for all terms.

[tsalo]

Based on the current code, the following macro call in the specification produces the table at the bottom. Some are missing descriptions right now and others are just missing completely, but it's still pretty cool!

EDIT: Check out https://bids-specification--762.org.readthedocs.build/en/762/04-modality-specific-files/01-magnetic-resonance-imaging-data.html#scanner-hardware

{{ MACROS___make_metadata_table(
   {
      "Manufacturer": "RECOMMENDED",
      "ManufacturersModelName": "RECOMMENDED",
      "DeviceSerialNumber": "RECOMMENDED",
      "StationName": "RECOMMENDED",
      "SoftwareVersions": "RECOMMENDED",
      "HardcopyDeviceSoftwareVersion": "DEPRECATED",
      "MagneticFieldStrength": "RECOMMENDED, but REQUIRED for Arterial Spin Labeling",
      "ReceiveCoilName": "RECOMMENDED",
      "ReceiveCoilActiveElements": "RECOMMENDED",
      "GradientSetType": "RECOMMENDED",
      "MRTransmitCoilSequence": "RECOMMENDED",
      "MatrixCoilMode": "RECOMMENDED",
      "CoilCombinationMethod": "RECOMMENDED",
   }
) }}

[tsalo]

Until we add conditional logic to the schema, this would be how we define requirement levels in the specification.

[tsalo]

Could we just assume that strings have a minimum length of 1? If the field is provided with a string, then I think it should have at least one character to be valid, from a BIDS standpoint. Missing data should be provided with "n/a", so there's not really a case for "" appearing in jsons.

[sappelhoff]

agreed! And then users can still abuse it by providing " ", but it might be sufficient to make them think twice :-)

[tsalo]

I think the same can apply to minItems when the type is array, so I'll remove all cases of minLength: 1 and minItems: 1 from the YAML files.

[tsalo]

I started replacing internal relative links (e.g., ../02-common-principles.html#definitions) with absolute versions (/02-common-principles.html#definitions). How does everyone feel about this change? I think it would be good to apply throughout the whole specification.

[sappelhoff]

Why are you doing that? I thought it's an advantage to point to the relative files and have Mkdocs generate the suitable links for us? Pointing to html directly feels error prone but I can't put my finger on it right now 🤔

[tsalo]

One reason is that relative links depend on where the metadata field is rendered, so what might work for one file might not work for another. It should also make it easier to resolve links in external applications (e.g., NIDM-Terms).

EDIT: To clarify- I'm not proposing actual links. Just "absolute" relative to the specification root, if that makes sense. So /02-common-principles.html instead of https://bids-specification.readthedocs.io/en/stable/02-common-principles.html.

[sappelhoff]

I see 🤔 I couldn't find info on whether that's good or bad: https://www.mkdocs.org/user-guide/writing-your-docs/#internal-links

it's not really mentioned in the docs over there. So maybe we should carefully try it, and if it doesn't break anything it's fine.

[tsalo]

What does everyone think of this approach for extending descriptions for modality- or table-specific purposes? The first entry in the tuple is just the requirement level, like normal, and the second entry is whatever needs to be added to the description to make it work for that modality/table.

This requires that there be a canonical "base" definition of each metadata field, but I think that it's a reasonable requirement.