automatic processing of OpenNeuro annotations

[surchs]

Context

We now have https://github.com/OpenNeuroDatasets-jsonld for OpenNeuro forks. But so far everything happens manually (pushing a change, pulling in an upstream change, listening to a change and then processing the new version). Now we want things to happen automatically.

https://hackmd.io/bbi4VBoGREyb4m820coQww

Why

• Part of Grow the amount and usefulness of data #11 and Grow userbase #49
• Follows OpenNeuro launch #38
• So I can annotate a single OpenNeuro dataset when I have a moment, and then
• let the bots take care of the rest so that I can then find it in the federation search

Outcomes

• there is a fork in https://github.com/OpenNeuroDatasets-jsonld for every OpenNeuro dataset
• when upstream changes, that change automatically makes it into the fork (e.g. cron job)
• when we (semi-) manually push to the fork from our side (e.g. to add annotations), this gets combined into a shared history
• when either upstream or direct changes happen to the default branch of the fork, a CI job processes the entire fork with the neurobagel CLI and makes a .jsonld file ready for the graph
  • we need help on this one
  • we know where to run the CLI (with enough resources) + where outputs should be stored (outputs should be all in one repo)
• that new .jsonld file is then stored somewhere we can easily (ideally automatically) pick it up to push it into the graph
• Enable CORS tools#19

What

• Add workflow to detect and auto-launch CLI when a repo has changes OpenNeuroDatasets-JSONLD/.github#16
• Build Docker on release, not (just) push bagel-cli#183

[alyssadai]

Here's our tentative script to run the CLI on one dataset: https://github.com/neurobagel/bulk_annotations/blob/main/datalad_get_single_dataset.sh

[alyssadai]

See also https://github.com/neurobagel/documentation/wiki/Annotating-OpenNeuro-datasets

[rmanaem]

[rmanaem]

Closing this as complete as all desired outcomes have been achieved.