Ncpi index fix

[alexwaldrop]

No description provided.

[YaphetKG]

Thanks Alex, so this means for this new round we are expecting a single gz file with sub directories denoting the types ?

[alexwaldrop]

single gz file

@YaphetKG the BDC files in this folder were the ones listed on NCPI, so I don't know if we want to treat that as a more definitive source than the original BDC data tarball we've been using to index BDC datasets. What if we just keep using the original BDC tarball to index BDC files and use this ncpi tarball to index all non-BDC datasets?

[YaphetKG]

Hi @alexwaldrop that sounds good to me, keeping the original files separate BDC files separate from NCPI new datasets. I was a little unclear on the structure of NCPI datasets, I see that there is a folder structure in this new tar ball, is that the way we know the different types that are under the NCPI (Anvil , KFDRC, CDRC etc...) ?

From the setup in airflow currently we are using a single tar file as an input to the tasks, so this means we can easily add more tar inputs and map them to a parser in roger, as an example for anvil Here is the file fetcher code and the annotation task ,

This is setup so that since our unit of data currently is a tar file (or Topmed sets), and we are able to parallelize at that level. I think if we group all NCPI into one tar file we would loose that.

As a remedy if we just tar them at the platform level I think we can easily achieve that, what do you think?

[alexwaldrop]

@YaphetKG just updated to make the each NCPI source it's own tarball directory in the data folder. Should be good to merge!

[YaphetKG]

LGTM!