Update GOLD NMDC ingest pipeline to ingest bioscales

[sujaypatil96]

Make GOLD NMDC dataset ingest pipeline generic enough to import all studies from GOLD.

For now, the pipeline has been tested on EMP500 and Bioscales, and the JSON output it produces is compatible with v2.0.0 of the NMDC Schema.

Updates to come:

• Make pipeline compatible with the latest version, i.e., v7 of the schema

CC: @cmungall @emileyfadrosh @turbomam @mslarae13 @aclum

[sujaypatil96]

{
            "id": "gold:Gb0291745",
            "name": "Rhizosphere soil microbial communities from poplar common garden site in Corvallis, Oregon, USA - BESC-56-Co3_6_53",
            "description": "Rhizosphere soil microbial communities from poplar common garden site in Corvallis, Oregon, USA",
            "part_of": [
                "gold:Gs0154044"
            ],
            "env_broad_scale": {
                "has_raw_value": "",
                "term": {
                    "id": ""
                }
            },
            "env_local_scale": {
                "has_raw_value": "ENVO_00005801",
                "term": {
                    "id": "ENVO:00005801"
                }
            },
            "env_medium": {
                "has_raw_value": "ENVO_00001998",
                "term": {
                    "id": "ENVO:00001998"
                }
            },
            "collected_from": "gold:BESC-56-Co3_6_53",
            "type": "nmdc:Biosample",
            "gold_biosample_identifiers": [
                "gold:Gb0291745"
            ],
            "alt": {
                "has_unit": ""
            },
            "depth": {
                "has_unit": ""
            },
            "elev": {
                "has_raw_value": "62",
                "has_unit": "meters"
            },
            "geo_loc_name": {
                "has_raw_value": "USA: Oregon"
            },
            "lat_lon": {
                "has_raw_value": "44.5882 -123.1925",
                "latitude": 44.5882,
                "longitude": -123.1925
            },
            "ecosystem": "Host-associated",
            "ecosystem_category": "Plants",
            "ecosystem_type": "Roots",
            "ecosystem_subtype": "Rhizosphere",
            "specific_ecosystem": "Soil",
            "add_date": "2021-05-03T00:00:00",
            "habitat": "Rhizosphere soil",
            "host_name": "Populus",
            "location": "USA",
            "mod_date": "2022-09-03T00:00:00",
            "ncbi_taxonomy_name": "rhizosphere metagenome",
            "sample_collection_site": "Rhizosphere Soil",
            "subsurface_depth": {
                "has_unit": ""
            }
        }

Above is an example of what a Bioscales biosample record dump looks like in the JSON output.

Few things to address here:

• empty properties when there are no values for depth, elev, subsurface_depth, etc. are not found in GOLD
  • if we were to set those properties as None in Python, then the entire record would be considered invalid because None is not an acceptable range for any of the slots
• FieldResearchSite is not being transformed properly in json_dumper output

Question: Is it okay to modify the json_dumper module in linkml-runtime to not output key value pairs where the value, for a certain key, or nested key, is empty? For example, depth is depth: { has_unit: "" } so don't include in output?

[sujaypatil96]

Here is a file with GOLD biosample records that have missing annotations.

biosamples_missing_annotations.txt

The course of action here is to relax the schema some of the slots on Biosample class by changing them from required: true to recommended: true.

[cmungall]

We need to merge in data from https://docs.google.com/spreadsheets/d/1A6bynpzssAUpnDzoAQPZ-8L5HU2y3IuWX7mRiRrasgk/edit#gid=195687079

[sujaypatil96]

        {
            "id": "nmdc:bsm-11-gy2fxa47",
            "name": "Rhizosphere soil microbial communities from poplar common garden site in Corvallis, Oregon, USA - BESC-904-Co3_16_51 rhizosphere",
            "description": "Rhizosphere soil microbial communities from poplar common garden site in Corvallis, Oregon, USA",
            "part_of": [
                "nmdc:sty-11-namt4020"
            ],
            "env_broad_scale": {
                "has_raw_value": "ENVO_00000446",
                "term": {
                    "id": "ENVO:00000446",
                    "name": "terrestrial biome"
                }
            },
            "env_local_scale": {
                "has_raw_value": "ENVO_00005801",
                "term": {
                    "id": "ENVO:00005801",
                    "name": "rhizosphere"
                }
            },
            "env_medium": {
                "has_raw_value": "ENVO_00001998",
                "term": {
                    "id": "ENVO:00001998",
                    "name": "soil"
                }
            },
            "collected_from": "gold:BESC-904-Co3_16_51_rhizosphere",
            "type": "nmdc:Biosample",
            "gold_biosample_identifiers": [
                "gold:Gb0291773"
            ],
            "elev": {
                "has_raw_value": "62",
                "has_unit": "meters"
            },
            "geo_loc_name": {
                "has_raw_value": "USA: Oregon"
            },
            "lat_lon": {
                "has_raw_value": "44.5881 -123.1926",
                "latitude": 44.5881,
                "longitude": -123.1926
            },
            "samp_taxon_id": "rhizosphere metagenome [NCBITaxon:939928]",
            "ecosystem": "Host-associated",
            "ecosystem_category": "Plants",
            "ecosystem_type": "Roots",
            "ecosystem_subtype": "Rhizosphere",
            "specific_ecosystem": "Soil",
            "add_date": "2021-05-03T00:00:00",
            "habitat": "Rhizosphere soil",
            "host_name": "Populus",
            "location": "USA",
            "mod_date": "2021-05-03T00:00:00",
            "ncbi_taxonomy_name": "rhizosphere metagenome",
            "sample_collection_site": "Rhizosphere Soil"
        }

Here is an example biosample record that is being outputted by the pipeline.

[aclum]

@sujaypatil96 "collected_from": "gold:BESC-904-Co3_16_51_rhizosphere", still uses a gold identifier. I was under the impression that this was now only allowed as an alternative identifier. I think this is supposed to be the sites, if so we need site identifiers and the rule for that should a match on the the tree name so 'BESC-904-Co3_16_51'. There will be some exceptions if samples or analysis failed but there should be on average 3 biosamples per site (tree)

[aclum]

What version of the nmdc schema is this code working against? you say in your original comment that it is 2.0.0, we'll need nmdc-schema version 7.4 for the using a MassIVE identifiers for metabolomics data at the study level.

[cmungall]

I don't see host taxon in the output json

[aclum]

I don't think host is modeled in the schema yet @turbomam is that correct?

[ssarrafan]

@sujaypatil96 are you working on this? Can I add you as the assignee?

[aclum]

@sujaypatil96 This looks good aside from my question about dev vs prod for the NMDC runtime API

[aclum]

should this call prod instead of dev? https://api.microbiomedata.org/

[turbomam]

@aclum and @sujaypatil96 do you need any more input from me regarding #113 (comment)

[aclum]

@turbomam Sujay will update the API to prod which is the last unresolved item that I had. Feel free to look over the json output in the grow and bioscales slack channels for another set of eyes.

[cmungall]

ideally this would happen in a separate PR

[cmungall]

this looks good but I am worried about the code split between this repo and the runtime repo. We need to prioritize moving across ASAP

[ssarrafan]

@sujaypatil96 can this one be closed now?