GOLD ecosystem pathway enumerations are out of date

[aclum]

I'm not sure when this was last updated but GOLD's last release of ecosystem pathways was in Sept 2023. I noticed this because value of Peat for column specific ecosystem does not validate and confirmed it is not listed in the enumeration SpecificEcosystemEnum

We should

• revisit all of the gold path fields https://microbiomedata.github.io/submission-schema/gold_path_field/ and their respective enumerations and get them up to date
• Figure out a maintenance schedule for GOLD pathways. Per Reddy this is contentiously updated w/no release cycle.

@turbomam @pkalita-lbl @mslarae13 @shreddd

[mslarae13]

I agree. See comment in microbiomedata/nmdc-schema#1108 (comment)

[mslarae13]

We are missing Bulk Soil which is the 'specific_ecosystem` that Hugh wants to list the NEON samples as.

[aclum]

@turbomam do you have time update the GOLD pathway enumerations? current values in GOLD can be found here https://gold.jgi.doe.gov/ecosystem_classification

[turbomam]

Where can I find a textual representation of the GOLD pathway elements?

[turbomam]

Maybe here? GOLD's 5-Level Ecosystem Classification Paths Excel Last generated: 11 Jan, 2024

Clicking the link downloaded this file: GOLDs5levelEcosystemClassificationPaths.xlsx

This should be noted in the schema

[turbomam]

Are we adding all values form all five categories into the enums? Here's list of all five, ranked by the number of paths they appear in. I could report it some other way if you want.

Deleting long list for now. Will post somewhere else soon.

[aclum]

@turbomam yes please

[turbomam]

How are the GOLD path elements modeled in the nmdc-schema and the submission schema?

Here's the definition of SpecificEcosystemEnum in the compiled submission schema

And the other four enums, which are contiguous at this point in time.

An example value for EcosystemSubtypeEnum is Floodplain and is currently modeled in this style

      Floodplain:
        text: Floodplain
        description: placeholder PV descr

Floodplain doesn't appear anywhere in the nmdc-schema

I think these enumeration origiante in https://github.com/microbiomedata/submission-schema/blob/main/schemasheets/tsv_in/enums.tsv which has been hand-curated up until now.

[turbomam]

@pkalita-lbl can you please help me think about the GOLD path enum lifecycle?

• What are the consequences of changing them?
• Do you still have code in the SubmissionPortal that checks the value in one column against the values in the other four columns?

[turbomam]

schemasheets/tsv_in/enums.tsv has the following columns:

• DH pulldown column
• DH pulldown option
• description
• term_id
• old SNTC name
• MIxS see also
• notes

For all practical purposes, we're just asserting the enum name and the permissible value name in DH pulldown column
and DH pulldown option. I have been asserting 'placeholder PV descr' as the description for some black-magic reason that I can't remember

[turbomam]

Fetching ecosystem path data from GOLD

• visit https://gold.jgi.doe.gov/ecosystem_classification
• find this link at the bottom of the screen: GOLD's 5-Level Ecosystem Classification Paths Excel
  • which resolves to https://gold.jgi.doe.gov/download?mode=ecosystempaths
• clicking the link downloaded this file: GOLDs5levelEcosystemClassificationPaths.xlsx

assets/GOLDs5levelEcosystemClassificationPaths.xlsx:
	curl -o $@ https://gold.jgi.doe.gov/download?mode=ecosystempaths

GOLD's source file calls the path elements

• ECOSYSTEM
• ECOSYSTEM CATEGORY
• ECOSYSTEM TYPE
• ECOSYSTEM SUBTYPE
• SPECIFIC ECOSYSTEM

We are calling the enums

• EcosystemCategoryEnum
• EcosystemEnum
• EcosystemSubtypeEnum
• EcosystemTypeEnum
• SpecificEcosystemEnum

[turbomam]

I started working on this out of nmdc-schema. We can move this later if it does what you want.

• all values for all 5 GOLD path elements nmdc-schema#1642

[pkalita-lbl]

Right submission-schema has an enum for each of the GOLD pathway levels. They are definitely not complete. Like, the first three levels only allow one permissible value each (EcosystemEnum, EcosystemCategoryEnum, EcosystemTypeEnum). The other two offer more options, but again definitely not complete (EcosystemTypeEnum, SpecificEcosystemEnum). I assume the incompleteness was done on purpose because there sure are a lot of options, but that decision predates my time on this project.

There is also custom code in the submission portal that alters the behavior of those five columns so that you only get suggestions for valid paths. The logic is driven in part by this file: https://gold.jgi.doe.gov/download?mode=biosampleEcosystemsJson (we bake a copy into the submission portal code; we don't constantly re-fetch it). So for example, when you go to to fill in the specific_ecosystem column the options that get presented to you are determined by what the GOLD JSON file says are valid values based on the values in the 4 previous columns and then we subset that by what's permissible according to SpecificEcosystemEnum.

I see two options going forward:

1. We could get rid of the 5 enums in submission-schema and make the range of the 5 slots string (mimicking what we do in nmdc-schema). Then the logic for the dropdowns in the submission portal would only be driven by the GOLD JSON file. That makes updating to get the latest GOLD terms easy; it's just that one file. The potential downside is that you lose the ability to exclude GOLD terms that we deem irrelevant.
2. We write some kind of script to inject all of the GOLD terms into the corresponding enums in submission-schema. The update process would then be: run that script and commit the changes to submission-schema, update the GOLD JSON file in nmdc-server. Depending on how sophisticated we make that script would could potentially exclude certain GOLD paths if that's desired.

[pkalita-lbl]

Also a long time ago I tried generating a LinkML schema that encoded the valid pathways as rules (code here https://github.com/pkalita-lbl/gold-ecosystems-linkml). The result was so unwieldy that it was basically unusable. So no one suggest doing that!

[turbomam]

Thanks @pkalita-lbl !

I have implemented at least half of option 2. from above as

• all values for all 5 GOLD path elements nmdc-schema#1642

I don't mind if you decide to go with option 1. instead

[turbomam]

@pkalita-lbl (or anyone): Are subsets of the GOLD paths being created for the the different environmental contexts like soil or water?

[pkalita-lbl]

Are subsets of the GOLD paths being created for the the different environmental contexts like soil or water?

I'm not sure but I think that's another thing that will influence how we implement the long-term process for keeping us in sync with GOLD. So I'm not sure we're ready to jump into implementing anything quite yet.

[mslarae13]

@pkalita-lbl

They are definitely not complete. Like, the first three levels only allow one permissible value each (EcosystemEnum, EcosystemCategoryEnum, EcosystemTypeEnum).

We did intentionally limit this. That said, how it's limited will vary from sample type to sample type (environmental extension to extension)

The other two offer more options, but again definitely not complete (EcosystemTypeEnum, SpecificEcosystemEnum).

The missing 'lower level' ecosystem terms are cuz GOLD updated and we didn't get the updates.

So for example, when you go to to fill in the specific_ecosystem column the options that get presented to you are determined by what the GOLD JSON file says are valid values based on the values in the 4 previous columns and then we subset that by what's permissible according to SpecificEcosystemEnum.

Yes, we don't want to lose this because it should build the same way the GOLD ecosystem tree does: https://gold.jgi.doe.gov/ecosystemtree

Are subsets of the GOLD paths being created for the the different environmental contexts like soil or water?

@turbomam pretty sure that's a yes. But we haven't done it. it's really just identifying where in the tree we would limit..
So, for water, https://gold.jgi.doe.gov/ecosystemtree
Environmental > Aquatic (then the other 3 are any sub of that).
@aclum please confirm.

[pkalita-lbl]

Done with:

• Update GOLD ecosystem classification path terms #173
• Upgrade to submission-schema 10.0.0 nmdc-server#1119