Running list of NMDC schema/data problems/ideas/thoughts

[bmeluch]

A place to collect our notes on interacting with the schema as we get experience being "users" making notebooks.

What works? What doesn't? Anything that we should bring up at the metadata meeting?

[brynnz22]

• No straightforward way to get sample types - environmental triad or environmental package. Requires cleaning @bmeluch
• Missing urls to get data in DataObject should url be required?

[bmeluch]

sample type discussion started here microbiomedata/nmdc-schema#2141

[samobermiller]

Issue: url is the same for three processed nom data objects. all other identifier columns I've pulled (see attached csv) are also different. found this while working on issue #33 . @bmeluch @brynnz22 @kheal may have ideas where these came from and next steps to resolve this redundancy
duplicate_url_info.csv

[bmeluch]

All other data in those rows is the same, and there are no other duplicate LC-MS filenames for this project. I'll try and find a good place to upload these in the Google Drive labyrinth
edit: here

[samobermiller]

Here are all the biosample fields for those biosample ids. looks like all the columns are the same except id
dupe_biosample_allinfo.csv

[kheal]

OK, great. Here's what I see as follow up for this exercise.

1. Sam and Bea drop duplicates from their analysis at the url level
2. We need to remove duplicate records for biosample / processed sample / data generation / data object (raw) / nom analysis / data object (processed). The who and how of this is less clear but that's OK.
3. We should update the metadata generation script to check for duplicate LC-MS filenames during metadata generation for NOM - I'm guessing this is a @brynnz22 task since she's the most in touch with those scripts).

[kheal]

Excellent sleuthing all! 🕵️‍♀️

[kheal]

FYI, these show up as duplicates on the data portal too. Womp womp.

[kheal]

was_generated_by on DataObject is an UNRELIABLE slot. has_input and has_output are required and are more reliable.

Should was_generated_by be required or removed?

[bmeluch]

Worth using md5_checksum as an integrity check to identify duplicate data objects? during ingest or otherwise?