239 update import logic to update omicsprocessing record #274

mbthornton-lbl · 2024-10-16T15:14:10Z

This PR provides updated to import automation to:

create a sequencing DataObject and update the has_output list of the Data Generation
ensure imported DataObjects and WorkflowExecution records pass validation w/ nmdc-schema before submission
rewrite of the run_import.py script

Includes updates to tests and test fixtures

turbomam · 2024-10-16T15:21:03Z

Thanks, @mbthornton-lbl . I haven't thought about this issue/PR much, so let me know if I'm missing the point!

How did you decide to construct record dictionaries instead of instantiating classes from the schema?

turbomam · 2024-10-16T15:21:37Z

Maybe that's just in the tests!

turbomam · 2024-10-16T15:32:10Z

Sorry, I'm not being very helpful here.

I don't see the "null", None, [], "" stripper in the files changed, suggested by https://nmdc-group.slack.com/archives/C05ASAQ1FM1/p1729091945783829

tests/test_imports.py

tests/fixtures/models/mags_analysis_record.json

nmdc_automation/import_automation/activity_mapper.py

nmdc_automation/models/nmdc.py

prof/test_workflow_process_node[read_qc_analysis_record.json-nmdc_ReadQcAnalysis].prof

tests/nmdc_materialized.yaml

tests/nmdc_materialized_patterns.yaml

tests/test_imports.py

…output

nmdc_automation/import_automation/activity_mapper.py

mbthornton-lbl added 3 commits October 15, 2024 14:52

add test fixtures

7777dcf

add test for DataObject construction w invalid D.O Type

80c6bdc

add test case for data object with invalid data category

52ea807

mbthornton-lbl requested review from aclum, turbomam and pkalita-lbl October 16, 2024 15:14

mbthornton-lbl linked an issue Oct 16, 2024 that may be closed by this pull request

update import logic to update OmicsProcessing record #239

Closed

mbthornton-lbl added 4 commits October 16, 2024 09:14

Refactor to move model classes

1b3bd9a

incorporate linkml.validate into workflow_process_factory

d611c2c

implement map_sequencing_data with test

00b335a

add map_data method and unit tests

cba6264

aclum reviewed Oct 17, 2024

View reviewed changes

tests/test_imports.py Outdated Show resolved Hide resolved

mbthornton-lbl added 6 commits October 18, 2024 11:09

update DataObject and serialization

6aa29aa

delete legacy warning

cf0b4cc

update click script

3235112

get validation working in run_import

6ba615e

add update and post objects api calls

42f8dca

clean up unused methods in activity mapper

0e1e3e5

mbthornton-lbl marked this pull request as ready for review October 18, 2024 22:57

This was linked to issues Oct 18, 2024

berkeley issue with DataObject records from import_automation #267

Closed

import automation generating mongo keys for which values are None or an empty list #259

Closed

aclum reviewed Oct 18, 2024

View reviewed changes

tests/fixtures/models/mags_analysis_record.json Outdated Show resolved Hide resolved

aclum reviewed Oct 18, 2024

View reviewed changes

nmdc_automation/import_automation/activity_mapper.py Outdated Show resolved Hide resolved

aclum reviewed Oct 18, 2024

View reviewed changes

nmdc_automation/import_automation/activity_mapper.py Outdated Show resolved Hide resolved