Refactor metaproteomics aggregation script

[kheal]

This PR will refactor the generate_metap_agg.py script to address #26.

Overall, the generate_metap_agg.py has been refactored to

1. Use the runtime API to reduce reliance with direct connection to MongoDB and take advantage of validation
2. Access the raw data objects (not mongo data) to generate the functional annotations for aggregation for metaproteomics results (see Add ADR for metaP mongo data issues#920)
3. Validate and submit data via API

Will not be ready for release until microbiomedata/nmdc-schema#2203 has been merged in (done).

[kheal]

@aclum - I'd appreciate any early feedback you have on the general approach, feel free to tag anyone else who should have eyes on this.

I'm not sure how exactly we'll be able to set the API bearer tokens as environmental variables, but that's how I've been doing development (I altered the readme to describe what environmental variables we'll need).

[kheal]

@aclum. Thanks for the input. I've rewritten to include a call to get a bearer token with the API username and password (set as environmental variables).

I've also pulled out the functions we can reuse for the classes into an abstract Aggregator class. That should make the work for translating the other aggregator to use the API more straightforward. So the other MetaGenomeFuncAgg really just has to write its own process_activity method and declare its filters to find appropriate records.

I'll leave this in draft until the next release since it depends on the migrated database and next schema release.

[kheal]

@picowatt I'll let you know when this is ready for review - I'm going to incorporate Alicia's comments and test this after new release first.

[kheal]

With my updated permissions (thanks @eecavanna), I checked that the json.submit endpoint is working as expected.

I loaded a single metaP's annotations to dev mongo's functional_annotation_agg set.
nmdc:wfmp-11-x0zhd078.1 is the ID of the workflow record that is now included in functional_annotation_agg set in dev mongo.

@aclum - is there a server/data portal issue to make sure the functional searches/ingests are expecting MetaProteomics records in the functional_annotation_agg?

[aclum]

No, there is not a corresponding nmdc-server ticket yet, would you please make one @kheal ?

@eecavanna @dwinston what is the max payload json:submit can handle? @kheal what is the max length for expected aggregation results?

[kheal]

I wrote the script so that the json:submit only submits one workflow's aggregation results at at time to avoid payload issues - though I haven't testing the full lot yet. I can run the whole script locally to write into dev mongo overnight as a test.

[kheal]

No, there is not a corresponding nmdc-server ticket yet, would you please make one @kheal ?

Associated ticket filed here: microbiomedata/nmdc-server#1468

[kheal]

Moving this back into draft. Testing revealed the first API call to be exceptionally slow, attempting to fix now.

[kheal]

Moving this back into draft. Testing revealed the first API call to be exceptionally slow, attempting to fix now.

I've implemented a partial fix for this that will likely not work for future subclasses of the Aggregator class. I will submit a follow up issue to fix this once the issue on nmdc-runtime is addressed. Issue filed here: #28

@kheal what is the max length for expected aggregation results?

I tested the full run of the MetaP aggregator sweep method and did not encounter any payload issues with json:submit.

[shreddd]

I think this looks reasonable (my comments are minor and can be ignored if they don't apply).

Also - Is there a way to test this code to make sure it works?

[kheal]

I think this looks reasonable (my comments are minor and can be ignored if they don't apply).

Also - Is there a way to test this code to make sure it works?

@shreddd I'll add your suggested logging options tomorrow, thanks for that helpful feedback. I've tested the script locally and it successfully loaded the records into the dev mongo. When I reran it, the script did nothing (as expected).