Retry experiment

[Kristian-A]

Description

1. Adds a new column retryParams in ExperimentExecution table which persists the last request parameters
2. Adds a new endpoint /retry which figures out the type of pipeline and executes it with the same parameters as before (i.e. retryParams)
3. Adds wrappers around the each pipeline run function which extracts the required parameters from the req object. This way the pipelines can be executed directly using the persisted parameters in the db (retryParams).

Details

URL to issue

https://github.com/biomage-org/issues/issues/129

Link to staging deployment URL (or set N/A)

N/A

Links to any PRs or resources related to this PR

biomage-org/ui#2

Integration test branch

master

Merge checklist

Your changes will be ready for merging after all of the steps below have been completed.

Code updates

Have best practices and ongoing refactors being observed in this PR

• Migrated any selector / reducer used to the new format.
• All new dependency licenses have been checked for compatibility

Manual/unit testing

• Tested changes using InfraMock locally or no tests required for change, e.g. Kubernetes chart updates.
• Validated that current unit tests for code work as expected and are sufficient for code coverage or no unit tests required for change, e.g. documentation update.
• Unit tests written or no unit tests required for change, e.g. documentation update.

Integration testing

You must check the box below to run integration tests on the latest commit on your PR branch.
Integration tests have to pass before the PR can be merged. Without checking the box, your PR
will not pass the required status checks for merging.

• Started end-to-end tests on the latest commit.

Documentation updates

• Relevant Github READMEs updated or no GitHub README updates required.
• Relevant Wiki pages created/updated or no Wiki updates required.

Optional

• Staging environment is unstaged before merging.
• Photo of a cute animal attached to this PR.

[codecov-commenter]

Codecov Report

Patch coverage: 72.26% and project coverage change: -0.06% ⚠️

Comparison is base (63b336a) 90.70% compared to head (d90a38f) 90.64%.
Report is 6 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master       #3      +/-   ##
==========================================
- Coverage   90.70%   90.64%   -0.06%     
==========================================
  Files         157      159       +2     
  Lines        3528     3485      -43     
  Branches      391      387       -4     
==========================================
- Hits         3200     3159      -41     
+ Misses        309      305       -4     
- Partials       19       21       +2     

Files Changed	Coverage Δ
src/api.v2/controllers/qcController.js	100.00% <ø> (ø)
src/api.v2/events/index.js	0.00% <0.00%> (ø)
src/api.v2/events/validateAndSubmitWork.js	100.00% <ø> (ø)
src/api.v2/helpers/pipeline/handleQCResponse.js	95.71% <ø> (-0.18%)	⬇️
...api.v2/helpers/pipeline/pipelineConstruct/index.js	86.56% <ø> (ø)
...api.v2/helpers/pipeline/pipelineConstruct/utils.js	98.52% <ø> (-0.05%)	⬇️
src/api.v2/helpers/worker/workSubmit/index.js	60.00% <ø> (-1.41%)	⬇️
src/api.v2/middlewares/authMiddlewares.js	56.38% <0.00%> (+0.71%)	⬆️
src/api.v2/model/ExperimentExecution.js	86.95% <0.00%> (-13.05%)	⬇️
src/api.v2/routes/kubernetes.js	100.00% <ø> (ø)
... and 16 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cosa65]

The functions exported by the controllers should be functions that handle params req res. They are supposed to handle routes directly.

If this function is no longer doing that, then it should be moved over to a different place (probably where all the pipeline stuff is, perhaps we can even merge this with other functions? idk).

I noticed this same situation with seurat and subset controller.

[cosa65]

Personally I'm leaning more towards separating the params and passing them individually if there are not many of them (3 or less perhaps?), but whatever you prefer

[cosa65]

The reason I'm leaning towards separating the params is that having them in an object kind of hides them, and it's not worth it when it's not many params

[Kristian-A]

I understand it is more tractable to separate the parameters, but the reason I went with passing an entire object is because when the /retry endpoint is called the object containing the last parameters that is in the db can be directly passed no matter what the function is. Afterwards each function (runSubset, runSeurat, runGem2s) extracts their necessary parameters.

Otherwise, in the retry endpoint I would have to use a switch statement and for each function I would extract the proper parameters. I really do not like that solution as it is not scalable and I think would lower the code quality.

imo we should leave it like this

[cosa65]

Ok, although I would suggest avoiding a switch when possible (there are usually better ways to define things).

For example, it could be stored in your const stateMachineToPipeline object which already stores functions as [SUBSET_PROCESS_NAME]: (...) => { blablabla; executeSubsetPipeline(...) }.

I see your point though, what do you think of having runSubset, runSeurat, runGem2s assert that all the necessary properties in these parameter objects are defined and throw an error if they aren't? That should help bring out to light any possible bugs we might introduce in the future/possible incorrect stored parameters. It should also help stop any possible data loss (the error would stop the wrong retry from executing).

[cosa65]

If you like the idea of the assert, since you are going to use it in 3 different files I would suggest checking if there isn't an already accessible assert function.

If there isn't, you could add a small function, something like this:

assert = (condition) => {
    if (!condition) throw new Error("Assertion failed")
}

[StefanBabukov]

Why not use the function validateRequest with a RetryRequest.yaml like we do for most requests in the API? Seems like the cleanest solution to ensure the correct params are passed after being fetched from the database.

[cosa65]

hmm I hadn't thought of that because it isn't a request, just a collection of parameters