load projects and samples #114

StefanBabukov · 2021-05-16T14:52:07Z

Background

Link to issue

Link to staging deployment URL

https://ui-again2-ui258-api114.scp-staging.biomage.net/data-management

Links to any Pull Requests related to this

API - #114
UI - hms-dbmi-cellenics/ui#258

Anything else the reviewers should know about the changes here

Changes

Code changes

Definition of DONE

Your changes will be ready for merging after each of the steps below have been completed:

Testing

Unit tests written
Tested locally with Inframock (with latest production data downloaded with biomage experiment pull)
Deployed to staging

To set up easy local testing with inframock, follow the instructions here: https://github.com/biomage-ltd/inframock
To deploy to the staging environment, follow the instructions here: https://github.com/biomage-ltd/biomage-utils

Documentation updates

Is all relevant documentation updated to reflect the proposed changes in this PR?

Relevant Github READMEs updated
Relevant wiki pages created/updated

Approvers

Approved by a member of the core engineering team
(UX changes) Approved by vickymorrison (this is her username, tag her if you need approval)

Just before merging:

After the PR is approved, the unstage script in here: https://github.com/biomage-ltd/biomage-utils is executed. This script cleans up your deployment to staging

Optional

Photo of a cute animal attached to this PR

src/api/route-services/samples.js

src/specs/api.yaml

cosa65 · 2021-05-18T13:26:47Z

src/api/route-services/projects.js

+      TableName: this.tableName,
+    };
+    const dynamodb = createDynamoDbInstance();
+    const response = await dynamodb.scan(params).promise();


This scan will need to be over the experiments table (and only getting projectId), then with the projectIds you find there you'll need ot get the projects from projects table (some of the experiments won't have projectIds so you could deal with that by adding a FilterExpression with attribute_exists in the params).

For loading projects based on the projectIds you got from experiments you could use BatchGetItem https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#batchGetItem-property,
also keep in mind that the experiments table has projectId field while projects table has projectUuid, so there's a decision here, either using projectId and changing the projects and samples tables or projectUuid and changing the experiments table. (or just doing a _.transform() between the result of the experiments scan and the projects batch get to change projectId -> projectUuid and leave this matter for some future refactoring) .

ivababukova · 2021-05-19T09:55:30Z

Looking at Data Management page in the staging link, I am not sure if I can see information for experiments/projects from the experiments DynamoDB table. @StefanBabukov can you check if these get displayed?

StefanBabukov · 2021-05-19T10:12:10Z

I made some fixes because of the merge conflics, check again after the build finishes

StefanBabukov · 2021-05-19T12:10:29Z

Samples cannot be fetched due to the permissions issue described here - https://biomage.atlassian.net/secure/RapidBoard.jspa?rapidView=1&projectKey=BIOMAGE&modal=detail&selectedIssue=BIOMAGE-966

src/api/route-services/projects.js

ivababukova · 2021-05-19T14:40:28Z

@StefanBabukov Marcell just resolved the permissions issue for samples for your staging link, it should be working now

cosa65 · 2021-05-19T16:45:30Z

I tried to create a project and am getting Error: ValidationException: Provided list of item keys contains duplicates, do you know if this is relateed to this PR?

marcellp · 2021-05-20T10:59:12Z

I will do some refactoring on this code as it has some idiosyncrasies compared to the rest of the code base. i will try to fix the error you are seeing as well @cosa65 .

marcellp · 2021-05-20T12:32:40Z

This should be in a ready state at this point. I need some unit tests for this that I'll write, but the code itself should be good as is.

cosa65 · 2021-05-20T12:45:34Z

We should split up the batchgetitem request array in groups of 100 and do one batchGetItem for each section due to this:

aerlaut · 2021-05-20T12:53:19Z

src/api/route-services/samples.js

+    const items = response.Items;

-    if (response.Items) {
+    if (items) {
      const prettyResponse = response.Items.map((item) => convertToJsObject(item));
      return prettyResponse;
    }


why not just keep line 32 as response.Items? that's how it's also used in the next line

ivababukova · 2021-05-20T16:14:59Z

On Thu, 20 May 2021 at 17:14, Marcell Pék ***@***.***> wrote: Overall looks good, see the comments on the review. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#114 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APTCL63X7727J7CQ6SUFU2DTOURO3ANCNFSM4463FZZA> .

Marcell, do you need more reviews on this one?

ivababukova

I am approving and merging this and we will do more unit tests and address the batch problem as part of other tickets that I will create just now

* Fixed configs for develop (#106) * change default-config values for new staging develop env * Bump hosted-git-info from 2.8.8 to 2.8.9 (#108) Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9. - [Release notes](https://github.com/npm/hosted-git-info/releases) - [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md) - [Commits](npm/hosted-git-info@v2.8.8...v2.8.9) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump lodash from 4.17.19 to 4.17.21 (#104) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.19 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.19...4.17.21) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Get data upload status (#105) * create projects and samples * update projects and samples * delete projects and samples * Add FastMNN and No integration methods (#107) * Worker failing to restart (#111) * Debug logging on helm command that launches worker * Get helm status * Dsiplay helm history on upgrade failure * Do not run helm update concurrently * fix async-lock * Cleanup * Better logging * Run npm audit fix because dependabot is complaining again (#115) * Launch new experiment (#113) * Store new experiment in DynamoDB * Add type to data processing report (#117) Co-authored-by: Martin Fosco <[email protected]> * Add end-to-end authentication and authorization (#110) * Add authentication check for API * fix * Add authorization to the API * add authorization to socketio * fix existing unit tests and add a new type of test * Add tests for authentication * fix-typo * Fix documentation * learn to spell unauthenticated * debug * more-debug * Suppress missing X-Ray context errors for auth middleware lookups * fix-error * Fix canWrite type issues * Throw if experiment does not have rbac_can_write * Fix broken merge * Remove Xray noisy log * Search for noisy Xray log * Try to convince setContextMissingStrategy not to log * Fix non-context Xray traces * Remove debugging logs * Move some commenst from PR to code Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Xavier Verges <[email protected]> * Add cognito user and admin details to rbac can write (#119) * Add authentication check for API * fix * Add authorization to the API * add authorization to socketio * fix existing unit tests and add a new type of test * Add tests for authentication * fix-typo * Fix documentation * learn to spell unauthenticated * debug * more-debug * Suppress missing X-Ray context errors for auth middleware lookups * fix-error * Fix canWrite type issues * Throw if experiment does not have rbac_can_write * Fix broken merge * Add create experiment endpoint to avoid error message experiment not found when creating new experiment * Add user permissions to saved experiment * Add adminArn * Clear out repetitions before setting rbac_can_write * Remove extra console logs * Test fixes * Add 401 and 403 responses to enabled options * Rename new auth middleware * Move adminArn into config Co-authored-by: Marcell Pek <[email protected]> Co-authored-by: Marcell Pék <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Xavier Verges <[email protected]> * load projects and samples (#114) * redoing the ticket * fix * fixes * fixes * comment fixe * fixes * move to inside function * change * Refactor code to work better with async/await * Make sure keys returned to getProjectsFromIds are unqiue. * Convert projectIds into an Array * Fix conversion error between keys * Add array spreading to other reference to projectIds * Add array spreading to other references to projectIds * Make module to filter out orphaned experiments more explicit * Remove based on items Co-authored-by: Marcell Pek <[email protected]> * Recover settings accidentally removed (#121) * api to load experiments (#126) * created gem2s endpoint (#109) * created gem2s endpoint * added skeleton parameter to state-machine-definition tests * passing all required params * fixed qc-pipeline skeleton * fixed gem2s pipeline skeleton name * renamed ARN to avoid clashes * converted gem2s pipeline steps into normal ones instead of maps * added last steps for gem2s * removed testing clutter * fix manifest grep * fixed some tests * using personal IAC to bypass runner name temporarily * added debug logs * removed old comment * added temporarily pipeline config until the updated one is deployed to iac * added authorization to new endpoints Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Address comments pr 126 (#128) * add route test * add test for getListOfExperiments * Save gem2s pipeline handle (#120) * created gem2s endpoint * added skeleton parameter to state-machine-definition tests * passing all required params * fixed qc-pipeline skeleton * fixed gem2s pipeline skeleton name * renamed ARN to avoid clashes * converted gem2s pipeline steps into normal ones instead of maps * added last steps for gem2s * removed testing clutter * fixed some tests * Save gem2s pipeline handle Co-authored-by: Pol Alvarez <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Get projects test (#123) * test for projects route * write tests for projects * rewrite AWS mock * Handle gem2s response (#127) * created gem2s endpoint * added skeleton parameter to state-machine-definition tests * passing all required params * fixed qc-pipeline skeleton * fixed gem2s pipeline skeleton name * renamed ARN to avoid clashes * converted gem2s pipeline steps into normal ones instead of maps * added last steps for gem2s * removed testing clutter * fixed some tests * Save gem2s pipeline handle * Save response to dynamodb * Add gem2s handle and status report to status * Fix dynamodb updates to experiments and other minor things * Fix tests and minor bugs * Minor fixes Co-authored-by: Pol Alvarez <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Require auth for saving and loading projects (#124) * require auth for saving and loading projects * api to load experiments * add auth to routes * Request GEM2S status from the backend (#118) * add gem2s when requesting backend status * create routes, add tests * change from pipelines to backend-status * remove package-lock.json * Delete project delete experiment [BIOMAGE-948] (#131) * delete experiments * add experimentDelete test * add checks for empty experiment * BIOMAGE-792 Return only experiments and projects that a user has access to after logging in (#122) * Only list projects with access * Check for missing user. No exception on empty list * Fixed query * Accpet empty arrays * Fix length check * Restore change lost on merge * Update tests * Fix transition between gem2s and data processing (#130) * Fix discrepancies in qc and pipeline renaming * Fix various bugs * When no samples, change 500 for 404 * Implement xaviers fix on statusResToSend * Change updateExperiment to work by doing one single request * Refactor changes in previous commit * Fix qc/pipeline discrepancy problem on work requests send * Change names to more readable options * Rename helpers * Minor rafctoring based on agi comment Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Xavier Verges <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Handle gem2s response refactoring (#129) * Adapt response swagger validation * Add missing change from status to taskName Co-authored-by: Martin Fosco <[email protected]> * Hide some wrong xray missing context logs + typos (#133) * fix nonexistent property access (#137) * [BIOMAGE-1037] - Pass metadata to gem2s (#135) * pass metadata to gem2s * exclude metadata if empty * [BIOMAGE-1036] - Run QC after GEM2S (#134) * run qc after gem2s * add test * await for pipeline runs * [BIOMAGE-1009] Fix error sns messages (#136) * Adapt code in api validation and handler in gem2s to new structure * Fix response handlers * Minor fixes and refactoring * Update test snapshots * Fix indentation * Fix validation swagger files * Fix tests Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * convert to async/await (#141) * Beter 404 for plots-tables (#140) * Beter 404 for plots-tables * Friendlier logging of expected errors * Broken if omg * Better logging * Log hook execution * Wrong condition, again :-( * Take care of falsy parsedMessage * Sanitize metadata key (#142) * [BIOMAGE-1027] Add safeBatchGetItem that divides batchGetItem in groups of requests of 100 keys (#138) * Add safeBatchGetItem * Add tests * Add explanatory comments * Rewrite safeBatchGetItem to work with limit of 100 for total of items instead of per table * Minor bugfix and add more unit tests * Remove obsolete object Co-authored-by: Martin Fosco <[email protected]> * [BIOMAGE-989] - Work request throw error if pipeline has not yet succeeded (#139) * new flow for releases (#143) * added deployment for release/hotfixes PRs to master (#145) * using tokens to allow flow to trigger testing * building releases/hotfixes on PR instead of push * Update ci-develop.yaml * Delete files in s3 for deleted samples (#146) * Add deletion of files of deleted project * Add basic delete single sample operation (still needs updates) * Remove samples unnecessary ids field * Fixes and add validation * Fix tests * Fix tests and cleanup code * Reorder * Get sample files instead of hardcoded files * Fix tests Co-authored-by: Martin Fosco <[email protected]> * notifying worker of job done (#151) * notifying worker of job done * Fix louvain cellsets storage in s3 * Add small comment * clean up Co-authored-by: Martin Fosco <[email protected]> * [BIOMAGE-1044] Add a needsRunning flag to gem2s pipeline status (#147) * Add needsRunning flag to gem2s status Adds a paramsHash to the gem2s handle * Minor readibility improvements and cleanup * Refactoring and fix issue with taskparams hash * Refactoring and fix tests * Fix errro during run * Refactor fix Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Pol Alvarez <[email protected]> * Deprecated change for ci.yaml Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Anugerah Erlaut <[email protected]> Co-authored-by: Juanlu <[email protected]> Co-authored-by: Xavier Vergés <[email protected]> Co-authored-by: ivababukova <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Marcell Pék <[email protected]> Co-authored-by: Marcell Pek <[email protected]> Co-authored-by: StefanBabukov <[email protected]> Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]>

[BIOMAGE-2111] - Use selected cell sets in Trajectory Plot

StefanBabukov added 3 commits May 16, 2021 17:50

redoing the ticket

54733d8

fix

3c25dbc

fixes

b76dcf8

StefanBabukov mentioned this pull request May 17, 2021

Load projects samples hms-dbmi-cellenics/ui#258

Merged

9 tasks

StefanBabukov changed the title ~~[draft] load projects and samples~~ load projects and samples May 17, 2021

StefanBabukov commented May 17, 2021

View reviewed changes

src/api/route-services/samples.js Outdated Show resolved Hide resolved

ivababukova reviewed May 18, 2021

View reviewed changes

src/specs/api.yaml Show resolved Hide resolved

cosa65 reviewed May 18, 2021

View reviewed changes

fixes

5e8cf39

StefanBabukov requested a review from cosa65 May 19, 2021 09:16

fixes

c151211

cosa65 reviewed May 19, 2021

View reviewed changes

src/api/route-services/projects.js Outdated Show resolved Hide resolved

src/api/route-services/projects.js Outdated Show resolved Hide resolved

StefanBabukov added 3 commits May 19, 2021 17:54

comment fixe

eb7eec6

fixes

c660595

move to inside function

2abca9e

change

1e189e2

marcellp added 8 commits May 20, 2021 12:02

Refactor code to work better with async/await

f2cb9a1

Make sure keys returned to getProjectsFromIds are unqiue.

25bf06f

Convert projectIds into an Array

43c4734

Fix conversion error between keys

158ba33

Add array spreading to other reference to projectIds

69a3ab4

Add array spreading to other references to projectIds

997d8a7

Make module to filter out orphaned experiments more explicit

e0c8eb6

Remove based on items

adbbc4d

aerlaut reviewed May 20, 2021

View reviewed changes

ivababukova approved these changes May 21, 2021

View reviewed changes

ivababukova merged commit fa2f5e0 into develop May 21, 2021

ivababukova deleted the loadSamplesProjects branch May 21, 2021 09:35

alexvpickering pushed a commit that referenced this pull request Mar 14, 2023

Merge pull request #114 from biomage-org/trajectory-cell-set

0df769a

[BIOMAGE-2111] - Use selected cell sets in Trajectory Plot

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

load projects and samples #114

load projects and samples #114

StefanBabukov commented May 16, 2021 •

edited

Loading

cosa65 May 18, 2021 •

edited

Loading

ivababukova commented May 19, 2021

StefanBabukov commented May 19, 2021

StefanBabukov commented May 19, 2021

ivababukova commented May 19, 2021

cosa65 commented May 19, 2021

marcellp commented May 20, 2021

marcellp commented May 20, 2021

cosa65 commented May 20, 2021

aerlaut May 20, 2021

ivababukova commented May 20, 2021 via email

ivababukova left a comment

load projects and samples #114

load projects and samples #114

Conversation

StefanBabukov commented May 16, 2021 • edited Loading

Background

Link to issue

Link to staging deployment URL

Links to any Pull Requests related to this

Anything else the reviewers should know about the changes here

Changes

Code changes

Definition of DONE

Testing

Documentation updates

Approvers

Just before merging:

Optional

cosa65 May 18, 2021 • edited Loading

Choose a reason for hiding this comment

ivababukova commented May 19, 2021

StefanBabukov commented May 19, 2021

StefanBabukov commented May 19, 2021

ivababukova commented May 19, 2021

cosa65 commented May 19, 2021

marcellp commented May 20, 2021

marcellp commented May 20, 2021

cosa65 commented May 20, 2021

aerlaut May 20, 2021

Choose a reason for hiding this comment

ivababukova commented May 20, 2021 via email

ivababukova left a comment

Choose a reason for hiding this comment

StefanBabukov commented May 16, 2021 •

edited

Loading

cosa65 May 18, 2021 •

edited

Loading