-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add harmony integration #98
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It works perfectly Alex, great job. Only the comment about the PCs I leave to your choice. However, before approving, I remember that I made changes to the files (api/route-services/__mocks__/experiment.js
and api/src/specs/models/processing-config-bodies/ProcessingConfigDataIntegration.v1.yaml
) in the api when I added FastMNN. Even if the link works, could you update these files or peer with some engineer to see if we are continue using these files?
scdata <- Seurat::RunPCA(scdata, npcs = 50, features = Seurat::VariableFeatures(object=scdata), verbose=FALSE) | ||
if (!have.pca) { | ||
scdata <- Seurat::ScaleData(scdata, verbose = FALSE) | ||
scdata <- Seurat::RunPCA(scdata, verbose=FALSE) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would leave here the comment related to the default value of PCs, just to have it in mind.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @JuanluOnieva-biomage I added the mocks for harmony in the api. I'm also going to refactor a bit as the integration function is getting a bit unwieldly.
* add harmony integration (#98) * add harmony * refactor * remove mistakes * add dispersions for harmony * Fix p.level (#102) * Merge master into dev branch (#100) * handle NA doublet scores (#61) * Added no integration (#59) * Added no integration * Update qc-runner/src/dataIntegration.r Co-authored-by: Juanlu <[email protected]> Co-authored-by: Juanlu <[email protected]> * Bump lodash from 4.17.20 to 4.17.21 in /local-runner (#58) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.20...4.17.21) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump hosted-git-info from 2.8.8 to 2.8.9 in /local-runner (#64) Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9. - [Release notes](https://github.com/npm/hosted-git-info/releases) - [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md) - [Commits](npm/hosted-git-info@v2.8.8...v2.8.9) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Na doublet score (#62) * handle NA doublet scores * treat NA as doublet_score=0 * Add FastMNN step 6-7 (#60) * Add FastMNN step 6-7 * Fix dockerfile * Fix scdata * change order * Fix try-catch * Change value to unisample and fastmnn * Fix assay * Add package * Add dependencies * Change to pak * Add active.reduciton and pak * Add var.explained * Add comments * Persist misc * added improved error handling and traceback (#55) * added improved error handling and traceback * removed dump.frames * no sample output for integration step * moved state_fail_logic to withCallingHandlers block * removed intended debug error * Display the command used to launch docker from lambda (#63) * Move GEM2S as a step in pipeline (#56) * refactor wrapper * Added gem2s functions to pipeline * Adapted pipeline to new schema for gem2s * renamed one task * fixes for gem2s * renamed qc-runner to pipeline-runner * fixed tryCatch from rebase * changed pipeline name to qc * init upload-to-aws * init meta_sets * Added sample sets * complete upload-to-aws * add s3/dynamodb functions * add buckets and tables * writing input to meta.json file * changed some pipeline references to gem2s/pipeline * avoiding creating new containers for gem2s or qc if they already exist * attaching both pipeline containers output (qc & gem2s) to pipeline process stdout * just a lonenly runaway letter * added dependencies * typo * make docker build for me * send dynamodb items to api * reflect nan bugfix from data-inegest * temporarily fix for paws.common version 0.3.11 * change SNS messag type * hotfix * fix * fix Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Pol Alvarez <[email protected]> Co-authored-by: Anugerah Erlaut <[email protected]> * Dependency issues and cellsets (#65) Co-authored-by: Oliver Gibson <[email protected]> * Add sns updates when tasks finish (#67) Co-authored-by: Martin Fosco <[email protected]> * fix per sample custom config (#66) * fix per sample custom config * loop through unique samples * Fix seurat obj not found (#71) * Sample ids property sent as list instead of string * fix seurat_obj not found Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Xavier Verges <[email protected]> * Handle gem2s response refactoring (#68) * Add sns updates when tasks finish * Handle gem2s response refactoring * Minor renaming * Fix * Fix Co-authored-by: Martin Fosco <[email protected]> * [BIOMAGE-1016] Sample uuids fix (#72) * Sample ids property sent as list instead of string * Fix * Draft fix for uuids * Fix * fix * fix * checks Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Fix fast mnn (#75) * Sample ids property sent as list instead of string * Add FastMNN * Restore 5_Upload-to-aws * [BIOMAGE-1040] - Fix metadata structures (#74) * fix data structures * fix for empty metadata Co-authored-by: Oliver Gibson <[email protected]> * [BIOMAGE-1009] Fix error sns messages (#73) * Refactor and add method dedicated to only sending sns error messages * Add experimentId * Move classifier to current position in ui * Move into one single line Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Add experimentID and ingestionDate (#77) * Add experimentID and ingestionDate * Update comment * Init filter stats (#76) * init calc_filter_stats * add filter stats for cellSizeDistribution * return filter stats for all filter steps * Test * restore spacing for ease * revert * reverting spacing changes * more de-change * fix conflict * check samples Co-authored-by: Martin Fosco <[email protected]> * move code from data-ingest (#78) * fix mtcontent plotdata error (#79) * new flow for releases (#83) * added deployment for release/hotfixes PRs to master (#85) * using tokens to allow flow to trigger testing * Fix doublet settings (#81) * test doublet settings * Fix error in doublet scores config Co-authored-by: Oliver Gibson <[email protected]> * building releases/hotfixes on PR instead of push * Fix determination of HOST_IP on Linux (#91) * Fix determination of HOST_IP on Linux * Replace spaces with tab * Add tty flag to fix local output (#92) * Update ci-develop.yaml * Fix default doublet config (#82) * use seed and fix config storage * use scDblFinder to set default threshold * remove gitignore * remove debug * change boolean logic * Rm auto and enabled (#88) * switch from orig.ident to samples (#95) * Move to multipart upload (#94) * Move to multipart upload * Add print * Fix progress-bar * hardcode DEBUG_PATH (#93) * hardcode DEBUG_PATH * fix function name * Remove hardcoded 10x (it is now hardcoded in the ui only) (#96) Co-authored-by: Martin Fosco <[email protected]> * re-enable per sample config (#97) * added r.utils dependency install which is required by seurat-wrappers (#99) * Deprecated change for ci.yaml Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: ogibson-biomage <[email protected]> Co-authored-by: Juanlu <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: seb-mueller <[email protected]> Co-authored-by: Xavier Vergés <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Anugerah Erlaut <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Anugerah Erlaut <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: James Robinson <[email protected]> * done i think (#101) * Add build as a dependency of run (#103) Also add new run-only target * Init R package and test classifier filter (#104) * init r package * ignore debug * remove file names within file * use styler * cleanup * fix Dockerfile * init filter_emptydrops tests * fix introduced error for demo image * add sample aware test * update README; make task names look distinct * update README * update gem2s step messages * add timer to gem2s steps * Enable Slack build failure notifications * Fix defaults (#109) Co-authored-by: Oliver Gibson <[email protected]> * Unlog y values (#112) * unlog cellsizedistribution plot y values * switch plot numbers to improve readibility * Clean pipeline code (#105) * Clean pipeline code * Update pipeline-runner/R/gem2s-6-prepare_experiment.R Co-authored-by: Alex Pickering <[email protected]> * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <[email protected]> * Update pipeline-runner/R/gem2s-helpers.R * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <[email protected]> * fixes * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Alex Pickering <[email protected]> * Init filter high mito and filter gene umi outlier tests (#114) * init filter_gene_umi_outlier tests * init mito filter tests; fix calc_filter_stats * Init cellszdoublet tests (#118) * init filter_gene_umi_outlier tests * init mito filter tests; fix calc_filter_stats * Added tests for cellsize and doublets * export filter_low_cellsize * simplify parts of filter_low_cellsize * restyle and update comment for filter_doublets test * fix filter_doublets documentation Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> * Removed embedding (#107) * Removed embedding * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Fix Co-authored-by: Oliver Gibson <[email protected]> * Fix (#113) * Fix * typo Co-authored-by: Oliver Gibson <[email protected]> * init renv (#110) * try restore remoter docker-compose * Update pipeline-runner/renv.lock.init * Update pipeline-runner/renv.lock * try update all packages * Fix * update README; remove R version * harmony init lockfile * fix Co-authored-by: ogibson-biomage <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> * Move defaultFilterSettings and missing per sample config additions to the pipeline (#119) * Add defaultFilterSettings and per sample config for classifier and mito content * Update gem2s-6-prepare_experiment.R Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: Juanlu <[email protected]> Co-authored-by: ogibson-biomage <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: seb-mueller <[email protected]> Co-authored-by: Xavier Vergés <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Anugerah Erlaut <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Anugerah Erlaut <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: James Robinson <[email protected]> Co-authored-by: StefanBabukov <[email protected]> Co-authored-by: James Robinson <[email protected]> Co-authored-by: Marcell Pek <[email protected]>
* add harmony integration (#98) * add harmony * refactor * remove mistakes * add dispersions for harmony * Fix p.level (#102) * Merge master into dev branch (#100) * handle NA doublet scores (#61) * Added no integration (#59) * Added no integration * Update qc-runner/src/dataIntegration.r Co-authored-by: Juanlu <[email protected]> Co-authored-by: Juanlu <[email protected]> * Bump lodash from 4.17.20 to 4.17.21 in /local-runner (#58) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.20...4.17.21) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump hosted-git-info from 2.8.8 to 2.8.9 in /local-runner (#64) Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9. - [Release notes](https://github.com/npm/hosted-git-info/releases) - [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md) - [Commits](npm/hosted-git-info@v2.8.8...v2.8.9) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Na doublet score (#62) * handle NA doublet scores * treat NA as doublet_score=0 * Add FastMNN step 6-7 (#60) * Add FastMNN step 6-7 * Fix dockerfile * Fix scdata * change order * Fix try-catch * Change value to unisample and fastmnn * Fix assay * Add package * Add dependencies * Change to pak * Add active.reduciton and pak * Add var.explained * Add comments * Persist misc * added improved error handling and traceback (#55) * added improved error handling and traceback * removed dump.frames * no sample output for integration step * moved state_fail_logic to withCallingHandlers block * removed intended debug error * Display the command used to launch docker from lambda (#63) * Move GEM2S as a step in pipeline (#56) * refactor wrapper * Added gem2s functions to pipeline * Adapted pipeline to new schema for gem2s * renamed one task * fixes for gem2s * renamed qc-runner to pipeline-runner * fixed tryCatch from rebase * changed pipeline name to qc * init upload-to-aws * init meta_sets * Added sample sets * complete upload-to-aws * add s3/dynamodb functions * add buckets and tables * writing input to meta.json file * changed some pipeline references to gem2s/pipeline * avoiding creating new containers for gem2s or qc if they already exist * attaching both pipeline containers output (qc & gem2s) to pipeline process stdout * just a lonenly runaway letter * added dependencies * typo * make docker build for me * send dynamodb items to api * reflect nan bugfix from data-inegest * temporarily fix for paws.common version 0.3.11 * change SNS messag type * hotfix * fix * fix Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Pol Alvarez <[email protected]> Co-authored-by: Anugerah Erlaut <[email protected]> * Dependency issues and cellsets (#65) Co-authored-by: Oliver Gibson <[email protected]> * Add sns updates when tasks finish (#67) Co-authored-by: Martin Fosco <[email protected]> * fix per sample custom config (#66) * fix per sample custom config * loop through unique samples * Fix seurat obj not found (#71) * Sample ids property sent as list instead of string * fix seurat_obj not found Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Xavier Verges <[email protected]> * Handle gem2s response refactoring (#68) * Add sns updates when tasks finish * Handle gem2s response refactoring * Minor renaming * Fix * Fix Co-authored-by: Martin Fosco <[email protected]> * [BIOMAGE-1016] Sample uuids fix (#72) * Sample ids property sent as list instead of string * Fix * Draft fix for uuids * Fix * fix * fix * checks Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Fix fast mnn (#75) * Sample ids property sent as list instead of string * Add FastMNN * Restore 5_Upload-to-aws * [BIOMAGE-1040] - Fix metadata structures (#74) * fix data structures * fix for empty metadata Co-authored-by: Oliver Gibson <[email protected]> * [BIOMAGE-1009] Fix error sns messages (#73) * Refactor and add method dedicated to only sending sns error messages * Add experimentId * Move classifier to current position in ui * Move into one single line Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Add experimentID and ingestionDate (#77) * Add experimentID and ingestionDate * Update comment * Init filter stats (#76) * init calc_filter_stats * add filter stats for cellSizeDistribution * return filter stats for all filter steps * Test * restore spacing for ease * revert * reverting spacing changes * more de-change * fix conflict * check samples Co-authored-by: Martin Fosco <[email protected]> * move code from data-ingest (#78) * fix mtcontent plotdata error (#79) * new flow for releases (#83) * added deployment for release/hotfixes PRs to master (#85) * using tokens to allow flow to trigger testing * Fix doublet settings (#81) * test doublet settings * Fix error in doublet scores config Co-authored-by: Oliver Gibson <[email protected]> * building releases/hotfixes on PR instead of push * Fix determination of HOST_IP on Linux (#91) * Fix determination of HOST_IP on Linux * Replace spaces with tab * Add tty flag to fix local output (#92) * Update ci-develop.yaml * Fix default doublet config (#82) * use seed and fix config storage * use scDblFinder to set default threshold * remove gitignore * remove debug * change boolean logic * Rm auto and enabled (#88) * switch from orig.ident to samples (#95) * Move to multipart upload (#94) * Move to multipart upload * Add print * Fix progress-bar * hardcode DEBUG_PATH (#93) * hardcode DEBUG_PATH * fix function name * Remove hardcoded 10x (it is now hardcoded in the ui only) (#96) Co-authored-by: Martin Fosco <[email protected]> * re-enable per sample config (#97) * added r.utils dependency install which is required by seurat-wrappers (#99) * Deprecated change for ci.yaml Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: ogibson-biomage <[email protected]> Co-authored-by: Juanlu <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: seb-mueller <[email protected]> Co-authored-by: Xavier Vergés <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Anugerah Erlaut <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Anugerah Erlaut <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: James Robinson <[email protected]> * done i think (#101) * Add build as a dependency of run (#103) Also add new run-only target * Init R package and test classifier filter (#104) * init r package * ignore debug * remove file names within file * use styler * cleanup * fix Dockerfile * init filter_emptydrops tests * fix introduced error for demo image * add sample aware test * update README; make task names look distinct * update README * update gem2s step messages * add timer to gem2s steps * Enable Slack build failure notifications * Fix defaults (#109) Co-authored-by: Oliver Gibson <[email protected]> * Unlog y values (#112) * unlog cellsizedistribution plot y values * switch plot numbers to improve readibility * Clean pipeline code (#105) * Clean pipeline code * Update pipeline-runner/R/gem2s-6-prepare_experiment.R Co-authored-by: Alex Pickering <[email protected]> * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <[email protected]> * Update pipeline-runner/R/gem2s-helpers.R * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <[email protected]> * fixes * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Alex Pickering <[email protected]> * Init filter high mito and filter gene umi outlier tests (#114) * init filter_gene_umi_outlier tests * init mito filter tests; fix calc_filter_stats * Init cellszdoublet tests (#118) * init filter_gene_umi_outlier tests * init mito filter tests; fix calc_filter_stats * Added tests for cellsize and doublets * export filter_low_cellsize * simplify parts of filter_low_cellsize * restyle and update comment for filter_doublets test * fix filter_doublets documentation Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> * Removed embedding (#107) * Removed embedding * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Fix Co-authored-by: Oliver Gibson <[email protected]> * Fix (#113) * Fix * typo Co-authored-by: Oliver Gibson <[email protected]> * init renv (#110) * try restore remoter docker-compose * Update pipeline-runner/renv.lock.init * Update pipeline-runner/renv.lock * try update all packages * Fix * update README; remove R version * harmony init lockfile * fix Co-authored-by: ogibson-biomage <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> * Move defaultFilterSettings and missing per sample config additions to the pipeline (#119) * Add defaultFilterSettings and per sample config for classifier and mito content * Update gem2s-6-prepare_experiment.R Co-authored-by: Martin Fosco <[email protected]> * init density kneeplot data (#111) * init density kneeplot data * update plot * use unique fdr/ranks * get_bcranks_plot_data works * fix * Fix * Rollback unnecessary empty plot data addition Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * fix empty guidata case Co-authored-by: Juanlu <[email protected]> Co-authored-by: Pol Alvarez <[email protected]> Co-authored-by: ogibson-biomage <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: seb-mueller <[email protected]> Co-authored-by: Xavier Vergés <[email protected]> Co-authored-by: Oliver Gibson <[email protected]> Co-authored-by: Anugerah Erlaut <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Anugerah Erlaut <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: James Robinson <[email protected]> Co-authored-by: StefanBabukov <[email protected]> Co-authored-by: James Robinson <[email protected]> Co-authored-by: Marcell Pek <[email protected]>
[BIOMAGE 2254] - Refactor cellset creation for subset experiment
Background
Link to issue
BIOMAGE-699
Link to staging deployment URL
https://ui-alex-ui331-pipeline98.scp-staging.biomage.net/
Links to any Pull Requests related to this
hms-dbmi-cellenics/ui#331
Anything else the reviewers should know about the changes here
Changes
Code changes
Definition of DONE
Your changes will be ready for merging after each of the steps below have been completed:
Testing
To set up easy local testing with inframock, follow the instructions here: https://github.com/biomage-ltd/inframock
To deploy to the staging environment, follow the instructions here: https://github.com/biomage-ltd/biomage-utils
Documentation updates
Is all relevant documentation updated to reflect the proposed changes in this PR?
Approvers
Just before merging:
unstage
script in here: https://github.com/biomage-ltd/biomage-utils is executed. This script cleans up your deployment to stagingOptional