azurize optimus #1228
Merged
azurize optimus #1228
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
khajoue2
approved these changes
Mar 1, 2024
|
This will require some minor doc updates @nikellepetrillo |
There was a problem hiding this comment.
@nikellepetrillo I made some comments about places that documentation needs to be updated for this PR. I know it isn't going into develop yet, so it isn't urgent, but I'll try to get to these today or tomorrow.
...s/skylab/smartseq2_single_nucleus_multisample/MultiSampleSmartSeq2SingleNucleus.changelog.md
Outdated
Show resolved
Hide resolved
pipelines/skylab/smartseq2_single_nucleus_multisample/MultiSampleSmartSeq2SingleNucleus.wdl
Show resolved
Hide resolved
|
retest this please |
…into np_pd-2368_optimus_runs_on_ToA
|
Optimus plumbing is failing in CompareH5ADOptimus. After comparing the h5ad files, it appears the differences are from EmptyDrops (nondeterminism). After plotting the cols/obs of the h5ads, the concordance between test and truth was 99% |
|
retest this please |
nikellepetrillo
added a commit
that referenced
this pull request
Jul 16, 2024
* small change * azurize optimus (#1228) * add logic to choose which docker * fix param_meta and import * add cloud provider to checkinput * handle hard coded white list paths in CheckInputs.wdl * last few dockers * last few dockers * last few dockers * change error msg * use ubuntu image * use ubuntu image * change whitelists * point to azure public whitelists * add sas token * echo whitelist * echo whitelist * testing for coa * testing for coa * change back to terra buckets for whitelists * change whitelists to point at public azure bucket * files to strings * print statemtns to checkinputs * string to files * change to terra bucket paths * strings not files * append sas token * append sas token * append sas and use strings * back to bucket urls * back to bucket urls * use google cloud urls * using public urls * trying to export sas_token * trying to export sas_token * trying to export sas_token * terra on gcp * update azure whitelist files * changelogs * changelogs * changelogs * changelogs * fix some inputs * fix some inputs * fix some inputs * fix some inputs * update optimus dockers * warp_tools_docker_path for staralign * stop using ice lake as default * update pipeline docs * 2 threads * counting mode * changelogs --------- Co-authored-by: phendriksen100 <[email protected]> Co-authored-by: kayleemathews <[email protected]> * Ph pd 2514 multiome on terra (#1237) * ph logic to pass in docker images based on cloud provider * determine which whitelist files to use * update tests * add parameter metadata * add error handling in atac * fix comment * PR comments * update image and add utils * add import --------- Co-authored-by: phendriksen100 <[email protected]> * lost the docker_path in joinmultiomebarcode task * update TestMultiome.wdl * update cutadapt wdl * update cutadapt wdl * try to fix changelog * try to fix changelog * remove cloud provider * try adding sas token to azure public bucket * try adding sas token to azure public bucket * try files not strings * try files not strings * remove cromwell root * update docs * quote whitelist * quote whitelist * dirs * dirs * dirs * add quotes to whitelist * mkdir cromwell_root * try vm family * try vm family * Standard_M128s * try not to use cromwell root * try not to use cromwell root * try using logic * try using logic * update snapatac2 docker * remove mkdir cromwell root * snap dpcker * fix snap dpcker * fix starsolo fastq for other flavors of optimus * merge conflicts * merge conflicts * fix StarAlign.wdl * put whitelists in quotes * PD-2516: Update Paired-Tag to run in Azure and GCP (#1212) * PD-2516: Update PairedTag to run in Azure and GCP * json formatting * update file location for GCP vs. Azure and documentation accordingly * merge conflicts after rebase, update pipeline cahngelog and readme version * more fixes after rebase * more fixes after rebase * more fixes after rebase * fix readme * adding sas tokens * fixing womtools error * update pipeline change logs and versions --------- Co-authored-by: npetrill <[email protected]> Co-authored-by: Nikelle Petrillo <[email protected]> * Fk pd 2513 reblock gvcf (#1261) * added azure docker and updated ReblockGVCF to support gcp and azure * updated verification test wdl * updated UltimaGenomicsWholeGenomeGermline wdl and changelog * Updated Changelog on BroadInternalUltimaGenomics * updated tasks to use the new docker * updated additional changelogs * changed path for import utils.wdl * update other wdls because of qc wdl change * update other wdls because of qc wdl change * update test wdls * still need to update many input jsons * Update ExomeGermlineSingleSample.changelog.md * Update UltimaGenomicsWholeGenomeGermline.changelog.md --------- Co-authored-by: npetrill <[email protected]> Co-authored-by: Nikelle Petrillo <[email protected]> * edited wdl and changelog for ToA support * added cloud provider input to test inputs json * added util class to log error if unsupported cloud provider used for cloud_provider input * added cloud provider parameter to test json * refactored task inputs * change to inputs * change to docker path generation * changes made to test wdl * updated reference to docker images for consistency with other azurized wdls * fix to docker image reference * fix to docker prefix call * updated var name * replace hard-coded cromwell_root with variable based on cloud env * wrap param name in quotations * provide absolute path * write out files using ls * add print for working dir * provide absolute paths for demultiplexing * testing * more testing * more testing * put things back * add lots more logging * only run cutadapt * only run cutadapt * write files to current working dir * write files to current working dir * write files to current working dir * add some of demultiplexing steps back in * typo * typo * typo * remove print * uncomment rest of workflow * add working dir to batch subdir * ls cromwell root batch * ls directories to find batch dir * update changelogs * set batch dir * remove extra leading slash * fixing what i messed up in resvoling conflicts * fix batch dir * need to loop through array * add lots of logging to batch logic * add lots of logging to batch logic * change output dir of bams * fix path of fastq * fix path of fastq * fix fastq remove path * add more echo statements * add more echo statements again * trigger update of wdl in workpace * remove some logging * fix logging error syntax * add missing quote * change working dir to batch dir * add ls * version change * fix paths * fix batch dir * fix batch dir * fix directories * moving around inputs * remove working dir from name of tar file * take basenames * recursively ls the root to find the batch dirs * add echo statement * added docker parameter for DragenTasks.CalibrateDragstrModel supporting azure * added docker parameter for Utils.ScatterIntervalList to support azure * added docker parameter for Calling.HaplotypeCaller_GATK4_VCF to support azure * added docker parameter for Calling.HaplotypeCaller_GATK4_VCF to support azure * added docker parameter to support azure * add echo statement * fix batch dir * fix batch dir * fix path to hisat index files * add pipline_inputs subdir to location of index files * add pipline_inputs subdir to location of index files again * fix tar command * some clean up, remove pipeline inputs subdir, edit single end task * copy fa file again * rename some inputs for paired tag demultiplexing * rename some intermediate inputs for paired tag demultiplexing * list cromwell root * fix batch dir * add conditional for input and output bams * fix logic for cloud provider vs. crowell root dir * add ls * fix batch dir in gcp * fix sytaxt error * ugh remove echo statements referencing variable that doesnt exist anymore * attempt to fix syntax error * add missing forward slash * fix directories, add ls commands * add more ls commands * add more ls commands * add more ls commands again * fix fasta file path * changing docker inputs to allow for tests to pass against pipelines that have not yet been azurized * add more logging, put in cromwell root for move rather than local dir * fix to docker task inputs to allow default values * add more logging, add more ls, try tarring files in current dir rather than previous subdir * use correct syntax for verbose untarring * fix to docker version * set correct base dirs that are nested * change to how cromwell root dir is set * correction to docker image being used * add debugging * formatting fix * add logic for base directory based on gcp vs azure * added notes to affected changelogs * updated changelog for WGS Single sample pipeline * updated pipeline version to match changelong * updated pipeline version to match changelong * updated pipeline version to match changelong * updated changelog and pipeline version * updated changelog and pipeline version * updated changelog and pipeline version * updated changelog and pipeline version * reverting snm3c changes * added notes for SlideSeq and changelog for updated tasks * remove logging and debugging statements * just testing * add input to test * changelogs * changelogs * changelogs * Np make vm size an input to multiome (#1289) * add vm size as input to Multiome.wdl * add vm size as input to Multiome.wdl * add new input to overviews * add new input to overviews * add new input to overviews * add new input to overviews * more disk and mem * Azurize Cell Bender in Multiome (#1299) * update docker for Summary_PerCellOutput * changelogs * changelogs * update dockers * more changelogs ugh * more changelogs ugh * more changelogs ugh * made minor updates, not patches * pointing to pinned version of dockers or the sha * extra space * Apply suggestions from code review Co-authored-by: ekiernan <[email protected]> * changelog * doc reformatting * Update pipelines/broad/arrays/imputation/Imputation.changelog.md * Update pipelines/skylab/slideseq/SlideSeq.changelog.md Co-authored-by: ekiernan <[email protected]> --------- Co-authored-by: phendriksen100 <[email protected]> Co-authored-by: kayleemathews <[email protected]> Co-authored-by: Nareh Sahakian <[email protected]> Co-authored-by: Farzaneh Khajouei <[email protected]> Co-authored-by: John Scira <[email protected]> Co-authored-by: John-Scira <[email protected]> Co-authored-by: Nareh Sahakian <[email protected]> Co-authored-by: aawdeh <[email protected]> Co-authored-by: ekiernan <[email protected]> Co-authored-by: ekiernan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Give your PR a concise yet descriptive title.
Please explain the changes you made here.
Explain the motivation for making this change. What existing problem does the pull request solve?
Mention any issues fixed, addressed, or otherwise related to this pull request, including issue numbers or hard links for issues in other repos.
You can delete these instructions once you have written your PR description.
Checklist
If you can answer "yes" to the following items, please add a checkmark next to the appropriate checklist item(s) and notify our WARP documentation team by tagging either @ekiernan or @kayleemathews in a comment on this PR.