* added nhash_id to multiome, optimus, paired-tag, and atac

* changed to null input

* Fix testing wdls for nhash_id

* fixing nhash id errors

* fixed nhash_id examples and verification wdls

* made gex_nash_id optional in STARAlign mergestaroutputs task

* fixing h5ad utils

* fixing nash id python variable in atac wdl

* fixed h5ad variable name in atac fragment file creation

* fixed echo in STARAlign for nhash id

* passing nhash id through pipeline

* updated changelogs

* updated documentation for nhash_id

…neither develop or master (#1329)

* rename some intermediate inputs for paired tag demultiplexing

* remove ss2 test/verify files

* remove ss2 from get_changed_pipeline_worklow_test_args.sh

* small change

* azurize optimus  (#1228)

* add logic to choose which docker

* fix param_meta and import

* add cloud provider to checkinput

* handle hard coded white list paths in CheckInputs.wdl

* last few dockers

* last few dockers

* last few dockers

* change error msg

* use ubuntu image

* use ubuntu image

* change whitelists

* point to azure public whitelists

* add sas token

* echo whitelist

* echo whitelist

* testing for coa

* testing for coa

* change back to terra buckets for whitelists

* change whitelists to point at public azure bucket

* files to strings

* print statemtns to checkinputs

* string to files

* change to terra bucket paths

* strings not files

* append sas token

* append sas token

* append sas  and use strings

* back to bucket urls

* back to bucket urls

* use google cloud urls

* using public urls

* trying to export sas_token

* trying to export sas_token

* trying to export sas_token

* terra on gcp

* update azure whitelist files

* changelogs

* changelogs

* changelogs

* changelogs

* fix some inputs

* fix some inputs

* fix some inputs

* fix some inputs

* update optimus dockers

* warp_tools_docker_path for staralign

* stop using ice lake as default

* update pipeline docs

* 2 threads

* counting mode

* changelogs

---------

Co-authored-by: phendriksen100 <[email protected]>
Co-authored-by: kayleemathews <[email protected]>

* Ph pd 2514 multiome on terra (#1237)

* ph logic to pass in docker images based on cloud provider

* determine which whitelist files to use

* update tests

* add parameter metadata

* add error handling in atac

* fix comment

* PR comments

* update image and add utils

* add import

---------

Co-authored-by: phendriksen100 <[email protected]>

* lost the docker_path in joinmultiomebarcode task

* update TestMultiome.wdl

* update cutadapt wdl

* update cutadapt wdl

* try to fix changelog

* try to fix changelog

* remove cloud provider

* try adding sas token to azure public bucket

* try adding sas token to azure public bucket

* try files not strings

* try files not strings

* remove cromwell root

* update docs

* quote whitelist

* quote whitelist

* dirs

* dirs

* dirs

* add quotes to whitelist

* mkdir cromwell_root

* try vm family

* try vm family

* Standard_M128s

* try not to use cromwell root

* try not to use cromwell root

* try using logic

* try using logic

* update snapatac2 docker

* remove mkdir cromwell root

* snap dpcker

* fix snap dpcker

* fix starsolo fastq for other flavors of optimus

* merge conflicts

* merge conflicts

* fix StarAlign.wdl

* put whitelists in quotes

* PD-2516: Update Paired-Tag to run in Azure and GCP (#1212)

* PD-2516: Update PairedTag to run in Azure and GCP

* json formatting

* update file location for GCP vs. Azure and documentation accordingly

* merge conflicts after rebase, update pipeline cahngelog and readme version

* more fixes after rebase

* more fixes after rebase

* more fixes after rebase

* fix readme

* adding sas tokens

* fixing womtools error

* update pipeline change logs and versions

---------

Co-authored-by: npetrill <[email protected]>
Co-authored-by: Nikelle Petrillo <[email protected]>

* Fk pd 2513 reblock gvcf (#1261)

* added azure docker and updated ReblockGVCF to support gcp and azure

* updated verification test wdl

* updated UltimaGenomicsWholeGenomeGermline wdl and changelog

* Updated Changelog on BroadInternalUltimaGenomics

* updated tasks to use the new docker

* updated additional changelogs

* changed path for import utils.wdl

* update other wdls because of qc wdl change

* update other wdls because of qc wdl change

* update test wdls

* still need to update many input jsons

* Update ExomeGermlineSingleSample.changelog.md

* Update UltimaGenomicsWholeGenomeGermline.changelog.md

---------

Co-authored-by: npetrill <[email protected]>
Co-authored-by: Nikelle Petrillo <[email protected]>

* edited wdl and changelog for ToA support

* added cloud provider input to test inputs json

* added util class to log error if unsupported cloud provider used for cloud_provider input

* added cloud provider parameter to test json

* refactored task inputs

* change to inputs

* change to docker path generation

* changes made to test wdl

* updated reference to docker images for consistency with other azurized wdls

* fix to docker image reference

* fix to docker prefix call

* updated var name

* replace hard-coded cromwell_root with variable based on cloud env

* wrap param name in quotations

* provide absolute path

* write out files using ls

* add print for working dir

* provide absolute paths for demultiplexing

* testing

* more testing

* more testing

* put things back

* add lots more logging

* only run cutadapt

* only run cutadapt

* write files to current working dir

* write files to current working dir

* write files to current working dir

* add some of demultiplexing steps back in

* typo

* typo

* typo

* remove print

* uncomment rest of workflow

* add working dir to batch subdir

* ls cromwell root batch

* ls directories to find batch dir

* update changelogs

* set batch dir

* remove extra leading slash

* fixing what i messed up in resvoling conflicts

* fix batch dir

* need to loop through array

* add lots of logging to batch logic

* add lots of logging to batch logic

* change output dir of bams

* fix path of fastq

* fix path of fastq

* fix fastq remove path

* add more echo statements

* add more echo statements again

* trigger update of wdl in workpace

* remove some logging

* fix logging error syntax

* add missing quote

* change working dir to batch dir

* add ls

* version change

* fix paths

* fix batch dir

* fix batch dir

* fix directories

* moving around inputs

* remove working dir from name of tar file

* take basenames

* recursively ls the root to find the batch dirs

* add echo statement

* added docker parameter for DragenTasks.CalibrateDragstrModel supporting azure

* added docker parameter for Utils.ScatterIntervalList to support azure

* added docker parameter for Calling.HaplotypeCaller_GATK4_VCF to support azure

* added docker parameter for Calling.HaplotypeCaller_GATK4_VCF to support azure

* added docker parameter to support azure

* add echo statement

* fix batch dir

* fix batch dir

* fix path to hisat index files

* add pipline_inputs subdir to location of index files

* add pipline_inputs subdir to location of index files again

* fix tar command

* some clean up, remove pipeline inputs subdir, edit single end task

* copy fa file again

* rename some inputs for paired tag demultiplexing

* rename some intermediate inputs for paired tag demultiplexing

* list cromwell root

* fix batch dir

* add conditional for input and output bams

* fix logic for cloud provider vs. crowell root dir

* add ls

* fix batch dir in gcp

* fix sytaxt error

* ugh remove echo statements referencing variable that doesnt exist anymore

* attempt to fix syntax error

* add missing forward slash

* fix directories, add ls commands

* add more ls commands

* add more ls commands

* add more ls commands again

* fix fasta file path

* changing docker inputs to allow for tests to pass against pipelines that have not yet been azurized

* add more logging, put in cromwell root for move rather than local dir

* fix to docker task inputs to allow default values

* add more logging, add more ls, try tarring files in current dir rather than previous subdir

* use correct syntax for verbose untarring

* fix to docker version

* set correct base dirs that are nested

* change to how cromwell root dir is set

* correction to docker image being used

* add debugging

* formatting fix

* add logic for base directory based on gcp vs azure

* added notes to affected changelogs

* updated changelog for WGS Single sample pipeline

* updated pipeline version to match changelong

* updated pipeline version to match changelong

* updated pipeline version to match changelong

* updated changelog and pipeline version

* updated changelog and pipeline version

* updated changelog and pipeline version

* updated changelog and pipeline version

* reverting snm3c changes

* added notes for SlideSeq and changelog for updated tasks

* remove logging and debugging statements

* just testing

* add input to test

* changelogs

* changelogs

* changelogs

* Np make vm size an input to multiome (#1289)

* add vm size as input to Multiome.wdl

* add vm size as input to Multiome.wdl

* add new input to overviews

* add new input to overviews

* add new input to overviews

* add new input to overviews

* more disk and mem

* Azurize Cell Bender in Multiome (#1299)

* update docker for Summary_PerCellOutput

* changelogs

* changelogs

* update dockers

* more changelogs ugh

* more changelogs ugh

* more changelogs ugh

* made minor updates, not patches

* pointing to pinned version of dockers or the sha

* extra space

* Apply suggestions from code review

Co-authored-by: ekiernan <[email protected]>

* changelog

* doc reformatting

* Update pipelines/broad/arrays/imputation/Imputation.changelog.md

* Update pipelines/skylab/slideseq/SlideSeq.changelog.md

Co-authored-by: ekiernan <[email protected]>

---------

Co-authored-by: phendriksen100 <[email protected]>
Co-authored-by: kayleemathews <[email protected]>
Co-authored-by: Nareh Sahakian <[email protected]>
Co-authored-by: Farzaneh Khajouei <[email protected]>
Co-authored-by: John Scira <[email protected]>
Co-authored-by: John-Scira <[email protected]>
Co-authored-by: Nareh Sahakian <[email protected]>
Co-authored-by: aawdeh <[email protected]>
Co-authored-by: ekiernan <[email protected]>
Co-authored-by: ekiernan <[email protected]>

* fix for memory issue in reblocking

* disk size

* changelogs

* forgot one

* fixing version numbers