Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azurized wdls #1239

Merged
merged 211 commits into from
Jul 16, 2024
Merged

Azurized wdls #1239

merged 211 commits into from
Jul 16, 2024

Conversation

nikellepetrillo
Copy link
Contributor

@nikellepetrillo nikellepetrillo commented Mar 14, 2024

Description

Summary of changes:

  1. Azurize Optimus pipeline (azurize optimus  #1228)
  2. PD-2514: Azurize Multiome pipeline (Ph pd 2514 multiome on terra #1237)
  3. PD-2516: Azurize PairedTag pipeline (PD-2516: Update Paired-Tag to run in Azure and GCP #1212)
  4. PD-2594: Azurize snm3C pipeline Js pd 2594 snm3c runs on to a #1267
  5. PD-2625: Azurize VariantCalling pipeline

Checklist

If you can answer "yes" to the following items, please add a checkmark next to the appropriate checklist item(s) and notify our WARP documentation team by tagging either @ekiernan or @kayleemathews in a comment on this PR.

  • Did you add inputs, outputs, or tasks to a workflow?
  • Did you modify, delete or move: file paths, file names, input names, output names, or task names?
  • If you made a changelog update, did you update the pipeline version number?

nikellepetrillo and others added 3 commits March 1, 2024 10:43
* add logic to choose which docker

* fix param_meta and import

* add cloud provider to checkinput

* handle hard coded white list paths in CheckInputs.wdl

* last few dockers

* last few dockers

* last few dockers

* change error msg

* use ubuntu image

* use ubuntu image

* change whitelists

* point to azure public whitelists

* add sas token

* echo whitelist

* echo whitelist

* testing for coa

* testing for coa

* change back to terra buckets for whitelists

* change whitelists to point at public azure bucket

* files to strings

* print statemtns to checkinputs

* string to files

* change to terra bucket paths

* strings not files

* append sas token

* append sas token

* append sas  and use strings

* back to bucket urls

* back to bucket urls

* use google cloud urls

* using public urls

* trying to export sas_token

* trying to export sas_token

* trying to export sas_token

* terra on gcp

* update azure whitelist files

* changelogs

* changelogs

* changelogs

* changelogs

* fix some inputs

* fix some inputs

* fix some inputs

* fix some inputs

* update optimus dockers

* warp_tools_docker_path for staralign

* stop using ice lake as default

* update pipeline docs

* 2 threads

* counting mode

* changelogs

---------

Co-authored-by: phendriksen100 <[email protected]>
Co-authored-by: kayleemathews <[email protected]>
* ph logic to pass in docker images based on cloud provider

* determine which whitelist files to use

* update tests

* add parameter metadata

* add error handling in atac

* fix comment

* PR comments

* update image and add utils

* add import

---------

Co-authored-by: phendriksen100 <[email protected]>
Copy link

Remember to squash merge!

Copy link

Remember to squash merge!

Copy link

Remember to squash merge!

Copy link

Remember to squash merge!

Copy link

Remember to squash merge!

@nikellepetrillo
Copy link
Contributor Author

nikellepetrillo commented Mar 15, 2024

optimus plumbing is failing with nondeterminism in empty drops. when using the emptyDrops comparison notebook (https://app.terra.bio/#workspaces/warp-pipelines/Multiome-test-workspace/analysis/launch/emptydrops_comparison.ipynb) the spearmen correlation between test/truth h5ad files was .99

scipy.stats.spearmanr(df.emptydrops_PValue_truth, df.emptydrops_PValue, axis=0, nan_policy='omit')

Copy link

Remember to squash merge!

Copy link

Remember to squash merge!

Copy link

Remember to squash merge!

Copy link

Remember to squash merge!

Copy link

Remember to squash merge!

@kayleemathews
Copy link
Contributor

I think there are probably some additional documentation updates that need to be made here. I can probably take a look either this afternoon or tomorrow morning.

Copy link

Remember to squash merge!

Copy link

Remember to squash merge!

Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
ExternalExomeReprocessing.wdl and ExternalExomeReprocessing.changelog.md do not have matching versions
Some WDLs or changelog files need updating. See output for details.
validation_failed

Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

Copy link

Remember to squash merge!

Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
ExternalExomeReprocessing.wdl and ExternalExomeReprocessing.changelog.md do not have matching versions
Some WDLs or changelog files need updating. See output for details.
validation_failed

Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

Copy link

Remember to squash merge!

Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

Copy link

Remember to squash merge!

Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

Copy link

Remember to squash merge!

Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

Copy link
Contributor

@kevinpalis kevinpalis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Largest PR I've seen in a while. LGTM.

Copy link

Remember to squash merge!

Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

@nikellepetrillo nikellepetrillo merged commit 9942eed into develop Jul 16, 2024
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants