Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make ReblockGVCFs more robust to large inputs #1330

Merged
merged 5 commits into from
Jul 16, 2024
Merged

Make ReblockGVCFs more robust to large inputs #1330

merged 5 commits into from
Jul 16, 2024

Conversation

meganshand
Copy link
Contributor

When running ReblockGVCFs with large input occasionally the VM was running out of disk space in the process of converting the input GVCF into an interval list for validation. This fixes that by including the interval list in the disk_size calculation.

Additionally a separate very large sample was running out of memory on the interval list generation step in the validation task. This allows the user to increase the memory of the machine and actually use it both in Validating and converting the GVCF to interval list.

In the next point release of GATK there will be a fix that's already been merged into htsjdk that will fix the requirement for a large amount of memory for the conversion, but for now giving the command more memory will do the trick.

@github-actions GitHub Actions
Copy link

Remember to squash merge!

@github-actions GitHub Actions
Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
WholeGenomeGermlineSingleSample.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeGermline.wdl has not been changed and needs updating
ExomeGermlineSingleSample.wdl has not been changed and needs updating
VariantCalling.wdl has not been changed and needs updating
ReblockGVCF.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeCramOnly.wdl has not been changed and needs updating
IlluminaGenotypingArray.wdl has not been changed and needs updating
BroadInternalUltimaGenomics.wdl has not been changed and needs updating
BroadInternalRNAWithUMIs.wdl has not been changed and needs updating
BroadInternalArrays.wdl has not been changed and needs updating
ExternalWholeGenomeReprocessing.wdl has not been changed and needs updating
ExternalExomeReprocessing.wdl has not been changed and needs updating
WholeGenomeReprocessing.wdl has not been changed and needs updating
ExomeReprocessing.wdl has not been changed and needs updating
CheckFingerprint.wdl has not been changed and needs updating
Arrays.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

@github-actions GitHub Actions
Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
WholeGenomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeGermline.changelog.md has not been changed and needs to be updated
ExomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
VariantCalling.changelog.md has not been changed and needs to be updated
ReblockGVCF.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeCramOnly.changelog.md has not been changed and needs to be updated
IlluminaGenotypingArray.changelog.md has not been changed and needs to be updated
BroadInternalUltimaGenomics.changelog.md has not been changed and needs to be updated
BroadInternalRNAWithUMIs.changelog.md has not been changed and needs to be updated
BroadInternalArrays.changelog.md has not been changed and needs to be updated
ExternalWholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExternalExomeReprocessing.changelog.md has not been changed and needs to be updated
WholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExomeReprocessing.changelog.md has not been changed and needs to be updated
CheckFingerprint.changelog.md has not been changed and needs to be updated
Arrays.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

@github-actions GitHub Actions
Copy link

Remember to squash merge!

@github-actions GitHub Actions
Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
WholeGenomeGermlineSingleSample.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeGermline.wdl has not been changed and needs updating
ExomeGermlineSingleSample.wdl has not been changed and needs updating
VariantCalling.wdl has not been changed and needs updating
ReblockGVCF.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeCramOnly.wdl has not been changed and needs updating
IlluminaGenotypingArray.wdl has not been changed and needs updating
BroadInternalUltimaGenomics.wdl has not been changed and needs updating
BroadInternalRNAWithUMIs.wdl has not been changed and needs updating
BroadInternalArrays.wdl has not been changed and needs updating
ExternalWholeGenomeReprocessing.wdl has not been changed and needs updating
ExternalExomeReprocessing.wdl has not been changed and needs updating
WholeGenomeReprocessing.wdl has not been changed and needs updating
ExomeReprocessing.wdl has not been changed and needs updating
CheckFingerprint.wdl has not been changed and needs updating
Arrays.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

@github-actions GitHub Actions
Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
WholeGenomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeGermline.changelog.md has not been changed and needs to be updated
ExomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
VariantCalling.changelog.md has not been changed and needs to be updated
ReblockGVCF.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeCramOnly.changelog.md has not been changed and needs to be updated
IlluminaGenotypingArray.changelog.md has not been changed and needs to be updated
BroadInternalUltimaGenomics.changelog.md has not been changed and needs to be updated
BroadInternalRNAWithUMIs.changelog.md has not been changed and needs to be updated
BroadInternalArrays.changelog.md has not been changed and needs to be updated
ExternalWholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExternalExomeReprocessing.changelog.md has not been changed and needs to be updated
WholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExomeReprocessing.changelog.md has not been changed and needs to be updated
CheckFingerprint.changelog.md has not been changed and needs to be updated
Arrays.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

@github-actions GitHub Actions
Copy link

Remember to squash merge!

@github-actions GitHub Actions
Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
UltimaGenomicsWholeGenomeGermline.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

@github-actions GitHub Actions
Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
UltimaGenomicsWholeGenomeGermline.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

@github-actions GitHub Actions
Copy link

Remember to squash merge!

@github-actions GitHub Actions
Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

@github-actions GitHub Actions
Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

@github-actions GitHub Actions
Copy link

Remember to squash merge!

@github-actions GitHub Actions
Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

@github-actions GitHub Actions
Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
UltimaGenomicsWholeGenomeGermline.wdl and UltimaGenomicsWholeGenomeGermline.changelog.md do not have matching versions
ExomeGermlineSingleSample.wdl and ExomeGermlineSingleSample.changelog.md do not have matching versions
UltimaGenomicsWholeGenomeCramOnly.wdl and UltimaGenomicsWholeGenomeCramOnly.changelog.md do not have matching versions
IlluminaGenotypingArray.wdl and IlluminaGenotypingArray.changelog.md do not have matching versions
BroadInternalUltimaGenomics.wdl and BroadInternalUltimaGenomics.changelog.md do not have matching versions
BroadInternalRNAWithUMIs.wdl and BroadInternalRNAWithUMIs.changelog.md do not have matching versions
BroadInternalArrays.wdl and BroadInternalArrays.changelog.md do not have matching versions
CheckFingerprint.wdl and CheckFingerprint.changelog.md do not have matching versions
Arrays.wdl and Arrays.changelog.md do not have matching versions
Some WDLs or changelog files need updating. See output for details.
validation_failed

@github-actions GitHub Actions
Copy link

Remember to squash merge!

@github-actions GitHub Actions
Copy link

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

@github-actions GitHub Actions
Copy link

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

@nikellepetrillo nikellepetrillo merged commit 25a9839 into develop Jul 16, 2024
9 of 11 checks passed
@meganshand meganshand deleted the ms_fix_mem_reblock branch July 19, 2024 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@khajoue2 khajoue2 khajoue2 approved these changes

@ekiernan ekiernan ekiernan approved these changes

@fleharty fleharty Awaiting requested review from fleharty

@gbggrant gbggrant Awaiting requested review from gbggrant

@kachulis kachulis Awaiting requested review from kachulis

@samuelklee samuelklee Awaiting requested review from samuelklee

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@meganshand @khajoue2 @ekiernan @nikellepetrillo