-
Notifications
You must be signed in to change notification settings - Fork 594
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Simplified genotype likelihood calculation (no change in output) #6351
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you have numbers for the performance here? How big is this code in the profiler before or after? I'm curious.
@@ -661,7 +661,9 @@ GenotypeAlleleCounts copy() { | |||
|
|||
|
|||
public void forEachAlleleIndexAndCount(final IntBiConsumer action) { | |||
new IndexRange(0, distinctAlleleCount).forEach(n -> action.accept(sortedAlleleCounts[2*n], sortedAlleleCounts[2*n+1])); | |||
for (int n = 0; n < distinctAlleleCount; n++) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did this show up in the profiler? I'm generally for optimizing away Java Stream calls wherever it makes a difference.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change was a shot in the dark from before I realized the cache-cleverness of the original code, but it doesn't show up in the profiler. I'm sure it would if the code was an IntStream
, but IndexRange
is just a simple object with two integers -- there's nothing lurking behind it's constructor there way there is for Java streams.
The unit tests are about 5% faster. In practice, this won't affect HaplotyeCaller because the overwhelming majority of the CPU cost of those tests comes from the ploidy = 20, allele count = 6 cases. For anything else the genotyping likelihoods calculation is not only not a bottleneck, it's completely negligible. |
Do we want to try to get this in at some point? @davidbenjamin if you'd like to push a quick rebase, I can try to take a stab at a review, although I'd guess @jamesemery would be better qualified. |
@samuelklee you are my hero. Let me see how hard the rebase is. |
06ff1fa
to
e9d52f1
Compare
@samuelklee It wasn't just a rebase, it was a complete rewrite because the old code had since become completely entangled with DRAGEN code. But I did it! Everything is passing, the code is dramatically simpler, and it's even a bit faster. I have done my best to make a coherent commit history. I would recommend reviewing one commit at a time in side-by-side diff mode. Note that some commits rip out old code and replace it with pseudocode, deferring the new code to a later commit. Other commits tell a story of what all the different caches meant in order to motivate the simpification of later commits. The baroqueness of the old code was motivated by three considerations:
The DRAGEN code relied on these caches in a rather complex way, which fortunately turned out not to be necessary and which could be dramatically simplified. My notes on tracking all the variables from the parent genotype calculator down to the DRAGEN calculator are in this google doc: https://docs.google.com/document/d/1v6s57mUAwfj38nL3VdktjA059kYBkJfokq18IDy79E8/edit?usp=sharing Good luck and don't hesitate to ask me to explain anything. |
* @param k number of successes | ||
* @return the binomial coefficient | ||
*/ | ||
public static long exactBinomialCoefficient(final int n, final int k) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps we should document the reasons we need our own implementation and beef up correctness and tests, or just consider using an available implementation?
Just iterating up over n and k, it appears that this implementation begins to disagree with CombinatoricsUtils.binomialCoefficient and Wolfram Alpha around n = 48:
n=48, k=21, MathUtils.exactBinomialCoefficient=22314239266529, CombinatoricsUtils.binomialCoefficient=22314239266528
n=48, k=22, MathUtils.exactBinomialCoefficient=27385657281649, CombinatoricsUtils.binomialCoefficient=27385657281648
n=48, k=26, MathUtils.exactBinomialCoefficient=27385657281649, CombinatoricsUtils.binomialCoefficient=27385657281648
n=48, k=27, MathUtils.exactBinomialCoefficient=22314239266529, CombinatoricsUtils.binomialCoefficient=22314239266528
n=49, k=18, MathUtils.exactBinomialCoefficient=11554258485617, CombinatoricsUtils.binomialCoefficient=11554258485616
n=49, k=19, MathUtils.exactBinomialCoefficient=18851684897585, CombinatoricsUtils.binomialCoefficient=18851684897584
n=49, k=20, MathUtils.exactBinomialCoefficient=28277527346377, CombinatoricsUtils.binomialCoefficient=28277527346376
n=49, k=21, MathUtils.exactBinomialCoefficient=39049918716426, CombinatoricsUtils.binomialCoefficient=39049918716424
n=49, k=22, MathUtils.exactBinomialCoefficient=49699896548179, CombinatoricsUtils.binomialCoefficient=49699896548176
n=49, k=23, MathUtils.exactBinomialCoefficient=58343356817427, CombinatoricsUtils.binomialCoefficient=58343356817424
n=49, k=24, MathUtils.exactBinomialCoefficient=63205303218877, CombinatoricsUtils.binomialCoefficient=63205303218876
n=49, k=25, MathUtils.exactBinomialCoefficient=63205303218877, CombinatoricsUtils.binomialCoefficient=63205303218876
...
I haven't carefully tracked down the sources of the discrepancies, but just from skimming the CombinatoricsUtils implementation, it looks like there's additional logic to account for possible overflow in intermediate quantities.
Aside from these discrepancies, there are differences in how the implementations handle overflow in the final, returned value. It appears this implementation can silently return incorrect values (e.g., n = 100, k =40 returns -1---and one might naively assume from the bounds in line 650 that we can safely handle this regime), whereas the CombinatoricsUtils implementation throws a MathArithmeticException.
Just skimming the rest of the code in this PR, it looks like this code is only intended to calculate indices into an array? So that might naturally delimit the reasonable use cases. Perhaps we should roll an encapsulated implementation that only accounts for these use cases here, if the CombinatoricsUtils implementation does not suffice? (Note that we already trust CombinatoricsUtils to generate the cached exact factorials.)
How about for other use cases in the GATK that might want to make use of a general MathUtils implementation? In which regions of input space do we really need an exact/cheap/etc. calculation? Is it ever desired behavior to throw an exception, or do we prefer returning an approximate value? And so on.
I'm going to submit this review comment separately so you can go ahead and start thinking about it, if you like---I expect that it will take me a bit of time to become acquainted with both old/new versions of the rest of the code, anyway!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, now that I'm resuming my review, let's just use CombinatoricsUtils unless we care about possible performance differences due to caching of the factorials. It seems like we probably don't, based on the conversation on the main thread.
More generally, it might be nice to quickly run some tests that give us a sanity check on performance differences. Seems like the unit tests aren't too sensitive, but are there existing integration tests that might be better for this? Or maybe we have some high ploidy test data we could run?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this work, @davidbenjamin! Kudos for significantly simplifying the main computation.
Many of the comments are notes to myself while I was working through the PR or are aimed at possible future work. (Also, sorry if some of them seem out of order---might be because I started off going through commit-by-commit, as you suggested.)
I think we might want to add some additional documentation or shuffle around a few more methods while we're here, though---primarily to better delineate the functionality of the GenotypeIndexCalculator/GenotypeLikelihoodCalculator(s) classes, which might better enable that future work!
There are a few comments regarding runtime/memory. More generally it might be nice to run and document a few quick sanity checks on realistic data, if you haven't already. Given the complexity of the old code, it's hard for me to say whether one might obviously expect regressions/improvments. But maintainability has definitely improved!
BTW, I only briefly skimmed the DRAGEN changes, but I did run a few of the exact match integration tests and checked coverage. Hopefully those tests do indeed cover changes, but maybe @jamesemery can take a quick look.
* @param k number of successes | ||
* @return the binomial coefficient | ||
*/ | ||
public static long exactBinomialCoefficient(final int n, final int k) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, now that I'm resuming my review, let's just use CombinatoricsUtils unless we care about possible performance differences due to caching of the factorials. It seems like we probably don't, based on the conversation on the main thread.
More generally, it might be nice to quickly run some tests that give us a sanity check on performance differences. Seems like the unit tests aren't too sensitive, but are there existing integration tests that might be better for this? Or maybe we have some high ploidy test data we could run?
...ava/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeLikelihoodCalculators.java
Outdated
Show resolved
Hide resolved
...ava/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeLikelihoodCalculators.java
Outdated
Show resolved
Hide resolved
src/main/java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeAlleleCounts.java
Outdated
Show resolved
Hide resolved
...java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeLikelihoodCalculator.java
Outdated
Show resolved
Hide resolved
...java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeLikelihoodCalculator.java
Outdated
Show resolved
Hide resolved
...java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeLikelihoodCalculator.java
Outdated
Show resolved
Hide resolved
...java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeLikelihoodCalculator.java
Outdated
Show resolved
Hide resolved
...java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeLikelihoodCalculator.java
Outdated
Show resolved
Hide resolved
...java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeLikelihoodCalculator.java
Outdated
Show resolved
Hide resolved
Things left for later:
|
@samuelklee I wrote some benchmarks for the exact combinatorics and you were right, my optimization was pointless. Although the I have removed this error in judgment. |
Travis reported job failures from build 38110
|
You know, I think I will clean up all the entangled genotype allele count caching and iterating. I have benchmarked pretty thoroughly and discovered that caching |
@samuelklee Now it's back to you. I agreed with and implemented all of your suggestions. |
Thanks @davidbenjamin for essentially refactoring this code three times now! Looking forward to reviewing your latest changes, but it may have to wait until early next week. Apologies for the delay. In the meantime, I see that there is a minor rebase conflict—up to you if you want to address it now, no biggie if you want to wait until after review. |
@davidbenjamin just letting you know---I will try my best to get this back to you before I head out on vacation next week! Hopefully still well before the other delaying factor. |
@davidbenjamin as we just discussed, back from vacation now and modulo other circumstances (which we also just discussed) should be able to get this back to you in a week. If you could squash the commits after the last review, that might actually be a little cleaner for me to review---perhaps comment a copy of the squashed commit messages here if you'd like to keep them for posterity. Thanks! |
@@ -87,8 +86,7 @@ static double hetLogLikelihood(final AlleleFractionGlobalParameters parameters, | |||
- n * log(majorFraction + minorFraction * lambda0RefMinor); | |||
final double refMinorLogLikelihood = logNotPi + logcRefMinor + Gamma.logGamma(rhoRefMinor) - rhoRefMinor * log(tauRefMinor); | |||
|
|||
final double outlierLogLikelihood = logPi + log10ToLog(log10Factorial(a) + log10Factorial(r) - log10Factorial(a + r + 1)); | |||
|
|||
final double outlierLogLikelihood = logPi - FastMath.log(a + r + 1) - CombinatoricsUtils.binomialCoefficientLog(a+r,a); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In principle, I support the removal of log10Factorial
, but also note #7652. Although this expression is mathematically equivalent, numerically, it will result in a slight change in the output of ModelSegments---which has been stable for 3 years now.
Not a problem, but @tmelman should be aware of this potential change and plan accordingly, depending on what order the PRs are merged. And perhaps @droazen can make sure that minor numerical changes are mentioned in the release notes when appropriate.
Minor nitpick: can you fix the whitespace in the arguments of binomialCoefficientLog
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whitespace has been fixed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cmnbroad may also want to be aware of these numerical changes, since I believe he is also seeing changes in his Java 17 branch.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@davidbenjamin sorry for the delays. I tried to take a stab at re-review---just went ahead and pushed my own rebased branch to db_glcalc_rebase
, but I think it's worth doing pushing the rebase here to re-enable tests on GitHub Actions and make any necessary fixes before I go further. Looks like there are some failing tests due to the numerical changes in MathUtils methods. And while I'm in favor of such changes, they might be tricky, so we should make them carefully---see comment about the upcoming PR #7652 as well.
But in general, the further refactorings post-review look much cleaner! So thanks again for revisiting this code so many times.
src/main/java/org/broadinstitute/hellbender/tools/walkers/genotyper/GenotypeAlleleCounts.java
Outdated
Show resolved
Hide resolved
.../org/broadinstitute/hellbender/tools/walkers/genotyper/afcalc/AlleleFrequencyCalculator.java
Show resolved
Hide resolved
src/main/java/org/broadinstitute/hellbender/tools/walkers/mutect/Mutect2Engine.java
Outdated
Show resolved
Hide resolved
src/main/java/org/broadinstitute/hellbender/utils/recalibration/RecalDatum.java
Show resolved
Hide resolved
@davidbenjamin any bandwidth for keeping up momentum on this one? Maybe we can try to get a rebase and activate GitHub Actions so we can see where we're at? |
615489c
to
ee3b494
Compare
Github actions tests reported job failures from actions build 2799863346
|
Github actions tests reported job failures from actions build 2811164603
|
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #6351 +/- ##
===============================================
+ Coverage 86.297% 86.658% +0.361%
+ Complexity 39445 38662 -783
===============================================
Files 2351 2329 -22
Lines 185118 181309 -3809
Branches 20397 19882 -515
===============================================
- Hits 159752 157119 -2633
+ Misses 18185 17210 -975
+ Partials 7181 6980 -201
|
#carrot(HaplotypeCaller CARROT Regression Tests, VariantCallingCarrotOrchestrated.gatk_docker, ) |
Does it work when I type the magical CARROT incantation? |
🥕CARROT🥕 run startedTest: HaplotypeCaller CARROT Regression Tests | Status: buildingRun: HaplotypeCaller CARROT Regression Tests_run_2023-05-12 15:09:42.984220519 UTC Full details
{
"run_id": "2891f1b5-e415-4d62-ab4e-49fba755b096",
"test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
"run_group_ids": [],
"name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-12 15:09:42.984220519 UTC",
"status": "building",
"test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
"test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
"test_wdl_dependencies": null,
"test_wdl_dependencies_hash": null,
"eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
"eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
"eval_wdl_dependencies": null,
"eval_wdl_dependencies_hash": null,
"test_input": {
"VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
"VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
"VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
"VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
"VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
"VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
"VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
"VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
"VariantCallingCarrotOrchestrated.contamination": 0.0,
"VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
"VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
"VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
"VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
"VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:2022-03-04-4.2.5.0-9-gb097f75c5-NIGHTLY-SNAPSHOT",
"VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|d791874a0490fec31b2587c06ad6c2f2ad397000",
"VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
"VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
"VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
},
"test_options": {
"read_from_cache": false
},
"eval_input": {
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
"BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
"BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
],
"BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
"HCR",
"LCR"
]
},
"eval_options": {
"read_from_cache": false
},
"test_cromwell_job_id": null,
"eval_cromwell_job_id": null,
"created_at": "2023-05-12T15:09:42.984289",
"created_by": null,
"finished_at": null,
"results": null,
"errors": null
}
|
It does! |
🥕CARROT🥕 run finishedTest: HaplotypeCaller CARROT Regression Tests | Status: succeededRun: HaplotypeCaller CARROT Regression Tests_run_2023-05-12 15:09:42.984220519 UTC Results
Full details
{
"run_id": "2891f1b5-e415-4d62-ab4e-49fba755b096",
"test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
"run_group_ids": [],
"name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-12 15:09:42.984220519 UTC",
"status": "succeeded",
"test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
"test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
"test_wdl_dependencies": null,
"test_wdl_dependencies_hash": null,
"eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
"eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
"eval_wdl_dependencies": null,
"eval_wdl_dependencies_hash": null,
"test_input": {
"VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
"VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
"VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
"VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
"VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
"VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
"VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
"VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
"VariantCallingCarrotOrchestrated.contamination": 0.0,
"VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
"VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
"VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
"VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
"VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:2022-03-04-4.2.5.0-9-gb097f75c5-NIGHTLY-SNAPSHOT",
"VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|d791874a0490fec31b2587c06ad6c2f2ad397000",
"VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
"VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
"VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
},
"test_options": {
"read_from_cache": false
},
"eval_input": {
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
"BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
"BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
],
"BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
"HCR",
"LCR"
]
},
"eval_options": {
"read_from_cache": false
},
"test_cromwell_job_id": "b7d06271-38b7-47d2-9d7c-af5543460de9",
"eval_cromwell_job_id": "beb77715-227e-4dbd-803f-4458c83607c8",
"created_at": "2023-05-12T15:09:42.984289",
"created_by": null,
"finished_at": "2023-05-13T01:07:11.594",
"results": {
"CHM controlHCprocesshours": "79.42513333333335",
"CHM controlHCsystemhours": "0.15543611111111108",
"CHM controlHCwallclockhours": "56.046666666666674",
"CHM controlHCwallclockmax": "3.0881333333333334",
"CHM controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-CHMSampleHeadToHead/BenchmarkComparison/f1b0b4cf-1a3f-47b3-84fa-529f118419ce/call-CONTROLRuntimeTask/monitoring.pdf",
"CHM controlindelF1Score": "0.8724",
"CHM controlindelPrecision": "0.8814",
"CHM controlsnpF1Score": "0.9784",
"CHM controlsnpPrecision": "0.9706",
"CHM controlsnpRecall": "0.9863",
"CHM controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-CHMSampleHeadToHead/BenchmarkComparison/f1b0b4cf-1a3f-47b3-84fa-529f118419ce/call-BenchmarkVCFControlSample/Benchmark/fb68536c-eb99-4d0d-a5c3-4f5accf94546/call-CombineSummaries/summary.csv",
"CHM evalHCprocesshours": "84.94748611111112",
"CHM evalHCsystemhours": "0.19002777777777768",
"CHM evalHCwallclockhours": "61.06326111111111",
"CHM evalHCwallclockmax": "3.2047833333333333",
"CHM evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-CHMSampleHeadToHead/BenchmarkComparison/f1b0b4cf-1a3f-47b3-84fa-529f118419ce/call-EVALRuntimeTask/monitoring.pdf",
"CHM evalindelF1Score": "0.8724",
"CHM evalindelPrecision": "0.8814",
"CHM evalsnpF1Score": "0.9784",
"CHM evalsnpPrecision": "0.9706",
"CHM evalsnpRecall": "0.9863",
"CHM evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-CHMSampleHeadToHead/BenchmarkComparison/f1b0b4cf-1a3f-47b3-84fa-529f118419ce/call-BenchmarkVCFTestSample/Benchmark/4353eabb-b85f-4cce-a275-4dba68f9d644/call-CombineSummaries/summary.csv",
"EXOME1 controlindelF1Score": "0.727",
"EXOME1 controlindelPrecision": "0.632",
"EXOME1 controlsnpF1Score": "0.9878",
"EXOME1 controlsnpPrecision": "0.9815",
"EXOME1 controlsnpRecall": "0.9941",
"EXOME1 controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-EXOME1SampleHeadToHead/BenchmarkComparison/bec4eb1e-a99f-4e27-a91c-a56274c3824a/call-BenchmarkVCFControlSample/Benchmark/4d248a53-86e2-46f5-9d28-3364c82b9d0c/call-CombineSummaries/summary.csv",
"EXOME1 evalindelF1Score": "0.727",
"EXOME1 evalindelPrecision": "0.632",
"EXOME1 evalsnpF1Score": "0.9878",
"EXOME1 evalsnpPrecision": "0.9815",
"EXOME1 evalsnpRecall": "0.9941",
"EXOME1 evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-EXOME1SampleHeadToHead/BenchmarkComparison/bec4eb1e-a99f-4e27-a91c-a56274c3824a/call-BenchmarkVCFTestSample/Benchmark/fcd023fe-e278-475d-8fce-613b57518972/call-CombineSummaries/summary.csv",
"NIST controlHCprocesshours": "96.55857222222222",
"NIST controlHCsystemhours": "0.1707444444444444",
"NIST controlHCwallclockhours": "69.28645",
"NIST controlHCwallclockmax": "3.8631972222222224",
"NIST controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-NISTSampleHeadToHead/BenchmarkComparison/243c7bf2-b0d7-48ed-acd0-e2ebd74b9fd3/call-CONTROLRuntimeTask/monitoring.pdf",
"NIST controlindelF1Score": "0.9902",
"NIST controlindelPrecision": "0.9903",
"NIST controlsnpF1Score": "0.9899",
"NIST controlsnpPrecision": "0.9887",
"NIST controlsnpRecall": "0.9911",
"NIST controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-NISTSampleHeadToHead/BenchmarkComparison/243c7bf2-b0d7-48ed-acd0-e2ebd74b9fd3/call-BenchmarkVCFControlSample/Benchmark/135b02c2-d7c5-4fd2-9cc5-cdeeed953bbc/call-CombineSummaries/summary.csv",
"NIST evalHCprocesshours": "103.49634722222224",
"NIST evalHCsystemhours": "0.20633611111111116",
"NIST evalHCwallclockhours": "75.91255833333332",
"NIST evalHCwallclockmax": "3.76305",
"NIST evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-NISTSampleHeadToHead/BenchmarkComparison/243c7bf2-b0d7-48ed-acd0-e2ebd74b9fd3/call-EVALRuntimeTask/monitoring.pdf",
"NIST evalindelF1Score": "0.9902",
"NIST evalindelPrecision": "0.9903",
"NIST evalsnpF1Score": "0.9899",
"NIST evalsnpPrecision": "0.9887",
"NIST evalsnpRecall": "0.9911",
"NIST evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-NISTSampleHeadToHead/BenchmarkComparison/243c7bf2-b0d7-48ed-acd0-e2ebd74b9fd3/call-BenchmarkVCFTestSample/Benchmark/ad8885d7-137d-4645-b37d-f54f8362713d/call-CombineSummaries/summary.csv",
"ROC_Plots_Reported": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/beb77715-227e-4dbd-803f-4458c83607c8/call-CreateHTMLReport/report.html"
},
"errors": null
}
|
🥕CARROT🥕 report map stub finishedfor test HaplotypeCaller CARROT Regression Tests (run: 2891f1b5-e415-4d62-ab4e-49fba755b096)
|
Rebased on top of latest master. Going to run carrot again. |
Github actions tests reported job failures from actions build 4985658896
|
#carrot(HaplotypeCaller CARROT Regression Tests, VariantCallingCarrotOrchestrated.gatk_docker, ) |
🥕CARROT🥕 run startedTest: HaplotypeCaller CARROT Regression Tests | Status: buildingRun: HaplotypeCaller CARROT Regression Tests_run_2023-05-15 22:38:43.733246263 UTC Full details
{
"run_id": "1bf5817a-d896-41de-b193-4a53a1264470",
"test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
"run_group_ids": [],
"name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-15 22:38:43.733246263 UTC",
"status": "building",
"test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
"test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
"test_wdl_dependencies": null,
"test_wdl_dependencies_hash": null,
"eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
"eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
"eval_wdl_dependencies": null,
"eval_wdl_dependencies_hash": null,
"test_input": {
"VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
"VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
"VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
"VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
"VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
"VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
"VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
"VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
"VariantCallingCarrotOrchestrated.contamination": 0.0,
"VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
"VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
"VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
"VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
"VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:2022-03-04-4.2.5.0-9-gb097f75c5-NIGHTLY-SNAPSHOT",
"VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|fde1064710b236c7cdab83248817093046494660",
"VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
"VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
"VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
},
"test_options": {
"read_from_cache": false
},
"eval_input": {
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
"BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
"BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
],
"BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
"HCR",
"LCR"
]
},
"eval_options": {
"read_from_cache": false
},
"test_cromwell_job_id": null,
"eval_cromwell_job_id": null,
"created_at": "2023-05-15T22:38:43.733338",
"created_by": null,
"finished_at": null,
"results": null,
"errors": null
}
|
🥕CARROT🥕 run finishedTest: HaplotypeCaller CARROT Regression Tests | Status: succeededRun: HaplotypeCaller CARROT Regression Tests_run_2023-05-15 22:38:43.733246263 UTC Results
Full details
{
"run_id": "1bf5817a-d896-41de-b193-4a53a1264470",
"test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
"run_group_ids": [],
"name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-15 22:38:43.733246263 UTC",
"status": "succeeded",
"test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
"test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
"test_wdl_dependencies": null,
"test_wdl_dependencies_hash": null,
"eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
"eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
"eval_wdl_dependencies": null,
"eval_wdl_dependencies_hash": null,
"test_input": {
"VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
"VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
"VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
"VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
"VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
"VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
"VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
"VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
"VariantCallingCarrotOrchestrated.contamination": 0.0,
"VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
"VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
"VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
"VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
"VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:2022-03-04-4.2.5.0-9-gb097f75c5-NIGHTLY-SNAPSHOT",
"VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|fde1064710b236c7cdab83248817093046494660",
"VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
"VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
"VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
},
"test_options": {
"read_from_cache": false
},
"eval_input": {
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
"BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
"BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
],
"BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
"HCR",
"LCR"
]
},
"eval_options": {
"read_from_cache": false
},
"test_cromwell_job_id": "c18d0f5f-52c9-4e60-92ed-e4b33c6553c7",
"eval_cromwell_job_id": "9c49383b-01a9-4bc0-90fa-cde7e1090a47",
"created_at": "2023-05-15T22:38:43.733338",
"created_by": null,
"finished_at": "2023-05-16T08:30:08.614",
"results": {
"CHM controlHCprocesshours": "78.81892777777776",
"CHM controlHCsystemhours": "0.15627777777777782",
"CHM controlHCwallclockhours": "55.94185833333335",
"CHM controlHCwallclockmax": "3.053286111111111",
"CHM controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-CHMSampleHeadToHead/BenchmarkComparison/deb85607-d693-4232-a4da-0fb88dd29cad/call-CONTROLRuntimeTask/monitoring.pdf",
"CHM controlindelF1Score": "0.8724",
"CHM controlindelPrecision": "0.8814",
"CHM controlsnpF1Score": "0.9784",
"CHM controlsnpPrecision": "0.9706",
"CHM controlsnpRecall": "0.9863",
"CHM controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-CHMSampleHeadToHead/BenchmarkComparison/deb85607-d693-4232-a4da-0fb88dd29cad/call-BenchmarkVCFControlSample/Benchmark/c0877490-fd2d-4f42-bb92-f06210e94d95/call-CombineSummaries/summary.csv",
"CHM evalHCprocesshours": "84.33091111111112",
"CHM evalHCsystemhours": "0.18621944444444444",
"CHM evalHCwallclockhours": "61.43",
"CHM evalHCwallclockmax": "3.073069444444444",
"CHM evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-CHMSampleHeadToHead/BenchmarkComparison/deb85607-d693-4232-a4da-0fb88dd29cad/call-EVALRuntimeTask/monitoring.pdf",
"CHM evalindelF1Score": "0.8724",
"CHM evalindelPrecision": "0.8814",
"CHM evalsnpF1Score": "0.9784",
"CHM evalsnpPrecision": "0.9706",
"CHM evalsnpRecall": "0.9863",
"CHM evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-CHMSampleHeadToHead/BenchmarkComparison/deb85607-d693-4232-a4da-0fb88dd29cad/call-BenchmarkVCFTestSample/Benchmark/a15fdeb6-16e8-48d7-82cb-168726f4dc18/call-CombineSummaries/summary.csv",
"EXOME1 controlindelF1Score": "0.727",
"EXOME1 controlindelPrecision": "0.632",
"EXOME1 controlsnpF1Score": "0.9878",
"EXOME1 controlsnpPrecision": "0.9815",
"EXOME1 controlsnpRecall": "0.9941",
"EXOME1 controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-EXOME1SampleHeadToHead/BenchmarkComparison/75ea4d64-414f-43aa-a8d6-9c34870b1491/call-BenchmarkVCFControlSample/Benchmark/0f001ca8-d7af-4d01-b9ef-d6ddbe35317d/call-CombineSummaries/summary.csv",
"EXOME1 evalindelF1Score": "0.727",
"EXOME1 evalindelPrecision": "0.632",
"EXOME1 evalsnpF1Score": "0.9878",
"EXOME1 evalsnpPrecision": "0.9815",
"EXOME1 evalsnpRecall": "0.9941",
"EXOME1 evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-EXOME1SampleHeadToHead/BenchmarkComparison/75ea4d64-414f-43aa-a8d6-9c34870b1491/call-BenchmarkVCFTestSample/Benchmark/540fbadc-ba57-4012-8ff1-76461ecb7bb3/call-CombineSummaries/summary.csv",
"NIST controlHCprocesshours": "99.85891111111113",
"NIST controlHCsystemhours": "0.17817777777777768",
"NIST controlHCwallclockhours": "70.22329166666665",
"NIST controlHCwallclockmax": "3.8036305555555554",
"NIST controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-NISTSampleHeadToHead/BenchmarkComparison/75625b9d-e48b-4859-803e-58989e3ccf62/call-CONTROLRuntimeTask/monitoring.pdf",
"NIST controlindelF1Score": "0.9902",
"NIST controlindelPrecision": "0.9903",
"NIST controlsnpF1Score": "0.9899",
"NIST controlsnpPrecision": "0.9887",
"NIST controlsnpRecall": "0.9911",
"NIST controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-NISTSampleHeadToHead/BenchmarkComparison/75625b9d-e48b-4859-803e-58989e3ccf62/call-BenchmarkVCFControlSample/Benchmark/21373bda-c620-4200-ad29-1e3886ea52ad/call-CombineSummaries/summary.csv",
"NIST evalHCprocesshours": "104.20126111111112",
"NIST evalHCsystemhours": "0.20587777777777783",
"NIST evalHCwallclockhours": "76.10080000000004",
"NIST evalHCwallclockmax": "3.949438888888889",
"NIST evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-NISTSampleHeadToHead/BenchmarkComparison/75625b9d-e48b-4859-803e-58989e3ccf62/call-EVALRuntimeTask/monitoring.pdf",
"NIST evalindelF1Score": "0.9902",
"NIST evalindelPrecision": "0.9903",
"NIST evalsnpF1Score": "0.9899",
"NIST evalsnpPrecision": "0.9887",
"NIST evalsnpRecall": "0.9911",
"NIST evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-NISTSampleHeadToHead/BenchmarkComparison/75625b9d-e48b-4859-803e-58989e3ccf62/call-BenchmarkVCFTestSample/Benchmark/b91bffd4-8057-453f-a8e2-4767648da91a/call-CombineSummaries/summary.csv",
"ROC_Plots_Reported": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/9c49383b-01a9-4bc0-90fa-cde7e1090a47/call-CreateHTMLReport/report.html"
},
"errors": null
}
|
🥕CARROT🥕 report map stub finishedfor test HaplotypeCaller CARROT Regression Tests (run: 1bf5817a-d896-41de-b193-4a53a1264470)
|
#carrot(HaplotypeCaller CARROT Regression Tests, VariantCallingCarrotOrchestrated.gatk_docker, ) |
🥕CARROT🥕 run startedTest: HaplotypeCaller CARROT Regression Tests | Status: buildingRun: HaplotypeCaller CARROT Regression Tests_run_2023-05-16 17:15:43.799630906 UTC Full details
{
"run_id": "00d212a9-8eac-4575-9cde-b0a911b3c2f2",
"test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
"run_group_ids": [],
"name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-16 17:15:43.799630906 UTC",
"status": "building",
"test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
"test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
"test_wdl_dependencies": null,
"test_wdl_dependencies_hash": null,
"eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
"eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
"eval_wdl_dependencies": null,
"eval_wdl_dependencies_hash": null,
"test_input": {
"VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
"VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
"VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
"VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
"VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
"VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
"VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
"VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
"VariantCallingCarrotOrchestrated.contamination": 0.0,
"VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
"VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
"VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
"VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
"VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:latest",
"VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|fde1064710b236c7cdab83248817093046494660",
"VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
"VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
"VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
},
"test_options": {
"read_from_cache": false
},
"eval_input": {
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
"BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
"BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
],
"BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
"HCR",
"LCR"
]
},
"eval_options": {
"read_from_cache": false
},
"test_cromwell_job_id": null,
"eval_cromwell_job_id": null,
"created_at": "2023-05-16T17:15:43.799702",
"created_by": null,
"finished_at": null,
"results": null,
"errors": null
}
|
🥕CARROT🥕 run finishedTest: HaplotypeCaller CARROT Regression Tests | Status: succeededRun: HaplotypeCaller CARROT Regression Tests_run_2023-05-16 17:15:43.799630906 UTC Results
Full details
{
"run_id": "00d212a9-8eac-4575-9cde-b0a911b3c2f2",
"test_id": "c3de522b-7ce5-4a51-8b57-1cea628dd93a",
"run_group_ids": [],
"name": "HaplotypeCaller CARROT Regression Tests_run_2023-05-16 17:15:43.799630906 UTC",
"status": "succeeded",
"test_wdl": "gs://dsp-methods-carrot-data/wdl-prod/8fce9006-acbf-48ed-984a-2ec988d82eea/test.wdl",
"test_wdl_hash": "272dc41890e3710cc96c32af03df25065cc4aa9dc389e3c2473bddba7be237db3e0698c15ef287c4619cff83e9b2e8e5e0a486eb4534658604e4bb312f308611",
"test_wdl_dependencies": null,
"test_wdl_dependencies_hash": null,
"eval_wdl": "gs://dsp-methods-carrot-data/wdl-prod/7e3704ce-f26c-4465-a6ab-f64faca619f4/eval.wdl",
"eval_wdl_hash": "8cecc1e6a3ade904ed3bfaae834df6aeac9b50fbc08966557f9e0c1628058b2c64d080f78d0cad222b4b02400db47d540d3a1bcdb8275c475b49a027bf330605",
"eval_wdl_dependencies": null,
"eval_wdl_dependencies_hash": null,
"test_input": {
"VariantCallingCarrotOrchestrated.CHM_base_file_name": "CHM113",
"VariantCallingCarrotOrchestrated.CHM_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.CHM_contamination": 0.0,
"VariantCallingCarrotOrchestrated.CHM_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm1_chm13_hiseqx_sm_hf3mo.bam",
"VariantCallingCarrotOrchestrated.CHM_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.NIST_base_file_name": "NA24385",
"VariantCallingCarrotOrchestrated.NIST_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/wgs_calling_regions.hg38.interval_list",
"VariantCallingCarrotOrchestrated.NIST_contamination": 0.0383312,
"VariantCallingCarrotOrchestrated.NIST_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bam",
"VariantCallingCarrotOrchestrated.NIST_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA24385_NA24385_O1D1_SM-G947H_v1_NS.bai",
"VariantCallingCarrotOrchestrated.agg_preemptible_tries": 3,
"VariantCallingCarrotOrchestrated.break_bands_at_multiples_of": 100000,
"VariantCallingCarrotOrchestrated.contamination": 0.0,
"VariantCallingCarrotOrchestrated.exome1_base_file_name": "NA12878Exome1",
"VariantCallingCarrotOrchestrated.exome1_calling_interval_list": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/exome_calling_regions.v1.interval_list",
"VariantCallingCarrotOrchestrated.exome1_input_bam": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram",
"VariantCallingCarrotOrchestrated.exome1_input_bam_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/NA12878_forPCRplus_1.cram.crai",
"VariantCallingCarrotOrchestrated.gatk_control_docker": "broadinstitute/gatk-nightly:latest",
"VariantCallingCarrotOrchestrated.gatk_docker": "image_build:gatk|fde1064710b236c7cdab83248817093046494660",
"VariantCallingCarrotOrchestrated.haplotype_scatter_count": 50,
"VariantCallingCarrotOrchestrated.monitoring_script": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/cromwell_monitoring_script.sh",
"VariantCallingCarrotOrchestrated.ref_dict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"VariantCallingCarrotOrchestrated.ref_fasta": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"VariantCallingCarrotOrchestrated.ref_fasta_index": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"VariantCallingCarrotOrchestrated.use_gatk3_haplotype_caller": true
},
"test_options": {
"read_from_cache": false
},
"eval_input": {
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.CHM_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.CHM_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcf": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.CHM_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthLabel": "CHM_GRCh38_SYNDIPv20180222",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.CHM_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/chm.full.m38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.twist_exome.interval_list",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.EXOME1_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcf": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.EXOME1_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthLabel": "NA12878_GRCh38_TWISTExome",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.EXOME1_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/GIAB_v3.3.2_NA12878_hg38.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_confidenceInterval": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark_noinconsistent.bed",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlLabel": "CONTROLSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_control_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_control_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_controlVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_control_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalLabel": "TESTSNAPSHOT2018HG002",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalMonitoringExample": "test_output:VariantCallingCarrotOrchestrated.NIST_representative_benchmarking",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalRuntimeSummaries": "test_output:VariantCallingCarrotOrchestrated.NIST_output_runtimes",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcf": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_evalVcfIndex": "test_output:VariantCallingCarrotOrchestrated.NIST_output_vcf_index",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthLabel": "HG002_GRCh38_GIAB",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcf": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz",
"BenchmarkVCFsHeadToHeadOrchestrated.NIST_truthVcfIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HG002_GRCh38_GIAB_1_22_v4.2.1_benchmark.broad-header.vcf.gz.tbi",
"BenchmarkVCFsHeadToHeadOrchestrated.hapMap": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.haplotype_database.txt",
"BenchmarkVCFsHeadToHeadOrchestrated.refDict": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.dict",
"BenchmarkVCFsHeadToHeadOrchestrated.refIndex": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta.fai",
"BenchmarkVCFsHeadToHeadOrchestrated.reference": "gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/Homo_sapiens_assembly38.fasta",
"BenchmarkVCFsHeadToHeadOrchestrated.referenceVersion": "HG38",
"BenchmarkVCFsHeadToHeadOrchestrated.stratIntervals": [
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/HCR_hg38.bed",
"gs://dsp-methods-carrot-data/test_data/haplotypecaller_tests/LCR_Hg38.interval_list"
],
"BenchmarkVCFsHeadToHeadOrchestrated.stratLabels": [
"HCR",
"LCR"
]
},
"eval_options": {
"read_from_cache": false
},
"test_cromwell_job_id": "b9fadac2-4e94-424f-a397-004684d1e51e",
"eval_cromwell_job_id": "acc9e2ac-b10a-4d6a-b586-cd3e47f04e41",
"created_at": "2023-05-16T17:15:43.799702",
"created_by": null,
"finished_at": "2023-05-17T02:34:53.616",
"results": {
"CHM controlHCprocesshours": "84.8981027777778",
"CHM controlHCsystemhours": "0.19177500000000003",
"CHM controlHCwallclockhours": "60.16600277777776",
"CHM controlHCwallclockmax": "3.0439777777777777",
"CHM controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-CHMSampleHeadToHead/BenchmarkComparison/1731c546-7466-4adf-9790-3f99d07df05b/call-CONTROLRuntimeTask/monitoring.pdf",
"CHM controlindelF1Score": "0.8724",
"CHM controlindelPrecision": "0.8814",
"CHM controlsnpF1Score": "0.9784",
"CHM controlsnpPrecision": "0.9706",
"CHM controlsnpRecall": "0.9863",
"CHM controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-CHMSampleHeadToHead/BenchmarkComparison/1731c546-7466-4adf-9790-3f99d07df05b/call-BenchmarkVCFControlSample/Benchmark/669edf6c-76a1-4d82-8cf7-5cd104df2496/call-CombineSummaries/summary.csv",
"CHM evalHCprocesshours": "83.2423166666667",
"CHM evalHCsystemhours": "0.18843333333333337",
"CHM evalHCwallclockhours": "61.06540555555557",
"CHM evalHCwallclockmax": "3.1854916666666666",
"CHM evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-CHMSampleHeadToHead/BenchmarkComparison/1731c546-7466-4adf-9790-3f99d07df05b/call-EVALRuntimeTask/monitoring.pdf",
"CHM evalindelF1Score": "0.8724",
"CHM evalindelPrecision": "0.8814",
"CHM evalsnpF1Score": "0.9784",
"CHM evalsnpPrecision": "0.9706",
"CHM evalsnpRecall": "0.9863",
"CHM evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-CHMSampleHeadToHead/BenchmarkComparison/1731c546-7466-4adf-9790-3f99d07df05b/call-BenchmarkVCFTestSample/Benchmark/8e83736f-3023-4bee-9c42-36c836b75297/call-CombineSummaries/summary.csv",
"EXOME1 controlindelF1Score": "0.727",
"EXOME1 controlindelPrecision": "0.632",
"EXOME1 controlsnpF1Score": "0.9878",
"EXOME1 controlsnpPrecision": "0.9815",
"EXOME1 controlsnpRecall": "0.9941",
"EXOME1 controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-EXOME1SampleHeadToHead/BenchmarkComparison/a98aa003-bddd-492a-8691-dfa50191e2c6/call-BenchmarkVCFControlSample/Benchmark/a332ee8c-3de3-4a6f-b1de-7b273c094e84/call-CombineSummaries/summary.csv",
"EXOME1 evalindelF1Score": "0.727",
"EXOME1 evalindelPrecision": "0.632",
"EXOME1 evalsnpF1Score": "0.9878",
"EXOME1 evalsnpPrecision": "0.9815",
"EXOME1 evalsnpRecall": "0.9941",
"EXOME1 evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-EXOME1SampleHeadToHead/BenchmarkComparison/a98aa003-bddd-492a-8691-dfa50191e2c6/call-BenchmarkVCFTestSample/Benchmark/269b49fd-36aa-4381-a08a-a3f2a4586967/call-CombineSummaries/summary.csv",
"NIST controlHCprocesshours": "103.49216944444444",
"NIST controlHCsystemhours": "0.21042500000000003",
"NIST controlHCwallclockhours": "74.8884888888889",
"NIST controlHCwallclockmax": "3.995058333333333",
"NIST controlMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-NISTSampleHeadToHead/BenchmarkComparison/56974c24-19c8-4f87-b7b1-b71028109732/call-CONTROLRuntimeTask/monitoring.pdf",
"NIST controlindelF1Score": "0.9902",
"NIST controlindelPrecision": "0.9903",
"NIST controlsnpF1Score": "0.9899",
"NIST controlsnpPrecision": "0.9887",
"NIST controlsnpRecall": "0.9911",
"NIST controlsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-NISTSampleHeadToHead/BenchmarkComparison/56974c24-19c8-4f87-b7b1-b71028109732/call-BenchmarkVCFControlSample/Benchmark/670f9cb4-5bb0-48e2-95c9-15a2e1ae7dee/call-CombineSummaries/summary.csv",
"NIST evalHCprocesshours": "103.23083611111107",
"NIST evalHCsystemhours": "0.2083694444444444",
"NIST evalHCwallclockhours": "76.16374166666664",
"NIST evalHCwallclockmax": "3.743883333333333",
"NIST evalMonitoringLogs": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-NISTSampleHeadToHead/BenchmarkComparison/56974c24-19c8-4f87-b7b1-b71028109732/call-EVALRuntimeTask/monitoring.pdf",
"NIST evalindelF1Score": "0.9902",
"NIST evalindelPrecision": "0.9903",
"NIST evalsnpF1Score": "0.9899",
"NIST evalsnpPrecision": "0.9887",
"NIST evalsnpRecall": "0.9911",
"NIST evalsummary": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-NISTSampleHeadToHead/BenchmarkComparison/56974c24-19c8-4f87-b7b1-b71028109732/call-BenchmarkVCFTestSample/Benchmark/b84fd1b7-a21e-4098-aeaf-05de3b35b2df/call-CombineSummaries/summary.csv",
"ROC_Plots_Reported": "gs://dsde-methods-carrot-prod-cromwell/BenchmarkVCFsHeadToHeadOrchestrated/acc9e2ac-b10a-4d6a-b586-cd3e47f04e41/call-CreateHTMLReport/report.html"
},
"errors": null
}
|
🥕CARROT🥕 report map stub finishedfor test HaplotypeCaller CARROT Regression Tests (run: 00d212a9-8eac-4575-9cde-b0a911b3c2f2)
|
Woohoo, eval was faster than control!!!!!!!!!!!!!!!! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With that fixed, my objections have dropped. 👍
Terrific! I'll merge once the bucket migration error gets resolved and that last failing test passes. |
Just curious, what resolved the memory/runtime issues? Apologies if I missed it somewhere above! |
@samuelklee The performance issues were a mirage. It turned out that there was a misconfiguration in the Carrot tests so it was pegging the "control" version to an older GATK release from a year ago. There was a slight regression in the past year the we haven't caught but that is a separate discussion. |
Github actions tests reported job failures from actions build 5010565729
|
The failing test is unrelated to this branch, and I have secured permission to merge. Here we go. . . |
@jamesemery Could you review this? I think you may appreciate it. It took several tries, but I was finally able to write a stripped-down version of the code that actually slightly outperforms the old version. What I realized after a lot of profiling the old code and various failed rewrites was that cache-friendliness is the critical thing here. It turns out that this can be achieved without too many buffers, without precomputing the log frequencies, and without storing 2D and 3D arrays as flattened 1D arrays.