GenotypeGVCFs error in 4.2.4.1 java.lang.IllegalStateException #7639
Comments
|
I have the same problem. The error seems to occur when the number of alleles is exactly one greater than the value of I have tried this on a number of different chromosomes (small test cases with 4 Chinook salmon) and always get the error immediately after a warning about having 7 alleles. On one chromosome, an earlier warning about 8 alleles causes no error, UNLESS I set Log from run with value of
|
|
Given the above behavior, and looking at the error message produced, I wonder if it might not be an issue with the GenomicsDB, but rather just a bug in the code that doesn't properly no-call individuals at sites with too many alleles. I wonder that because the error message is emitted at the bottom of this block: and it only gets there for diploids if neither of |
|
If you would like the GenomicsDB for chromosome https://drive.google.com/file/d/1LzZCkWfmNb8IcZpdreaNIxtJ8GQQ-b7g/view?usp=sharing It is 385 Mb, and it has an SHA1 hash (from the Unix |
|
I'm having the same problem, running GenotypeGVCFs 4.2.4.1 following GenomicsDBImport 4.2.4.1 on 3 human WGS samples, NOT using -all-sites (as in original post), and using default If I add So in my case, it is also failing after a site where number of alt alleles (7) is exactly 1 more than --max-alternate-alleles (6), but not always on the first such variant with 7 alt alleles, depending on other command line options. If I revert GenotypeGVCFs to previous versions from 4.1.9.0 to 4.2.4.0 they all work (using the same GVCF database from GenomicsDBImport 4.2.4.1), so this apparent bug is new in GenotypeGVCFs 4.2.4.1. |
|
@nalinigans @mlathara do you have any insights to if this would result from any of the GenomicsDB changes in 4.2.4.1? |
|
I think changes in 4.2.4.1 are exposing an issue, but I think @eriqande is right that the issue isn't with GenomicsDB. The change was that previously GATK wasn't correctly passing the It looks like the AF calculation used to ignore sites if they didn't have likelihoods, but has now been updated slightly to also allow sites with GQ or sites where any alleles are called+nonref+not symbolic. gatk/src/main/java/org/broadinstitute/hellbender/utils/GenotypeUtils.java Lines 139 to 141 in caa48f9 Possibly that condition needs to be tweaked? @ldgauthier might be the best person to ask. As for this working with previous versions, that is only because the |
|
To elaborate & clarify, in my use case: GenomicsDBImport 4.2.4.0 plus GenotypeGVCFs 4.2.4.0: works So I agree the issue appears to be the change to AF calculation in GenotypeGVCFs 4.2.4.1, not the |
|
Thanks Seboyden. That is super helpful. I am just finishing up a long series of GenotypeGVCFs 4.2.4.0 using genomicsDBs from GenomicsDBImport 4.2.4.1, without having passed any |
|
We're also hitting this, and will try to downgrade. However, can anyone from GATK comment on whether it's possible to add a fix to GATK itself? |
|
@eriqande thanks for sharing your data, but I'll need your reference fasta as well to be able to reproduce your error. |
|
Never mind -- refseq to the rescue. I'm working on a patch now. |
|
Oops! Of course! Sorry I didn't include that as well. Glad you were able
to get it off RefSeq.
Thanks so much for looking into this for all of us. Much appreciated!
eric
…On Thu, Jan 27, 2022 at 10:51 AM ldgauthier ***@***.***> wrote:
Never mind -- refseq to the rescue. I'm working on a patch now.
—
Reply to this email directly, view it on GitHub
<#7639 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPQ4JSI4TKOZNLDMPAACQDUYFSYXANCNFSM5L75WXOA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
I havent looked into this in detail, but if the description above is right, would running this with a GenomicsDB data source and "--max-alternate-alleles=1" likely trigger this? The reasoning is that there are probably sites in your existing test data with more than one ALT? |
|
Yep, right now that's what would happen. The previous behavior was that GenomicsDB would return likelihoods for up to 50 alts and then no likelihoods for more than 50 and GenotypeGVCFs I think would subset 7-49 ALTs down to 6 and drop sites with more than 50 ALTs, which is sort of the computational efficiency compromise. I personally don't trust a diploid site with 50 ALTs. I believe the old CombineGVCFs behavior capped the number of likelihoods, so ploidy 1 could have more than 50 ALTs, but higher ploidies would have fewer. So the current plan is to separate the max alts for GGVCFs and GenomicsDB so that the GenomicsDB max alts to combine can be greater and thus convey enough information for GGVCFs to subset down to its max alts, if desired. But if you want to be conservative with memory, you could set them to be equal and low. Should we filter sites with too many alts, which is the recommendation for the GnarlyGenotyper pipeline, rather than drop them, like GGVCFs used to do? |
|
@mlathara The underlying issue here is that in previous versions of GATK, users would run GenotypeGVCFs with --max-alternate-alleles set to some low value such as 6, while GenomicsDB would at the same time run with a limit of 50 alleles. Now that GenomicsDB is respecting the --max-alternate-alleles argument, we're running with the same value (eg., 6) in both GenotypeGVCFs and GenomicsDB and getting these exceptions. We're probably going to need a separate max alts argument specifically for GenomicsDB.... |
|
To reiterate, for me, GenotypeGVCFs 4.2.4.1 gives an As @mlathara stated: "the AF calculation [in GenotypeGVCFs] used to [in 4.2.4.0] ignore sites if they didn't have likelihoods, but has now been updated slightly [in 4.2.4.1] to also allow sites with GQ or sites where any alleles are called+nonref+not symbolic. The stack traces above show the AlleleFrequencyCalculator as the culprit: |
|
Yep, this should fix your issue as well. |
|
We will do a GATK point release within the next few days to patch the GenotypeGVCFs bug, and also add a |
|
Let me paraphrase to make sure I understand: GenotypeGVCFs somehow uses the "extra" PL fields returned by GenomicsDB when they each have different max-alternate-alleles arguments. We want to preserve this because this offers a good balance of compute time vs accuracy. Is that right? So, outlining a few cases:
Am I thinking about that correctly? Do we need to worry about max-genotype-count here? That is also tied together in GenomicsDB and GATK. And that is actually the argument more directly tied to the number of likelihoods... |
|
@mlathara do you mean because of the <NON_REF> allele? As in GenomicsDB max should be at least GGVCFs max plus one for the non-ref that GGVCFs won't output since the GGVCFs arg describes the genotyped output? |
|
@ldgauthier you mean this part?
I just wasn't sure whether that particular permutation makes sense (either from a user perspective, or one that GenotypeGVCFs should support). I suppose as long as the argument description defines the potential interaction between the arguments, all permutations can be supported. |
|
@eriqande your data looks good with the update in #7655 Hopefully we can get a point release out next week, but if you're in a rush you can roll your own jar from that branch. There are two args now (both to be used in the GenotypeGVCFs command). I would suggest setting the |
|
@ldgauthier, would it be possible to rename Based on @mlathara's comment, do we need to worry about |
|
Fixed in GATK 4.2.5.0 |
|
Hello all. Thank you for this incredibly useful software tome of tools and documentation. I can't imagine progressing far in any meaningful way without your hard work. I am re-opening this thread because I have encountered a problem and this thread seems to be closest to the issue I am experiencing. I am running GATK 4.2.5.0 on a GenomicsDB consisting of ~500 wolf exomes, created using GATK 4.2.2.0. I have tried running GGVCFs with 500gb memory, 600gb memory, and 1.6tb memory. In each case the run stops without issuing a warning or exception at exactly the same spot. I can't run a stack trace as the java process terminates. My exact command is gatk --java-options "-Xmx1600g -DGATK_STACKTRACE_ON_USER_EXCEPTION=true" GenotypeGVCFs This runs perfectly until it reaches the 2 millionth variant mark whereupon everything stops, and all processes are terminated. You will notice this isn't occurring at chr1= ~200k (as in previous reports), but instead on the variants processed = ~2million. It seems odd that previous posts had a similar error (reported alternately as chr position ~200k or 2m). If I try running "SelectVariants" on any interval in the database gatk SelectVariants The process stalls (doesn't terminate) without reporting any variants with the progress meter as below. 18:46:19.529 INFO SelectVariants - Done initializing engine The stack trace on the stalled "SelectVariants" command looks like: "G1 Refine#78" os_prio=0 tid=0x00007ff536ea6000 nid=0x1af0e0 runnable "G1 Refine#79" os_prio=0 tid=0x00007ff536ea8000 nid=0x1af0e1 runnable "G1 Refine#80" os_prio=0 tid=0x00007ff536ea9800 nid=0x1af0e2 runnable "G1 Refine#81" os_prio=0 tid=0x00007ff536eab800 nid=0x1af0e3 runnable "G1 Refine#82" os_prio=0 tid=0x00007ff536ead000 nid=0x1af0e4 runnable "G1 Young RemSet Sampling" os_prio=0 tid=0x00007ff536eaf000 nid=0x1af0e5 runnable JNI global references: 13 Heap This is the screen log for GGVCFs command, although I was hoping by adding the flag "-DGATK_STACKTRACE_ON_USER_EXCEPTION=true" I would get more information to share. As it takes 11 hours to get to the point where the error occurs it has been difficult to trouble shoot, I am hoping that I can fix this without rebuilding everything which is why I decided to write. Thanks for any information or suggestions you may have. Dan Using GATK jar /home/dan_vanderpool/src/gatk-4.2.5.0/gatk-package-4.2.5.0-local.jar ............. 13:59:13.758 INFO ProgressMeter - chr1:93376595 638.0 1968000 3084.5 Chromosome chr1 position 93497885 (TileDB column 93497884) has too many alleles in the combined VCF record : 11 : current limit : 10. Fields, such as PL, with length equal to the number of genot Chromosome chr1 position 93497887 (TileDB column 93497886) has too many alleles in the combined VCF record : 13 : current limit : 10. Fields, such as PL, with length equal to the number of genot Chromosome chr1 position 93497889 (TileDB column 93497888) has too many alleles in the combined VCF record : 16 : current limit : 10. Fields, such as PL, with length equal to the number of genot Chromosome chr1 position 93497891 (TileDB column 93497890) has too many alleles in the combined VCF record : 43 : current limit : 10. Fields, such as PL, with length equal to the number of genot GENOMICSDB_TIMER,GenomicsDB iterator next() timer,Wall-clock time(s),1.009227539,Cpu time(s),1.0086322770000005 ............. GENOMICSDB_TIMER,GenomicsDB iterator next() timer,Wall-clock time(s),0.203489618,Cpu time(s),0.2034794189999999 GENOMICSDB_TIMER,GenomicsDB iterator next() timer,Wall-clock time(s),0.7820505179999996,Cpu time(s),0.7815515469999997 GENOMICSDB_TIMER,GenomicsDB iterator next() timer,Wall-clock time(s),0.7910685349999996,Cpu time(s),0.7911189049999999 |
|
@xylebori There were a couple of issues with the GenomicsDB / GenotypeGVCFs pipeline in the 4.2.5.0 release that will be fixed with the patch in #7670. I'm not certain they are related to the issue you're encountering, but you could try running with that patch (or waiting for 4.2.6.0, which should be out shortly) and see if it helps. If it doesn't help, file a new ticket (rather than commenting on this closed ticket) and we'll take a look. |
A user running GenotypeGVCFs with a GenomicsDB ran into a new issue with 4.2.4.1. They were previously running 4.1.9.0.
Their complete program log is below:
This request was created from a contribution made by Andrius Jonas Dagilis on January 06, 2022 16:24 UTC.
Link: https://gatk.broadinstitute.org/hc/en-us/community/posts/360073212811-GenotypeGVCFs-stalls-while-using-all-sites#community_comment_4414746059419
The text was updated successfully, but these errors were encountered: