-
Notifications
You must be signed in to change notification settings - Fork 165
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Salmon v0.10.0: Error in function boost::math::digamma<double>(double): numeric overflow] #229
Comments
This seems to be a separate issue than the other (and a more informative exception). Once I've resolved the other issue, I would probably try to bug you for a sample that causes this --- though I have a reasonable idea about how to fix it. It would be nice to have the fix for both issues in the same hotfix. To be more specific : this is, as the exception says, a numeric underflow issue when evaluating the digamma function. The solution here is just to bump up the value that is required before evaluating this function. This should be straightforward, but I suspect the issue is also related to this log message:
|
I wondered about that too but other samples gave me 2 and 3 degenerate classes and still passed... |
Right --- such classes should be removed (specifically because they can cause such underflow issues). I suspect that I just need to make the bound for evaluation more conservative. I chose the smallest value that worked on my testing machine, but I imagine that when underflow occurs could be slightly different on different machines. I should just find / choose a stricter bound. |
Think you could tell me which line to edit? Since I can now work on a linux box this issue is now more pressing for me than the other one... |
Yes. There are two places to look. The first is possibly changing this cutoff. The other is putting a guard around this line that only evaluates the |
I was able to fix the other bug you encountered (on OSX) --- it should now be fixed in the develop branch with a clean build. That makes this bug my current top priority. I'd like to fix it so I can merge both changes into master and cut a v0.10.1 release. Let me know the best way I can help address this one. |
I tried wrapping the code around alphaSum but it didn't work. #include <boost/math/special_functions/digamma.hpp>
int main() {
double logNorm = boost::math::digamma(1e-50);
printf("%f\n", logNorm);
return logNorm;
} |
I have a thought --- can you try compiling the above with P.S. nevermind --- doesn't seem to make a difference on my end. |
It also works with -O3. |
I believe that would be the appropriate behavior, yes. However, I'm quite curious now what argument to digamma is causing this as I'm able to go to tiny values of the argument without seeing an exception. |
If you implement the try/catch and print the value in the catch code I might be able to tell you ;-) |
I'll be happy to go ahead an implement this; but the numeric_error exception that is thrown doesn't contain the value that caused the error (just that string that is printed out) |
Ok, I pushed a commit to develop that wraps this in try/catch (and also tries to print out the relevant value of the argument from the context). Please let me know if (1) this averts the uncaught exception and (2) what argument to digamma is triggering this strange behavior :)! |
[2018-05-31 20:01:46.042] [jointLog] [info] Marked 1 weighted equivalence classes as degenerate :-O |
Could it be coming from some other part of the code or some dependency? |
Well, in the other branch, |
A quick grep shows these as the only uses of
|
### salmon (mapping-based) v0.10.1 |
the fact that this |
Also, I assumed (according to the documentation) that these underflow and overflow errors inherit from |
I have added |
Brilliant work ... and incredibly strange. Somehow, either one of the inputs in this sum is infinite, or adding them up leads to an infinity! |
EQVecT i:82396 M:310732 alphaIn:11098.990511 priorAlphas:0.025489 |
Interesting: so here |
EQVecT means we are in the function with |
Yes, so in the loop that computes alphaSum, |
BTW I don't know if it could be related but 67% tests passed, 1 tests failed out of 3 Total Test time (real) = 3.37 sec The following tests FAILED: |
itNum: 0 [2018-05-31 22:48:54.609] [jointLog] [info] finished quantifyLibrary() |
probably unrelated, but also unexpected. Make test on my linux and osx box looks like:
It looks the same on the continuous integration server :
Also, you can look, in the build directory, in the subdirectory |
Ok, so it looks like we complete iteration 0, so that the initial alphas (before we start the VBEM) is OK. But after one call to |
cat Testing/Temporary/LastTestsFailed.log |
ha ... that log isn't particularly interesting --- what happens if you run |
./src/unitTestsAll tests passed (108 assertions in 4 test cases) |
hrmmmmm .... so then the failure is stochastic? |
Well... |
can you try |
$ sudo make install Installation complete. Please ensure the following paths are set properly. Please add /usr/local/bin to your PATH $ make test 67% tests passed, 1 tests failed out of 3 Total Test time (real) = 3.32 sec The following tests FAILED: |
Ok, GMT+2 here, have to go ;-) |
It must be a different |
Of course; thank you so much for your help today! Glad we were able to squash the other bug. I'll think more about this vbOpt bug. If I can find a dataset to run that triggers it, I can probably debug it faster (like the other bug). |
Sure, after all it is quite selfish ;-) I have been trying to generate smaller transcript db and fastq files but to no avail (can't make it fail again) However it looks like your test These two lines result from: groupSize: 2 |
I have replaced |
Just to get more info about exactly the equivalence class causing this problem, can you try to change this definition of verbose{false} to verbose{true} ? https://github.com/COMBINE-lab/salmon/blob/develop/src/CollapsedEMOptimizer.cpp#L376 |
Also, out of curiosity. Does it work if you make |
I'll tell you later but it might: int main() { inf |
It did pass on one of the problematic samples. |
Awesome! Thanks for checking it out. I think having |
These fixes are now live in v0.10.1 --- thanks again! |
Great! Thanks! |
Sorry, me (#228) again...
While most of my samples seem to work on linux, I got an exception for one of them:
Version Info: This is the most recent version of Salmon.
### salmon (mapping-based) v0.10.0
### [ program ] => salmon
### [ command ] => quant
### [ index ] => { salmon010.index.all_combined }
### [ libType ] => { A }
### [ mates1 ] => { R1.fastq.gz }
### [ mates2 ] => { R2.fastq.gz }
### [ posBias ] => { }
### [ gcBias ] => { }
### [ seqBias ] => { }
### [ useVBOpt ] => { }
### [ validateMappings ] => { }
### [ output ] => { processed_salmon0100_k31_allcombined/R }
Logs will be written to processed_salmon0100_k31_allcombined/R/logs
[2018-05-31 16:54:42.310] [jointLog] [info] Fragment incompatibility prior below threshold. Incompatible fragments will be ignored.
[2018-05-31 16:54:42.310] [jointLog] [info] Usage of --validateMappings implies use of range factorization. rangeFactorizationBins is being set to 4
[2018-05-31 16:54:42.310] [jointLog] [info] parsing read library format
[2018-05-31 16:54:42.310] [jointLog] [info] There is 1 library.
[2018-05-31 16:54:42.480] [jointLog] [info] Loading Quasi index
[2018-05-31 16:54:42.501] [jointLog] [info] Loading 32-bit quasi index
[2018-05-31 16:54:42.501] [stderrLog] [info] Loading Suffix Array
[2018-05-31 16:55:01.293] [stderrLog] [info] Loading Transcript Info
[2018-05-31 16:55:06.428] [stderrLog] [info] Loading Rank-Select Bit Array
[2018-05-31 16:55:07.107] [stderrLog] [info] There were 310732 set bits in the bit array
[2018-05-31 16:55:07.158] [stderrLog] [info] Computing transcript lengths
[2018-05-31 16:55:07.159] [stderrLog] [info] Waiting to finish loading hash
[2018-05-31 16:55:25.973] [jointLog] [info] done
[2018-05-31 16:55:25.973] [jointLog] [info] Index contained 310732 targets
[2018-05-31 16:55:25.973] [stderrLog] [info] Done loading index
processed 67500000 fragmentsointLog] [info] Automatically detected most likely library type as IU
hits: 224580543, hits per frag: 3.35031[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 1.04% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.96% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.84% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.96% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.88% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 1.00% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.92% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.84% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.84% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.86% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.94% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.96% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.98% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.92% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.94% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.98% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.92% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 1.04% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 1.02% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.92% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.92% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.94% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.88% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.90% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.94% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.86% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.86% zero probability fragments
[2018-05-31 17:08:09.486] [jointLog] [info] Thread saw mini-batch with a maximum of 0.86% zero probability fragments
[2018-05-31 17:08:09.498] [jointLog] [info] Thread saw mini-batch with a maximum of 0.90% zero probability fragments
[2018-05-31 17:08:09.534] [jointLog] [info] Thread saw mini-batch with a maximum of 0.90% zero probability fragments
[2018-05-31 17:08:09.572] [jointLog] [info] Thread saw mini-batch with a maximum of 1.04% zero probability fragments
[2018-05-31 17:08:09.634] [jointLog] [info] Thread saw mini-batch with a maximum of 0.90% zero probability fragments
[2018-05-31 17:08:09.641] [jointLog] [info] Thread saw mini-batch with a maximum of 1.00% zero probability fragments
[2018-05-31 17:08:09.649] [jointLog] [info] Thread saw mini-batch with a maximum of 0.88% zero probability fragments
[2018-05-31 17:08:09.690] [jointLog] [info] Thread saw mini-batch with a maximum of 0.96% zero probability fragments
[2018-05-31 17:08:09.702] [jointLog] [info] Thread saw mini-batch with a maximum of 0.92% zero probability fragments
[2018-05-31 17:08:09.721] [jointLog] [info] Thread saw mini-batch with a maximum of 0.88% zero probability fragments
[2018-05-31 17:08:09.728] [jointLog] [info] Thread saw mini-batch with a maximum of 0.90% zero probability fragments
[2018-05-31 17:08:09.782] [jointLog] [info] Thread saw mini-batch with a maximum of 0.86% zero probability fragments
[2018-05-31 17:08:09.786] [jointLog] [info] Thread saw mini-batch with a maximum of 0.90% zero probability fragments
[2018-05-31 17:08:10.483] [jointLog] [info] Computed 582392 rich equivalence classes for further processing
[2018-05-31 17:08:10.483] [jointLog] [info] Counted 59985194 total reads in the equivalence classes
[2018-05-31 17:08:10.551] [jointLog] [info] Mapping rate = 88.2723%
[2018-05-31 17:08:10.551] [jointLog] [info] finished quantifyLibrary()
[2018-05-31 17:08:10.551] [jointLog] [info] Starting optimizer
[2018-05-31 17:08:11.488] [jointLog] [info] Marked 1 weighted equivalence classes as degenerate
[2018-05-31 17:08:11.523] [jointLog] [info] iteration = 0 | max rel diff. = 127.399
Exception : [Error in function boost::math::digamma(double): numeric overflow]
salmon quant was invoked improperly.
For usage information, try salmon quant --help
Exiting.
The text was updated successfully, but these errors were encountered: