A better diagnostic during model fitting would be nice

[tedgoldstein]

This is not very informative.The problem probably is in the glm package.

glm.nb failed; trying again with glm and family=negative.binomial(theta=0.1)
glm and family=negative.binomial(theta=0.1) failed, falling back to scale(log10(y+1))
|================================================================= | 95%glm.nb failed; trying again with glm and family=negative.binomial(theta=0.1)
glm and family=negative.binomial(theta=0.1) failed, falling back to scale(log10(y+1))
|================================================================== | 95%glm.nb failed; trying again with glm and family=negative.binomial(theta=0.1)
glm and family=negative.binomial(theta=0.1) failed, falling back to scale(log10(y+1))
|=====================================================================| 100%
|=====================================================================| 100%
There were 50 or more warnings (use warnings() to see the first 50)

[ChristophH]

The problem probably is in the input. For some genes with high variance due to mostly zero counts and very few non-zero counts the theta estimation does not converge. And sometimes, even with a manually specified small theta, the glm function also fails to converge. In the next release the messages will contain more details including the name of the gene.

[andrewwbutler]

Latest version has incorporated these changes now