Include custom picard shell script, to increase Java memory heap space

[lucventurini]

While trialling Eager on a genome of interest for my group (H. vulgare, which is over 4GBs long) we ran into a problem with the Picard shipped in BioConda and included in the Docker image. Namely, the launch script has an extremely low memory allocation (-Xms512m -Xmx1g) which is insufficient for our genome. This causes the pipeline to crash. We are aware that we can provide the genome from outside with the --seq_dict parameter after generating it with a non-bioconda version of picard, however, we find the situation a bit inconvenient.

A possible solution would be to change the picard shell script with a modified version with more generous memory allocation. I have attached the Dockerfile specification and the Picard modified script to the issue.

Is there by any chance any other possible solution that you might be aware of?

Thank you

@lucventurini and @hesterjvs
Patch.zip

[apeltzer]

Which step is failing? The makeSeqDict or markDuplicates ?

In both cases, it should be possible to add a directive to use more memory. Otherwise, I'd rather want to update the bioconda recipe to make configuring this possible instead - that should help as well!

[lucventurini]

The makeSeqDict step. I have not been able to find any way to pass a directive to the picard shell script (which can be found here: https://github.com/bioconda/bioconda-recipes/blob/master/recipes/picard/picard.sh) even in a bare conda environment, even less so in nextflow.

It is definitely possible to update the Conda recipe, I do have the authority to do it myself. However, I am not sure that setting the memory at 4GB (the needed amount of memory for our genome) would be a good idea for a general-purpose recipe such as that in BioConda.

[apeltzer]

No, making it configurable would be step 1 - then I can adjust the script call here in the pipeline to allow for example specific settings for larger genomes in the Nextflow script.

That would benefit users here as well as the bioconda community, as they could simply have a script that is configurable 👍

[lucventurini]

Dear @apeltzer ,
after further testing with the existing script, I realised that the script present in bioconda is capable of parsing the java values - it was my mistake in interpreting the code that led me astray.

Specifically, it is possible to add more memory like this:

picard -Xmx<memory value> -Xms<memory value> CreateSequenceDictionary [...]

Would it be possible to modify the picard calls so that they refer to e.g. the "NXF_OPTS" variable?

Best

[apeltzer]

Can do that - e.g. running the job with more memory by default should be fine 👍

[lucventurini]

That would be great! Thanks for the support, and please let us know whether we can help in any way.

[apeltzer]

I think #145 addresses your issue - will wait for tests to pass, a review and then merge into the dev branch if all things are ok 👍

[apeltzer]

Fixed in #145