conda?

[AntonioBaeza]

Looks like a great pipeline. Any chance you can have it in conda?

[mahesh-panchal]

Or better yet, would you consider adding/merging this to nf-core (https://nf-co.re/)?
All the modules would then support conda, docker, and singularity.
There's an old workflow that needs some modernization called hybrid-denovo where the assembly/annotation discussions are going on.
There is going to be an nf-core hackathon in October (https://nf-co.re/events/2021/hackathon-october-2021), where one could get help transforming your modules ready for an nf-core DSL2 workflow.
It would be really nice to see this integrated into nf-core.

[fmalmeida]

Hi @AntonioBaeza and @mahesh-panchal,

First of all, thanks for your interest in the pipeline.

about conda

Yes, updating the pipeline to also use conda environments instead of only Docker is definitely in my TODO/whish list.

However, I was not able to fully work on it yet because some of the tools are not yet available in conda. I am not very much of an expert in conda package development. I am still studying on how to help on that and create a bioconda package for the tools that yet don't have it. And a help on that would be deeply appreciated if any of you have some knowledge on it, or know someone that would like to collaborate.

about nf-core

About the nf-core consideration, I would gladly merge the pipeline into nf-core. However I have two considerations on it:

1. nf-core says that new pipelines must not overlap already existent nf-core pipelines. Thus, it may be a problem for bringing the pipeline to nf-core since they already have some of their own.
2. This pipeline "hybrid-denovo" seems to be focused on hybrid assemblies only, thus I don't know if it'd be of interest for them because MpGAP performs not only hybrid assemblies, but short reads only and long reads only too. Thus, we must properly discuss if it would be a possibility.

I will try to participate on the hackathon that you mentioned to see if the community together can help on solving this problems and work together on the best solution for that: either merging it in nf-core or developing new bioconda packages so that the pipeline can also run with conda-only.

In the mean time, do you know where I could starting discussing it with the nf-core community and developers?

Thanks again for your interests and ideas. Any help is immensely welcomed.

[mahesh-panchal]

Re: nf-core

Please join us on slack and discuss this in the #new-pipelines channel (https://nf-co.re/join/slack).
The primary issues I think are that all tools need to be in bioconda.
Although I mention 'hybrid-denovo', it is an old pipeline and I think one that needs to be replaced. I'm sure the community would benefit a workflow that could support as many types of input as possible, launching, short-read only, long-read only, or hybrid assemblies.

[fmalmeida]

I have re-checked the tools in the bioconda channel and it now seems that all the tools are available there. Thus, in the meantime, I will start re-shaping the pipeline so it can either run with Docker and conda as suggested by @AntonioBaeza.

Re: nf-core

Thank you again @mahesh-panchal for the information. I will bring the suggestion and discussion to our lab. group so we can then start discussions with nf-core community.

[onertipaday]

Adding to nf-core would be really great! I was about to ask for the possibility to use the pipeline with sge and singularity. Both features come out of the box with nf-core.

[fmalmeida]

In theory, both capabilities should be possible because it just depends on adjusting the nextflow config file to understand the profiles sge and singularity.

But definitely it would be easier if added to nf-core. The only problem is setting the processes to use conda and transforming the pipeline into the nf-core format.

I'll definitely need help on this.

I forgot to do it before, I will ask in the nf-core slack channel whether they're interested in adding it.

[fmalmeida]

I will just finish the implementation of the multi-sample workflow that is described in #15 before that.

This is very near to it's release and when done.

[fmalmeida]

Now that the pipeline has been reconfigured to understand a YAML so it can analyse multiple samples at once. I will start to take a look on the possibility to set the modules to use bioconda channels and bioconda docker images so it has an easy integration for using conda, docker or singularity.

This will be tackled in branch https://github.com/fmalmeida/MpGAP/tree/issue-17-conda

[fmalmeida]

Made some tests with a simpler pipeline (https://github.com/fmalmeida/ngs-preprocess) and was able to make it run with docker, conda and singularity.

I will try then to implement this possibility in MpGAP using what I've learnt from the implementation in this other pipeline, focusing on using bioconda packages.

[AntonioBaeza]

Obrigado Felipe!!!

…

On Mon, Nov 8, 2021 at 4:16 PM Felipe Marques de Almeida < ***@***.***> wrote: Made some tests with a simpler pipeline ( https://github.com/fmalmeida/ngs-preprocess) and was able to make it run with docker, conda and singularity. I will try then to implement this possibility in MpGAP using what I've learnt from the implementation in this other pipeline, focusing on using bioconda packages. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALIJRA5E72XWF774OFL2TDTULA42RANCNFSM5EJT2Q2A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- J. Antonio Baeza Associate Professor Department of Biological Sciences, Clemson University South Carolina, USA & Research Associate Smithsonian Marine Station at Fort Pierce, Florida, USA & Adjunct Faculty Universidad Catolica del Norte, Coquimbo, Chile Email: ***@***.*** & ***@***.*** Website: http://baezaantonio.wix.com/baezalabclemson Website (CI-team): https://baezaantonio.wixsite.com/clemsonmitogenomics Website ResearchGate: https://www.researchgate.net/profile/J_Baeza/ Website SemanticScholar: https://www.semanticscholar.org/author/Juan-Antonio-Baeza/144723920

[fmalmeida]

Hello guys,

Just to update that I tried to address this issue in PR #25.

This is the first time that I try to implement my pipelines with either conda and singularity at once, moreover, some tools used in it are either only available for linux-64, or its conda packages are neither available nor up-to-date, causing incompatibility with other tools.

Thus, to solve the tools that had this problem, I have made pre-compiled linux-64 binaries of them available inside the main/root pipeline bin directory. Therefore, the -profile conda will only work in linux-64 systems. Users in other operating systems must use docker or singularity.

Please, since it is the first time I did it, the conda profile must be considered rather experimental and please, do not hesitate to contact me if a problem arise.

Read more at: https://github.com/fmalmeida/MpGAP#selecting-between-profiles

😄