Merge pull request #531 from drpatelh/https

Direct download of Singularity biocontainer images. Fixes #494 and #496
nf-core · Dec 12, 2020 · 2c732fa · 2c732fa
2 parents 69d024f + e25e3c8
commit 2c732fa
Show file tree

Hide file tree

Showing 65 changed files with 407 additions and 157 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -20,7 +20,7 @@ jobs:
     strategy:
       matrix:
         # Nextflow versions: check pipeline minimum and current latest
-        nxf_ver: ['20.07.1', '']
+        nxf_ver: ['20.11.0-edge']
     steps:
       - name: Check out pipeline code
         uses: actions/checkout@v2
@@ -39,7 +39,7 @@ jobs:
     if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/rnaseq') }}
     runs-on: ubuntu-latest
     env:
-      NXF_VER: '20.07.1'
+      NXF_VER: '20.11.0-edge'
       NXF_ANSI_LOG: false
     strategy:
       matrix:
@@ -71,7 +71,7 @@ jobs:
     if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/rnaseq') }}
     runs-on: ubuntu-latest
     env:
-      NXF_VER: '20.07.1'
+      NXF_VER: '20.11.0-edge'
       NXF_ANSI_LOG: false
     strategy:
       matrix:
@@ -98,7 +98,7 @@ jobs:
     if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/rnaseq') }}
     runs-on: ubuntu-latest
     env:
-      NXF_VER: '20.07.1'
+      NXF_VER: '20.11.0-edge'
       NXF_ANSI_LOG: false
     strategy:
       matrix:
@@ -123,7 +123,7 @@ jobs:
     if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/rnaseq') }}
     runs-on: ubuntu-latest
     env:
-      NXF_VER: '20.07.1'
+      NXF_VER: '20.11.0-edge'
       NXF_ANSI_LOG: false
     strategy:
       matrix:
@@ -148,7 +148,7 @@ jobs:
     if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/rnaseq') }}
     runs-on: ubuntu-latest
     env:
-      NXF_VER: '20.07.1'
+      NXF_VER: '20.11.0-edge'
       NXF_ANSI_LOG: false
     steps:
       - name: Check out pipeline code

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,16 +3,23 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## v2.1dev - [date]
+## [[3.0](https://github.com/nf-core/rnaseq/releases/tag/3.0)] - 2020-12-15
 
-### Major enhancements
+### :warning: Major enhancements
 
-* The aligned BAM files generated by `--aligner star` will now be quantified using Salmon instead of featureCounts. As a result, the name of this option has now been changed to `--aligner star_salmon` and this will now be the default route through the pipeline. This decision was made primarily because of the limitations of featureCounts to appropriately quantify gene expression data. Please see [Zhao et al., 2015](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0141910#pone-0141910-t001) and [Soneson et al., 2015](https://f1000research.com/articles/4-1521/v1)).
-* For similar reasons, quantification will not be performed if using `--aligner hisat2` due to the lack of an appropriate option to calculate accurate expression estimates from HISAT2 derived genomic alignments. However, you can use this route if you have a preference for the alignment, QC and other types of downstream analysis compatible with the output of HISAT2.
+* You will need to install Nextflow `>=20.11.0-edge` to run the pipeline. If you are using Singularity, then features introduced in that release now enable the pipeline to directly download Singularity images hosted by Biocontainers as opposed to performing a conversion from Docker images (see [#496](https://github.com/nf-core/rnaseq/issues/496)).
+* The previous default of aligning BAM files using STAR and quantifying using featureCounts (`--aligner star`) has been removed. The new default is to align with STAR and quantify using Salmon (`--aligner star_salmon`).
+  * This decision was made primarily because of the limitations of featureCounts to appropriately quantify gene expression data. Please see [Zhao et al., 2015](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0141910#pone-0141910-t001) and [Soneson et al., 2015](https://f1000research.com/articles/4-1521/v1)).
+* For similar reasons, **quantification will not be performed** if using `--aligner hisat2` due to the lack of an appropriate option to calculate accurate expression estimates from HISAT2 derived genomic alignments.
+  * This pipeline option is still available for those who have a preference for the alignment, QC and other types of downstream analysis compatible with the output of HISAT2. No gene-level quantification results will be generated.
+  * In a future release we hope to add back quantitation for HISAT2 using different tools.
 
 ### Enhancements & fixes
 
 * Updated pipeline template to nf-core/tools `1.12.1`
+* Bump Nextflow version `20.07.1` -> `20.11.0-edge`
+* [[#494](https://github.com/nf-core/rnaseq/issues/494)] - Issue running rnaseq v2.0 (DSL2) with test profile
+* [[#496](https://github.com/nf-core/rnaseq/issues/496)] - Direct download of Singularity images via HTTPS
 * [[#498](https://github.com/nf-core/rnaseq/issues/498)] - Significantly different versions of STAR in star_rsem (2.7.6a) and star (2.6.1d)
 * [[#499](https://github.com/nf-core/rnaseq/issues/499)] - Use of salmon counts for DESeq2
 * [[#500](https://github.com/nf-core/rnaseq/issues/500), [#509](https://github.com/nf-core/rnaseq/issues/509)] - Error with AWS batch params
@@ -22,13 +29,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Parameters
 
-| Old parameter                | New parameter               |
-|------------------------------|-----------------------------|
-| `--fc_extra_attributes`      | `--gtf_extra_attributes`    |
-| `--fc_group_features`        | `--gtf_group_features`      |
-| `--fc_count_type`            | `--gtf_count_type`          |
-| `--fc_group_features_type`   | `--gtf_group_features_type` |
-| `--skip_featurecounts`       | `-`                         |
+| Old parameter                | New parameter                         |
+|------------------------------|---------------------------------------|
+| `--fc_extra_attributes`      | `--gtf_extra_attributes`              |
+| `--fc_group_features`        | `--gtf_group_features`                |
+| `--fc_count_type`            | `--gtf_count_type`                    |
+| `--fc_group_features_type`   | `--gtf_group_features_type`           |
+|                              | `--singularity_pull_docker_container` |
+| `--skip_featurecounts`       |                                       |
 
 > **NB:** Parameter has been __updated__ if both old and new parameter information is present.  
 > **NB:** Parameter has been __added__ if just the new parameter information is present.  

diff --git a/README.md b/README.md
@@ -5,7 +5,7 @@
 [![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/rnaseq/results)
 [![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.1400710-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.1400710)
 
-[![Nextflow](https://img.shields.io/badge/nextflow%20DSL2-%E2%89%A520.07.1-23aa62.svg?labelColor=000000)](https://www.nextflow.io/)
+[![Nextflow](https://img.shields.io/badge/nextflow%20DSL2-%E2%89%A520.11.0--edge-23aa62.svg?labelColor=000000)](https://www.nextflow.io/)
 [![run with conda](http://img.shields.io/badge/run%20with-conda-3EB049?labelColor=000000&logo=anaconda)](https://docs.conda.io/en/latest/)
 [![run with docker](https://img.shields.io/badge/run%20with-docker-0db7ed?labelColor=000000&logo=docker)](https://www.docker.com/)
 [![run with singularity](https://img.shields.io/badge/run%20with-singularity-1d355c.svg?labelColor=000000)](https://sylabs.io/docs/)
@@ -64,7 +64,7 @@ On release, automated continuous integration tests run the pipeline on a [full-s
     ```
 
     > * Please check [nf-core/configs](https://github.com/nf-core/configs#documentation) to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use `-profile <institute>` in your command. This will enable either `docker` or `singularity` and set the appropriate execution settings for your local compute environment.
-    > * If you are using `singularity`, it is highly recommended to use the [`NXF_SINGULARITY_CACHEDIR` or `singularity.cacheDir`](https://www.nextflow.io/docs/latest/singularity.html?#singularity-docker-hub) settings to store the images in a central location for future pipeline runs.
+    > * If you are using `singularity` then the pipeline will auto-detect this and attempt to download the Singularity images directly as opposed to performing a conversion from Docker images. If you are persistently observing issues downloading Singularity images directly due to timeout or network issues then please use the `--singularity_pull_docker_container` parameter to pull and convert the Docker image instead. It is also highly recommended to use the [`NXF_SINGULARITY_CACHEDIR` or `singularity.cacheDir`](https://www.nextflow.io/docs/latest/singularity.html?#singularity-docker-hub) settings to store the images in a central location for future pipeline runs.
     > * If you are using `conda`, it is highly recommended to use the [`NXF_CONDA_CACHEDIR` or `conda.cacheDir`](https://www.nextflow.io/docs/latest/conda.html) settings to store the environments in a central location for future pipeline runs.
 
 4. Start running your own analysis!

diff --git a/conf/modules.config b/conf/modules.config
@@ -63,12 +63,11 @@ params {
             publish_dir   = "genome/index"
         }
         'star_align' {
-            args          = "--quantMode TranscriptomeSAM --twopassMode Basic --outSAMtype BAM Unsorted --readFilesCommand zcat --runRNGseed 0"
+            args          = "--quantMode TranscriptomeSAM --twopassMode Basic --outSAMtype BAM Unsorted --readFilesCommand zcat --runRNGseed 0 --outFilterMultimapNmax 20 --alignSJDBoverhangMin 1 --outSAMattributes NH HI AS NM MD --quantTranscriptomeBan Singleend"
             publish_dir   = "${params.aligner}"
             publish_files = ['out':'log', 'tab':'log']
         }
         'star_salmon_quant' {
-            args          = "--seqBias --useVBOpt --gcBias"
             publish_dir   = "${params.aligner}"
         }
         'star_salmon_tximport' {
@@ -107,7 +106,7 @@ params {
             publish_dir   = "genome/index"
         }
         'salmon_quant' {
-            args          = "--validateMappings --seqBias --useVBOpt --gcBias"
+            args          = ""
         }
         'salmon_tximport' {
             publish_by_id = true

diff --git a/modules/local/process/bedtools_genomecov.nf b/modules/local/process/bedtools_genomecov.nf
@@ -11,9 +11,13 @@ process BEDTOOLS_GENOMECOV {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:getSoftwareName(task.process), publish_id:meta.id) }
 
-    conda     (params.enable_conda ? "bioconda::bedtools=2.29.2" : null)
-    container "quay.io/biocontainers/bedtools:2.29.2--hc088bd4_0"
-
+    conda (params.enable_conda ? "bioconda::bedtools=2.29.2" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/bedtools:2.29.2--hc088bd4_0"
+    } else {
+        container "quay.io/biocontainers/bedtools:2.29.2--hc088bd4_0"
+    }
+
     input:
     tuple val(meta), path(bam)
 

diff --git a/modules/local/process/cat_additional_fasta.nf b/modules/local/process/cat_additional_fasta.nf
@@ -12,8 +12,12 @@ process CAT_ADDITIONAL_FASTA {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:'genome', publish_id:'') }
 
-    conda     (params.enable_conda ? "conda-forge::python=3.8.3" : null)
-    container "quay.io/biocontainers/python:3.8.3"
+    conda (params.enable_conda ? "conda-forge::python=3.8.3" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/python:3.8.3"
+    } else {
+        container "quay.io/biocontainers/python:3.8.3"
+    }
 
     input:
     path fasta

diff --git a/modules/local/process/cat_fastq.nf b/modules/local/process/cat_fastq.nf
@@ -13,8 +13,12 @@ process CAT_FASTQ {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:'merged_fastq', publish_id:meta.id) }
 
-    conda     (params.enable_conda ? "conda-forge::sed=4.7" : null)
-    container "biocontainers/biocontainers:v1.2.0_cv1"
+    conda (params.enable_conda ? "conda-forge::sed=4.7" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://containers.biocontainers.pro/s3/SingImgsRepo/biocontainers/v1.2.0_cv1/biocontainers_v1.2.0_cv1.img"
+    } else {
+        container "biocontainers/biocontainers:v1.2.0_cv1"
+    }
 
     input:
     tuple val(meta), path(reads)

diff --git a/modules/local/process/deseq2_qc.nf b/modules/local/process/deseq2_qc.nf
@@ -11,9 +11,13 @@ process DESEQ2_QC {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:getSoftwareName(task.process), publish_id:'') }
 
-    conda     (params.enable_conda ? "conda-forge::r-base=4.0.3 conda-forge::r-optparse=1.6.6 conda-forge::r-ggplot2=3.3.2 conda-forge::r-rcolorbrewer=1.1_2 conda-forge::r-pheatmap=1.0.12 bioconda::bioconductor-deseq2=1.28.0 bioconda::bioconductor-biocparallel=1.22.0 bioconda::bioconductor-tximport=1.16.0 bioconda::bioconductor-complexheatmap=2.4.2" : null)
-    container "quay.io/biocontainers/mulled-v2-8849acf39a43cdd6c839a369a74c0adc823e2f91:ab110436faf952a33575c64dd74615a84011450b-0"
-
+    conda (params.enable_conda ? "conda-forge::r-base=4.0.3 conda-forge::r-optparse=1.6.6 conda-forge::r-ggplot2=3.3.2 conda-forge::r-rcolorbrewer=1.1_2 conda-forge::r-pheatmap=1.0.12 bioconda::bioconductor-deseq2=1.28.0 bioconda::bioconductor-biocparallel=1.22.0 bioconda::bioconductor-tximport=1.16.0 bioconda::bioconductor-complexheatmap=2.4.2" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/mulled-v2-8849acf39a43cdd6c839a369a74c0adc823e2f91:ab110436faf952a33575c64dd74615a84011450b-0"
+    } else {
+        container "quay.io/biocontainers/mulled-v2-8849acf39a43cdd6c839a369a74c0adc823e2f91:ab110436faf952a33575c64dd74615a84011450b-0"
+    }
+
     input:
     path counts
     path pca_header_multiqc

diff --git a/modules/local/process/dupradar.nf b/modules/local/process/dupradar.nf
@@ -11,8 +11,12 @@ process DUPRADAR {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:getSoftwareName(task.process), publish_id:meta.id) }
 
-    conda     (params.enable_conda ? "bioconda::bioconductor-dupradar=1.18.0" : null)
-    container "quay.io/biocontainers/bioconductor-dupradar:1.18.0--r40_1"
+    conda (params.enable_conda ? "bioconda::bioconductor-dupradar=1.18.0" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/bioconductor-dupradar:1.18.0--r40_1"
+    } else {
+        container "quay.io/biocontainers/bioconductor-dupradar:1.18.0--r40_1"
+    }
 
     input:
     tuple val(meta), path(bam)

diff --git a/modules/local/process/get_chrom_sizes.nf b/modules/local/process/get_chrom_sizes.nf
@@ -12,8 +12,12 @@ process GET_CHROM_SIZES {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:"genome", publish_id:'') }
 
-    conda     (params.enable_conda ? "bioconda::samtools=1.10" : null)
-    container "quay.io/biocontainers/samtools:1.10--h9402c20_2"
+    conda (params.enable_conda ? "bioconda::samtools=1.10" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/samtools:1.10--h9402c20_2"
+    } else {
+        container "quay.io/biocontainers/samtools:1.10--h9402c20_2"
+    }
 
     input:
     path fasta

diff --git a/modules/local/process/get_software_versions.nf b/modules/local/process/get_software_versions.nf
@@ -11,8 +11,12 @@ process GET_SOFTWARE_VERSIONS {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:'pipeline_info', publish_id:'') }
 
-    conda     (params.enable_conda ? "conda-forge::python=3.8.3" : null)
-    container "quay.io/biocontainers/python:3.8.3"
+    conda (params.enable_conda ? "conda-forge::python=3.8.3" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/python:3.8.3"
+    } else {
+        container "quay.io/biocontainers/python:3.8.3"
+    }
 
     cache false
 

diff --git a/modules/local/process/gffread.nf b/modules/local/process/gffread.nf
@@ -10,8 +10,12 @@ process GFFREAD {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:getSoftwareName(task.process), publish_id:'') }
 
-    conda     (params.enable_conda ? "bioconda::gffread=0.12.1" : null)
-    container "quay.io/biocontainers/gffread:0.12.1--h8b12597_0"
+    conda (params.enable_conda ? "bioconda::gffread=0.12.1" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/gffread:0.12.1--h8b12597_0"
+    } else {
+        container "quay.io/biocontainers/gffread:0.12.1--h8b12597_0"
+    }
 
     input:
     path fasta

diff --git a/modules/local/process/gtf2bed.nf b/modules/local/process/gtf2bed.nf
@@ -13,8 +13,12 @@ process GTF2BED {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:'genome', publish_id:'') }
 
-    conda     (params.enable_conda ? "conda-forge::perl=5.26.2" : null)
-    container "quay.io/biocontainers/perl:5.26.2"
+    conda (params.enable_conda ? "conda-forge::perl=5.26.2" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/perl:5.26.2"
+    } else {
+        container "quay.io/biocontainers/perl:5.26.2"
+    }
 
     input:
     path gtf

diff --git a/modules/local/process/gtf_gene_filter.nf b/modules/local/process/gtf_gene_filter.nf
@@ -9,8 +9,12 @@ process GTF_GENE_FILTER {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:'genome', publish_id:'') }
 
-    conda     (params.enable_conda ? "conda-forge::python=3.8.3" : null)
-    container "quay.io/biocontainers/python:3.8.3"
+    conda (params.enable_conda ? "conda-forge::python=3.8.3" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/python:3.8.3"
+    } else {
+        container "quay.io/biocontainers/python:3.8.3"
+    }
 
     input:
     path fasta

diff --git a/modules/local/process/gunzip.nf b/modules/local/process/gunzip.nf
@@ -10,8 +10,12 @@ process GUNZIP {
         mode: params.publish_dir_mode,
         saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:getSoftwareName(task.process), publish_id:'') }
 
-    conda     (params.enable_conda ? "conda-forge::sed=4.7" : null)
-    container "biocontainers/biocontainers:v1.2.0_cv1"
+    conda (params.enable_conda ? "conda-forge::sed=4.7" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://containers.biocontainers.pro/s3/SingImgsRepo/biocontainers/v1.2.0_cv1/biocontainers_v1.2.0_cv1.img"
+    } else {
+        container "biocontainers/biocontainers:v1.2.0_cv1"
+    }
 
     input:
     path archive
@@ -24,7 +28,7 @@ process GUNZIP {
     def software = getSoftwareName(task.process)
     gunzip       = archive.toString() - '.gz'
     """
-    gunzip --force $options.args $archive
+    gunzip -f $options.args $archive
     echo \$(gunzip --version 2>&1) | sed 's/^.*(gzip) //; s/ Copyright.*\$//' > ${software}.version.txt
     """
 }