PR for release 2.0.3

.travis.yml

@@ -13,7 +13,7 @@ before_install:
   # Pull the docker image first so the test doesn't wait for this
   - docker pull nfcore/eager
   # Fake the tag locally so that the pipeline runs properly
-  - docker tag nfcore/eager nfcore/eager:2.0.2
+  - docker tag nfcore/eager nfcore/eager:latest
 
 install:
   # Install Nextflow
@@ -37,16 +37,16 @@ script:
   # Lint the pipeline code
   - nf-core lint ${TRAVIS_BUILD_DIR}
   # Run the basic pipeline with the test profile
-  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --pairedEnd
+  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --pairedEnd --saveReference
   # Run the basic pipeline with single end data (pretending its single end actually)
-  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --singleEnd 
+  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --singleEnd --bwa_index results/reference_genome/bwa_index/
   # Run the same pipeline testing optional step: fastp, complexity 
-  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --pairedEnd --complexity_filter 
+  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --pairedEnd --complexity_filter --bwa_index results/reference_genome/bwa_index/
   # Test BAM Trimming
-  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --pairedEnd --trim_bam 
+  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --pairedEnd --trim_bam --bwa_index results/reference_genome/bwa_index/
   # Test running with CircularMapper
   - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --pairedEnd --circularmapper --circulartarget 'NC_007596.2'
   # Test running with BWA Mem
-  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --pairedEnd --bwamem
+  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,docker --pairedEnd --bwamem --bwa_index results/reference_genome/bwa_index/
   # Test basic pipeline with Conda too 
-  - nextflow run ${TRAVIS_BUILD_DIR} -profile test,conda --pairedEnd
+  - travis_wait 25 nextflow run ${TRAVIS_BUILD_DIR} -profile test,conda --pairedEnd --bwa_index results/reference_genome/bwa_index/

CHANGELOG.md

@@ -4,9 +4,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
-## unpublished
+## [Unpublished]
 
 
+## [2.0.3] - 2018-12-09
+
+### `Added`
+* [#80](https://github.com/nf-core/eager/pull/80) - BWA Index file handling 
+* [#77](https://github.com/nf-core/eager/pull/77) - Lots of documentation updates by [@jfy133](https://github.com/jfy133)
+* [#81](https://github.com/nf-core/eager/pull/81) - Renaming of certain BAM options
+* [#92](https://github.com/nf-core/eager/issues/92) - Complete restructure of BAM options
+
+### `Fixed`
+* [#84](https://github.com/nf-core/eager/pull/85) - Fix for [Samtools index issues](https://github.com/nf-core/eager/issues/84)
+* [#96](https://github.com/nf-core/eager/issues/96) - Fix for [MarkDuplicates issues](https://github.com/nf-core/eager/issues/96) found by [@nilesh-tawari](https://github.com/nilesh-tawari)
+
+### Other
+* Added Slack button to repository readme
+
 ## [2.0.2] - 2018-11-03
 
 ### `Changed`

Dockerfile

@@ -3,4 +3,4 @@ FROM nfcore/base
 LABEL description="Docker image containing all requirements for nf-core/eager pipeline"
 COPY environment.yml /
 RUN conda env create -f /environment.yml && conda clean -a
-ENV PATH /opt/conda/envs/nf-core-eager-2.0.2/bin:$PATH
+ENV PATH /opt/conda/envs/nf-core-eager-2.0.3dev/bin:$PATH

README.md

@@ -2,8 +2,7 @@
 
 [![Build Status](https://travis-ci.org/nf-core/eager.svg?branch=master)](https://travis-ci.org/nf-core/eager)
 [![Nextflow](https://img.shields.io/badge/nextflow-%E2%89%A50.32.0-brightgreen.svg)](https://www.nextflow.io/)
-[![Gitter](https://img.shields.io/badge/gitter-%20join%20chat%20%E2%86%92-4fb99a.svg)](https://gitter.im/nf-core/eager)
-[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/)
+[![Slack Status](https://nf-core-invite.herokuapp.com/badge.svg)](https://nf-core-invite.herokuapp.com)[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/)
 [![Docker Container available](https://img.shields.io/docker/automated/nfcore/eager.svg)](https://hub.docker.com/r/nfcore/eager/)
 ![Singularity Container available](https://img.shields.io/badge/singularity-available-7E4C74.svg)
 [![DOI](https://zenodo.org/badge/135918251.svg)](https://zenodo.org/badge/latestdoi/135918251)
@@ -12,28 +11,60 @@
 
 ## Introduction
 
-**nf-core/eager** is a bioinformatics best-practice analysis pipeline for ancient DNA data analysis.
+**nf-core/eager** is a bioinformatics best-practice analysis pipeline for NGS 
+sequencing based ancient DNA (aDNA) data analysis.
 
-The pipeline uses [Nextflow](https://www.nextflow.io), a bioinformatics workflow tool. It pre-processes raw data from FastQ inputs, aligns the reads and performs extensive quality-control on the results. It comes with docker / singularity containers making installation trivial and results highly reproducible.
+The pipeline uses [Nextflow](https://www.nextflow.io), a bioinformatics 
+workflow tool. It pre-processes raw data from FASTQ inputs, aligns the reads 
+and performs extensive general NGS and aDNA specific quality-control on the 
+results. It comes with docker, singularity or conda containers making 
+installation trivial and results highly reproducible.
 
-### Pipeline steps
+## Pipeline steps
 
-* Create reference genome indices (optional)
-    * BWA 
-    * Samtools Index
-    * Sequence Dictionary
-* QC with FastQC
-* AdapterRemoval for read clipping and merging
-* Read mapping with BWA, BWA Mem or CircularMapper
-* Samtools sort, index, stats & conversion to BAM
-* DeDup or MarkDuplicates read deduplication
-* QualiMap BAM QC Checking
-* Preseq Library Complexity Estimation
-* DamageProfiler damage profiling
-* BAM Clipping for UDG+/UDGhalf protocols
-* PMDTools damage filtering / assessment
+By default the pipeline currently performs the following:
+
+* Create reference genome indices for mapping (`bwa`, `samtools`, and `picard`)
+* Sequencing quality control (`FastQC`)
+* Sequencing adapter removal and for paired end data merging (`AdapterRemoval`)
+* Read mapping to reference using (`bwa aln`, `bwa mem` or `CircularMapper`)
+* Post-mapping processing, statistics and conversion to bam (`samtools`)
+* Ancient DNA C-to-T damage pattern visualisation (`DamageProfiler`)
+* PCR duplicate removal (`DeDup` or `MarkDuplicates`)
+* Post-mapping statistics and BAM quality control (`Qualimap`)
+* Library Complexity Estimation (`preseq`)
+* Overall pipeline statistics summaries (`MultiQC`)
+
+Additional functionality contained by the pipeline currently includes:
+
+* Illumina two-coloured sequencer poly-G tail removal (`fastp`)
+* Automatic conversion of unmapped reads to FASTQ (`samtools`)
+* Damage removal/clipping for UDG+/UDG-half treatment protocols (`BamUtil`)
+* Damage reads extraction and assessment (`PMDTools`)
+
+## Quick Start
+
+1. Install [`nextflow`](docs/installation.md)
+2. Install one of [`docker`](https://docs.docker.com/engine/installation/), [`singularity`](https://www.sylabs.io/guides/3.0/user-guide/) or [`conda`](https://conda.io/miniconda.html)
+3. Download the EAGER pipeline
+
+```bash
+nextflow pull nf-core/eager
+```
+
+4. Set up your job with default parameters
+
+```bash
+nextflow run nf-core -profile <docker/singularity/conda> --reads'*_R{1,2}.fastq.gz' --fasta '<REFERENCE.fasta'
+```
+
+5. See the overview of the run with under `<OUTPUT_DIR>/MultiQC/multiqc_report.html`
+
+Modifications to the default pipeline are easily made using various options
+as described in the documentation.
+
+## Documentation
 
-### Documentation
 The nf-core/eager pipeline comes with documentation about the pipeline, found in the `docs/` directory:
 
 1. [Installation](docs/installation.md)
@@ -44,5 +75,25 @@ The nf-core/eager pipeline comes with documentation about the pipeline, found in
 4. [Output and how to interpret the results](docs/output.md)
 5. [Troubleshooting](docs/troubleshooting.md)
 
-### Credits
-This pipeline was written by Alexander Peltzer ([apeltzer](https://github.com/apeltzer)), with major contributions from Stephen Clayton, ideas and documentation from James Fellows-Yates, Raphael Eisenhofer and Judith Neukamm. If you want to contribute, please open an issue and ask to be added to the project - happy to do so and everyone is welcome to contribute here!
+## Credits
+
+This pipeline was written by Alexander Peltzer ([apeltzer](https://github.com/apeltzer)), 
+with major contributions from Stephen Clayton, ideas and documentation from 
+James Fellows Yates, Raphael Eisenhofer and Judith Neukamm. If you want to 
+contribute, please open an issue and ask to be added to the project - happy to 
+do so and everyone is welcome to contribute here!
+
+## Tool References
+
+* **EAGER v1**, CircularMapper, DeDup* Peltzer, A., Jäger, G., Herbig, A., Seitz, A., Kniep, C., Krause, J., & Nieselt, K. (2016). EAGER: efficient ancient genome reconstruction. Genome Biology, 17(1), 1–14. [https://doi.org/10.1186/s13059-016-0918-z](https://doi.org/10.1186/s13059-016-0918-z)  Download: [https://github.com/apeltzer/EAGER-GUI](https://github.com/apeltzer/EAGER-GUI) and [https://github.com/apeltzer/EAGER-CLI](https://github.com/apeltzer/EAGER-CLI)
+* **FastQC** download: [https://www.bioinformatics.babraham.ac.uk/projects/fastqc/](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)
+* **AdapterRemoval v2** Schubert, M., Lindgreen, S., & Orlando, L. (2016). AdapterRemoval v2: rapid adapter trimming, identification, and read merging. BMC Research Notes, 9, 88. [https://doi.org/10.1186/s13104-016-1900-2](https://doi.org/10.1186/s13104-016-1900-2) Download: [https://github.com/MikkelSchubert/adapterremoval](https://github.com/MikkelSchubert/adapterremoval)
+* **bwa** Li, H., & Durbin, R. (2009). Fast and accurate short read alignment with Burrows-Wheeler transform. Bioinformatics , 25(14), 1754–1760. [https://doi.org/10.1093/bioinformatics/btp324](https://doi.org/10.1093/bioinformatics/btp324) Download: [http://bio-bwa.sourceforge.net/bwa.shtml](http://bio-bwa.sourceforge.net/bwa.shtml)
+* **SAMtools** Li, H., Handsaker, B., Wysoker, A., Fennell, T., Ruan, J., Homer, N., … 1000 Genome Project Data Processing Subgroup. (2009). The Sequence Alignment/Map format and SAMtools. Bioinformatics , 25(16), 2078–2079. [https://doi.org/10.1093/bioinformatics/btp352](https://doi.org/10.1093/bioinformatics/btp352) Download: [http://www.htslib.org/](http://www.htslib.org/)
+* **DamageProfiler** Judith Neukamm (Unpublished)
+* **QualiMap** Okonechnikov, K., Conesa, A., & García-Alcalde, F. (2016). Qualimap 2: advanced multi-sample quality control for high-throughput sequencing data. Bioinformatics , 32(2), 292–294. [https://doi.org/10.1093/bioinformatics/btv566](https://doi.org/10.1093/bioinformatics/btv566) Download: [http://qualimap.bioinfo.cipf.es/](http://qualimap.bioinfo.cipf.es/)
+* **preseq** Daley, T., & Smith, A. D. (2013). Predicting the molecular complexity of sequencing libraries. Nature Methods, 10(4), 325–327. [https://doi.org/10.1038/nmeth.2375](https://doi.org/10.1038/nmeth.2375). Download: [http://smithlabresearch.org/software/preseq/](http://smithlabresearch.org/software/preseq/)
+* **PMDTools** Skoglund, P., Northoff, B. H., Shunkov, M. V., Derevianko, A. P., Pääbo, S., Krause, J., & Jakobsson, M. (2014). Separating endogenous ancient DNA from modern day contamination in a Siberian Neandertal. Proceedings of the National Academy of Sciences of the United States of America, 111(6), 2229–2234. [https://doi.org/10.1073/pnas.1318934111](https://doi.org/10.1073/pnas.1318934111) Download: [https://github.com/pontussk/PMDtools](https://github.com/pontussk/PMDtools)
+* **MultiQC** Ewels, P., Magnusson, M., Lundin, S., & Käller, M. (2016). MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics , 32(19), 3047–3048. [https://doi.org/10.1093/bioinformatics/btw354](https://doi.org/10.1093/bioinformatics/btw354) Download: [https://multiqc.info/](https://multiqc.info/)
+* **BamUtils** Jun, G., Wing, M. K., Abecasis, G. R., & Kang, H. M. (2015). An efficient and scalable analysis framework for variant extraction and refinement from population-scale DNA sequence data. Genome Research, 25(6), 918–925. [https://doi.org/10.1101/gr.176552.114](https://doi.org/10.1101/gr.176552.114) Download: [https://genome.sph.umich.edu/wiki/BamUtil](https://genome.sph.umich.edu/wiki/BamUtil)
+* **FastP** Chen, S., Zhou, Y., Chen, Y., & Gu, J. (2018). fastp: an ultra-fast all-in-one FASTQ preprocessor. Bioinformatics , 34(17), i884–i890. [https://doi.org/10.1093/bioinformatics/bty560](https://doi.org/10.1093/bioinformatics/bty560) Download: [https://github.com/OpenGene/fastp](https://github.com/OpenGene/fastp)

Singularity

@@ -4,10 +4,10 @@ Bootstrap:docker
 %labels
     MAINTAINER Alexander Peltzer <[email protected]>
     DESCRIPTION Container image containing all requirements for the nf-core/eager pipeline
-    VERSION 2.0.2
+    VERSION 2.0.3dev
 
 %environment
-    PATH=/opt/conda/envs/nf-core-eager-2.0.2/bin:$PATH
+    PATH=/opt/conda/envs/nf-core-eager-2.0.3dev/bin:$PATH
     export PATH
 
 %files

conf/acad-pheonix.config

@@ -0,0 +1,23 @@
+/*
+ * ----------------------------------------------------------------------------
+ *  Nextflow config file for use with Singularity on Phoenix Cluster Adelaide
+ * ----------------------------------------------------------------------------
+ * Defines basic usage limits and singularity image id.
+ */
+
+singularity {
+    enabled = true
+    envWhitelist='SINGULARITY_BINDPATH'
+    autoMounts = true
+}
+
+process {
+    beforeScript = 'module load Singularity/2.5.2-GCC-5.4.0-2.26'
+    executor = 'slurm'
+}
+
+params {
+  max_memory = 128.GB
+  max_cpus = 32
+  max_time = 48.h
+}

conf/binac.config

@@ -10,7 +10,7 @@ singularity {
 }
 
 process {
-    beforeScript = 'module load devel/singularity/2.4.1'
+    beforeScript = 'module load devel/singularity/2.6.0'
     executor = 'pbs'
     queue = 'short'
 }

conf/multiqc_config.yaml

@@ -5,3 +5,15 @@ report_comment: >
 report_section_order:
     nf-core/eager-software-versions:
         order: -1000
+    fastqc:
+        after: 'nf-core/eager-software-versions'
+    adapterRemoval:
+        after: 'fastqc'
+    Samtools:
+        after: 'adapterRemoval'
+    dedup:
+        after: 'Samtools'
+    qualimap:
+        after: 'dedup'
+    preseq:
+        after: 'qualimap'

conf/shh.config

@@ -0,0 +1,36 @@
+/*
+ * -------------------------------------------------------------
+ *  Nextflow config file for use with Singularity at SHH Clusters
+ * -------------------------------------------------------------
+ * Defines basic usage limits and singularity image id.
+ */
+
+singularity {
+    enabled = true
+}
+
+/*
+* To be improved by process specific resource requests
+* By default, take the medium queue, smaller processes might just go to short (e.g. multiqc or similar things)
+*/
+
+process {
+    executor = 'slurm'
+    queue = 'medium'
+
+
+    withName:makeFastaIndex {
+        queue = 'short'
+        time = 2.h
+    }
+    withName:makeSeqDict {
+        queue = 'short'
+        time = 2.h
+    }
+}
+
+params {
+  max_memory = 734.GB
+  max_cpus = 64
+  max_time = 48.h
+}

docs/configuration/adding_your_own.md

@@ -51,7 +51,6 @@ Note that the dockerhub organisation name annoyingly can't have a hyphen, so is
 ### Singularity image
 Many HPC environments are not able to run Docker due to security issues.
 [Singularity](http://singularity.lbl.gov/) is a tool designed to run on such HPC systems which is very similar to Docker.
->>>>>>> TEMPLATE
 
 To specify singularity usage in your pipeline config file, add the following:
 
@@ -81,5 +80,4 @@ To use conda in your own config file, add the following:
 
 ```nextflow
 process.conda = "$baseDir/environment.yml"
->>>>>>> TEMPLATE
 ```

docs/configuration/local.md

@@ -11,7 +11,7 @@ First, install docker on your system: [Docker Installation Instructions](https:/
 
 Then, simply run the analysis pipeline:
 ```bash
-nextflow run nf-core/eager -profile docker --reads '<path to your reads>'
+nextflow run nf-core/eager -profile docker --reads '<path to your reads>' --pairedEnd
 ```
 
 Nextflow will recognise `nf-core/eager` and download the pipeline from GitHub. The `-profile docker` configuration lists the [nf-core/eager](https://hub.docker.com/r/nfcore/eager/) image that we have created and is hosted at dockerhub, and this is downloaded.
@@ -23,9 +23,13 @@ The public docker images are tagged with the same version numbers as the code, w
 
 
 ## Singularity image
-Many HPC environments are not able to run Docker due to security issues. [Singularity](http://singularity.lbl.gov/) is a tool designed to run on such HPC systems which is very similar to Docker. Even better, it can use create images directly from dockerhub.
+Many HPC environments are not able to run Docker due to security issues. [Singularity](http://singularity.lbl.gov/) is a tool designed to run on such HPC systems which is very similar to Docker. There is a particular profile that will download the singularity image for you.
 
-To use the singularity image for a single run, use `-with-singularity 'docker://nfcore/eager'`. This will download the docker container from dockerhub and create a singularity image for you dynamically.
+```bash
+nextflow run nf-core/eager -profile singularity --reads '<path to your reads>' --pairedEnd
+```
+
+Additionally, it can use create images directly from dockerhub. To use the singularity image for a single run, use `-with-singularity 'docker://nfcore/eager'`. This will download the docker container from dockerhub and create a singularity image for you dynamically.
 
 If you intend to run the pipeline offline, nextflow will not be able to automatically download the singularity image for you. Instead, you'll have to do this yourself manually first, transfer the image file and then point to that.
 
@@ -38,5 +42,14 @@ singularity pull --name nf-core-eager.img docker://nfcore/eager
 Then transfer this file and run the pipeline with this path:
 
 ```bash
-nextflow run /path/to/nf-core/eager -with-singularity /path/to/nf-core-eager.img
+nextflow run /path/to/nf-core/eager -with-singularity /path/to/nf-core-eager.img --reads --pairedEnd
 ```
+
+## Conda
+
+You may also use conda (utilising the bioconda repository) to download the pipeline dependencies for you.
+
+```bash
+nextflow run nf-core/eager -profile conda --reads '<path to your reads>' --pairedEnd
+```
+