+
+ 404
+ +Page not found :(
+The requested page could not be found.
+ +
+ 
+ 
+
+
+
+
+
+
+
+
+
+
+ Page not found :(
+The requested page could not be found.
+ +
+
+ The simplest option is running an EC2 instance interactively where the pipeline can be installed as pointed in the previous documentation pages.
+ +Last available Amazon Machine (AMI) we provide is:
+When running an instance among the different available types, minimum CPU and memory requirements must be taken into account. These must fit with Nextflow executor process configuration values. In our sample configuration files we use m4.2xlarge for CPU and p3.2xlarge for GPU as examples.
Keep in mind that not all Amazon infrastructure regions may have the same instance types. As a example, in January 2020 Frankfurt has GPU nodes, but not Paris.
+ +Launch an instane from the AMI image above (Go to EC2 > Images > AMI and copy-paste the ID provided above filtering in public images). Once you find that image, you can launch an instance from it.
+ +You can connect to the launched instance by using this command below:
+ +ssh -i "key-nf.pem" ubuntu@xxx.eu-central-1.compute.amazonaws.com
+where key-nf.pem is your private key (reference) and host details can be obtained from Connect popup in EC2 instances dashboard.
For sake of commodity, you may prefer to automate deployment of EC2 instances and S3 buckets. Terraform is a convenient tool for this.
+ +Place terraform binary in your local workstation path and move where your are keeping your tf files. Examples are provided in the terraform directory of this repository.
+ +Adapt terraform configuration files to include your credentials, use your chosen instance types, which key pair they are associated with, or whether allow files in S3 bucket to be kept or not (force_destroy parameter).
Initialize terraform directory:
+ +terraform init
+Validate terraform files:
+terraform validate
+Inspect what changes are going to be performed in your AWS account:
+ +terraform plan
+Proceed:
+terraform apply
+Once analyses are finished, infrastructure can be dismantled with:
+terraform destroy
+Amazon Simple Storage Service (S3) is a convenient web service storage system for sharing raw input and final output files between your premises and your computing cloud instances.
+ +Below we provide some instructions and advices to set up a S3 bucket.
+ +Some convenient instructions for S3 permisions in your EC2 instance can be found here. From the previous link you can learn how to retrieve the key and password to place in /root/.passwd-s3fs
Ensure proper permission as well: chmod 600 /root/.passwd-s3fs
You can include in /etc/fstab the following mounting point (adapt according to your case):
frankfurt-nf /mnt/frankfurt-nf fuse.s3fs _netdev,allow_other,passwd_file=/root/.passwd-s3fs,uid=1000,gid=1000 0 0
+If not mounted, you can mount it therefore straightforward by running:
+ +sudo mount /mnt/frankfurt-nf
+Adapt your S3 bucket and mounting point names according to your choice.
+ +Specially for huge amount of data, we suggest to use AWS CLI to transfer files from your premises to a S3 Bucket (Ref). For instance, the commandline below uploads the data example file in a pre-existing bucket.
+aws s3 cp multifast5_1.fast5 s3://frankfurt-nf
+Modify your Nexflow configuration files in order to point your input files at the mounted S3 bucket. Both input and final output files can be placed in that mounted S3 storage, but we do not recommend that work Nextflow directory (containing pipeline intermediary files) is kept there, since it significatively delays the whole process. Choose a suitable disk size for your instance depending on the amount of data to be processed.
+
+ 
+
+
+
+
Nextflow pipeline for analysis of Nanopore data from direct RNA sequencing. This is a joint project between CRG bioinformatics core and Epitranscriptomics and RNA Dynamics research group.
+ +The direct RNA sequencing platform offered by Oxford Nanopore Technologies allows for direct measurement of RNA molecules without the need of conversion to complementary DNA (cDNA), and as such, is virtually capable of detecting any given RNA modification present in the molecule that is being sequenced.
+ +Although the technology has been publicly available since 2017, the complexity of the raw current intensity output data generated by nanopore sequencing, together with lack of systematic and reproducible pipelines for the analysis of direct RNA sequencing datasets, have greatly hindered the access of this technology to the general user. Here we provide an in silico scalable and parallelizable workflow for the analysis of direct RNA sequencing reads, which converts raw current intensities into multiple types of processed data, providing metrics of the quality of the run, per-gene counts, RNA modification predictions and polyA tail length predictions.
+ +The workflow named Master of Pores, which has been built using the Nextflow framework and is distributed with Docker and Singularity containers, can be executed on any Unix-compatible OS (both Linux and Mac OSX) on a computer, cluster or cloud without the need of installing any additional software or dependencies. The workflow is easily scalable, as it can incorporate updated software versions or algorithms that may be released in the future in a modular manner. We expect that our pipeline will make the analysis of direct RNA sequencing datasets highly simplified and accessible to the non-bioinformatic expert, and thus boost our understanding of the epitranscriptome with single molecule resolution.
+ +The MasterOfPores workflow includes all steps needed to process raw FAST5 files produced by Nanopore direct RNA sequencing and executes the following steps, allowing users a choice among different algorithms. The pipeline consists of 4 modules:
+ +This module takes as input the raw Fast5 reads and produces as output base-called FASTQ and BAM. The pre-processing module performs base-calling, demultiplexing, filtering, quality control, mapping, read counting, generating a final report of the performance and results of each of the steps performed. It automatically detects the kind of input fast5 file (single or multi sequence).
+The NanoPreprocess module comprises 8 main steps:
+ +This module is essentially nanopreprocess without basecalling. So it needs basecalled fastq files and optionally fast5 (if you need demultiplexing with DeePlexiCon).
+This module takes as input the output produced by the NanoPreprocess module and produces polyA tail estimations.
+The NanoTail module estimates polyA tail lengths using Nanopolish (https://github.com/jts/nanopolish) and Tailfindr (https://github.com/adnaniazi/tailfindr), producing a plain text file that includes polyA tail length estimates for each read, computed using both algorithms. The correlation between the two algorithms is also reported as a plot.
+ +This module takes as input the rthe output produced by the NanoPreprocess module and produces a flat text file which includes the predicted RNA modifications.
+The NanoMod module predicts RNA modifications using Tombo (https://github.com/nanoporetech/tombo) and EpiNano (https://github.com/enovoa/EpiNano), producing a plain text file that is intersection of predicted sites both algorithms, to reduce the number of false positives.
+ +If you use this tool please cite our paper:
+ +“MasterOfPores: A Workflow for the Analysis of Oxford Nanopore Direct RNA Sequencing Datasets” Luca Cozzuto, Huanle Liu, Leszek P. Pryszcz, Toni Hermoso Pulido, Anna Delgado-Tejedor, Julia Ponomarenko, Eva Maria Novoa. Front. Genet., 17 March 2020. https://doi.org/10.3389/fgene.2020.00211
+ + + +
+
+ For using the pipeline Nextflow and a linux container engine (either Docker or Singularity) need to be installed. +The pipeline can be run in Mac OSX and Linux operative systems.
+ +curl -s https://get.nextflow.io | bash
+The pipeline can be cloned in this way using git:
+ +git clone --depth 1 https://github.com/biocorecrg/master_of_pores.git
+Because of redistribution restriction of the basecallers Albacore and Guppy we cannot provide them inside the docker image, so you would need to download the binaries from the official website https://nanoporetech.com and place them inside the master_of_pores/NanoPreprocess/bin folder.
+ +cd master_of_pores/NanoPreprocess/bin
+tar -zvxf ont-guppy_3.1.5_linux64.tar.gz
+ln -s ont-guppy_3.1.5_linux64/ont-guppy/bin/guppy_* .
+pip3 install --target=./albacore ont_albacore-2.1.7-cp36-cp36m-manylinux1_x86_64.whl
+ln -s albacore/bin/read_fast5_basecaller.py .
+Download the wheel file.
+ +cd master_of_pores/NanoPreprocess/bin
+pip3 install --target=./albacore ont_albacore-2.1.7-cp36-cp36m-manylinux1_x86_64.whl
+$ ln -s albacore/bin/multi_to_single_fast5
+$ ln -s albacore/bin/read_fast5_basecaller.py .
+Please note Guppy versions older than 3.1 (e.g. 3.0.3) only runs on CPUs. +Newer versions (e.g. 3.1.5 and above) works on both CPUs and GPUs. The difference of speed between CPUs and GPU is more than 10 times.
+ +cd master_of_pores/NanoPreprocess/bin
+tar -zvxf ont-guppy_3.1.5_linux64.tar.gz
+ln -s ont-guppy_3.1.5_linux64/ont-guppy/bin/guppy_* .
+In case you want to use the GPU you need to install the CUDA drivers
+ +Using Singularity:
+cd master_of_pores/NanoPreprocess/
+nextflow run preprocessing.nf -with-singularity
+Using Docker:
+cd master_of_pores/NanoPreprocess/
+nextflow run preprocessing.nf -with-docker
+This module allows to predict the loci with RNA modifications starting from data produced by NanoPreprocess.
+ +
WT1 KO1
+WT2 KO2
+WT3 KO3
+Three folders are produced by this module:
+ +#Kmer,Window,Ref,Coverage,q1,q2,q3,q4,q5,mis1,mis2,mis3,mis4,mis5,ins1,ins2,ins3,ins4,ins5,del1,del2,del3,del4,del5,prediction,dist,ProbM,Pro
+bU
+AGTGG,394404:394405:394406:394407:394408,chr2,8.0:8.0:7.0:7.0:7.0,21.5,21.25,19.857,23.0,16.285999999999998,0.0,0.0,0.0,0.0,0.0,0.0,0.062,0.0
+71,0.0,0.0,0.0,0.0,0.0,0.0,0.0,unm,19.26143361547619,3.00000089999998e-14,0.9999999999999699
+TTTTT,12150:12151:12152:12153:12154,chr8,3.0:3.0:3.0:3.0:3.0,0.0,16.5,18.5,16.0,16.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.3329999999
+9999996,0.33299999999999996,0.0,0.0,unm,2.5976484688977424,0.06071658133381308,0.9392834186661868
+ACATT,438165:438166:438167:438168:438169,chr13,67.0:67.0:67.0:68.0:68.0,13.635,13.446,9.323,9.6,12.127,0.03,0.045,0.015,0.147,0.0740000000000
+0001,0.0,0.0,0.0,0.0,0.0,0.06,0.03,0.075,0.11800000000000001,0.07400000000000001,unm,0.08435556637195174,0.519879422458087,0.4801205775419129
+5...
+and three plots in pdf indicating possible events related to insertion, deletion and mismatches, see the example below.
+ +
>chr11:455562:- Est. Frac. Alternate: 0.98
+TGACA
+>chr12:1008723:- Est. Frac. Alternate: 0.98
+TATCT
+>chr15:491587:+ Est. Frac. Alternate: 0.96
+TATAT
+>chr10:425794:- Est. Frac. Alternate: 0.95
+ATGTT
+>chr13:510759:+ Est. Frac. Alternate: 0.95
+...
+And for convenience a 6 bed files with the coordinates of the event.
+ + +
+
+ This module takes as input the raw fast5 reads - single or multi - and produces a number of outputs (basecalled fast5, sequences in fastq format, aligned reads in BAM format etc). The pre-processing module performs base-calling, demultiplexing (optional), filtering, quality control, mapping to a genome / transcriptome reference, feature counting and it generates a final report of the performance and results of each of the steps performed. It automatically detects the kind of input fast5 file (single or multi sequence).
+ +
| Process name | +Description | +
|---|---|
| testInput | +Detection of kind of fast5 (multi or single) | +
| baseCalling | +Basecalling with Albacore or Guppy (up to guppy 4.0) | +
| demultiplexing | +Demultiplexing (optional) | +
| concatenateFastQFiles | +This process concatenates the fastq files produces for each single basecalling | +
| QC | +Performed with MinIONQC | +
| fastQC | +Executed on fastq files | +
| mapping | +Mapping to genome / transcriptome with either minimap2, graphmap orgraphmap2 | +
| counting | +If mapping to the genome, it obtains counts per gene with htseq-count. Otherwise, if mapping to the transcriptome, transcript counts are generated with NanoCount. Reads are also assigned to a gene or to a transcript if they are uniquely mapping. A report file is also generated. | +
| alnQC2 | +QC of aligned reads with NanoPlot. The plots PercentIdentityvsAverageBaseQuality_kde, LengthvsQualityScatterPlot_dot, HistogramReadlength and Weighted_HistogramReadlength are then merged together in a single picture. | +
| alnQC | +QC of aligned reads with bam2stats. | +
| cram_conversion | +Generating cram file from alignment. | +
| joinAlnQCs | +Merging the QC files generated by the alnQC step. | +
| joinCountQCs | +Merging the report files generated by the counting step. | +
| multiQC | +Final report generation - enventually sent by mail to the user too. | +
| Parameter name | +Description | +
|---|---|
| fast5 files | +Path to fast5 input files (single or multi-fast5 files). They should be inside a folder that will be used as sample name. [/Path/sample_name/*.fast5] | +
| reference | +File in fasta format. [Reference_file.fa] | +
| ref_type | +Specify if the reference is a genome or a transcriptome. [genome / transcriptome] | +
| kit | +Kit used in library prep - required for basecalling. | +
| flowcell | +Flowcell used in sequencing - required for basecalling. | +
| annotation | +Annotation file in GTF format. It is optional and needed only in case of mapping to the genome and when interested in gene counts. [Annotation_file.gtf] | +
| seq_type | +Sequence type. [RNA / DNA] | +
| output | +Output folder name. [/Path/to_output_folder] | +
| granularity | +indicates the number of input fast5 files analyzed in a single process. It is by default 4000 for single-sequence fast5 files and 1 for multi-sequence fast5 files. In case GPU option is turned on this value is not needed since every file will be analyzed sequentially. | +
| basecaller | +Algorithm to perform the basecalling. guppy or albacore are supported. [albacore / guppy] | +
| basecaller_opt | +Command line options for basecaller program. Check available options in respective basecaller repository. | +
| GPU | +Allow the pipeline to run with GPU. [OFF / ON] | +
| demultiplexing | +Demultiplexing algorithm to be used. [OFF / deeplexicon / guppy / guppy-readucks] | +
| demultiplexing_opt | +Command line options for the demultiplexing software. | +
| demulti_fast5 | +If performing demultiplexing, also generate demultiplexed multifast5 files. [OFF / ON] | +
| filter | +Program to filter fastq files. [nanofilt / OFF] | +
| filter_opt | +Command line options of the filtering program. | +
| mapper | +Mapping algorithm. [minimap2 / graphmap / graphmap2] | +
| mapper_opt | +Command line options of the mapping algorithm. | +
| map_type | +Spliced - recommended for genome mapping - or unspliced - recommended for transcriptome mapping. [spliced / unspliced] | +
| counter | +Generating gene counts (genome mapping) or transcript counts (transcriptome mapping). [YES / “”] | +
| counter_opt | +Command line options of the counter program: NanoCount for transcripts and Htseq-count for genes. | +
| Users email for receving the final report when the pipeline is finished. [user_email] | +
You can change them by editing the params.config file or using the command line - please, see next section.
+ +Before launching the pipeline, user should decide which containers to use - either docker or singularity [-with-docker / -with-singularity]. +Then, to launch the pipeline, please use the following command:
+nextflow run nanopreprocess.nf -with-singularity > log.txt
+nextflow run nanopreprocess.nf -with-singularity -bg > log.txt
+nextflow run nanopreprocess.nf -with-singularity -bg --output test2 > log.txt
+nextflow run nanopreprocess.nf -with-singularity -bg -w /path/working_directory > log.txt
+nextflow run nanopreprocess.nf -with-singularity -bg -w /path/working_directory -profile cluster > log.txt
+nextflow run nanopreprocess.nf -with-singularity -bg -w /path/working_directory -profile local > log.txt
+Checking what has gone wrong:
+If there is an error, please see the log file (log.txt) for more details. Furthermore, if more information is needed, you can also find the working directory of the process in the file. Then, access that directory and check both the .command.log and .command.err files.
+
Resume an execution:
+Once the error has been solved or if you change a specific parameter, you can resume the execution with the Netxtlow parameter -resume (only one dash!). If there was an error, the pipeline will resume from the process that had the error and proceed with the rest. If a parameter was changed, only processes affected by this parameter will be re-run.
nextflow run nanopreprocess.nf -with-singularity -bg -resume > log_resumed.txt
+To check whether the pipeline has been resumed properly, please check the log file. If previous correctly executed process are found as Cached, resume worked!
+...
+
+[warm up] executor > crg
+[e8/2e64bd] Cached process > baseCalling (RNA081120181_1)
+[b2/21f680] Cached process > QC (RNA081120181_1)
+[c8/3f5d17] Cached process > mapping (RNA081120181_1)
+...
+
+IMPORTANT: To resume the execution, temporary files generated previously by the pipeline must be kept. Otherwise, pipeline will re-start from the beginning.
+ +Several folders are created by the pipeline within the output directory specified by the output parameter and the input folder name is taken as sample name.
+ +This is a light version of NanoPreprocess that does not perform the basecalling step. It allows to make the same analysis starting from basecalled reads in fastq format. You can also provide fast5 files if you need to demultiplex using DeepLexiCon.
+ +This module will allow to run the pipeline on multiple input samples by using this syntax in the params.file:
+ +fastq = "$baseDir/../../../org_data/**/*.fastq.gz"
+In this way it will produces a number of output files with the same sample name indicated by the two asterisks.
+ + + +
+
+ This module allows to estimates polyA sizes by using two different methods (nanopolish and talifindr). it reads directly the output produced by NanoPreprocess and in particular it needs the read counts / assignment.
+ +
Three folders are created by the pipeline within the output folder:
+"Read name" "Tailfindr" "Nanopolish" "Gene Name"
+"013a5dde-9c52-4de1-83eb-db70fb2cd130" 52.16 49.39 "YKR072C"
+"01119f62-ca68-458d-aa1f-cf8c8c04cd3b" 231.64 274.28 "YDR133C"
+"0154ce9c-fe6b-4ebc-bbb1-517fdc524207" 24.05 24.24 "YFL044C"
+"020cde28-970d-4710-90a5-977e4b4bbc46" 41.27 56.79 "YGL238W"
+A plot is also produced for showing the correlation between the two methods.
+ + +