+ {{ content }}
+
+
+ 
+
+
+ {% include search.html %}
+
+Page not found :(
+The requested page could not be found.
diff --git a/MOP1.1/docs/_config.yml b/MOP1.1/docs/_config.yml new file mode 100644 index 0000000..da3939e --- /dev/null +++ b/MOP1.1/docs/_config.yml @@ -0,0 +1,13 @@ +title: Master of Pores +description: Nextflow pipeline for analysis of Nanopore data from direct RNA sequencing. +url: "https://biocorecrg.github.io/master_of_pores/" + +# Build settings +markdown: kramdown +kramdown: + syntax_highlighter_opts: + disable : true + +sass: + sass_dir: _sass + style: compressed diff --git a/MOP1.1/docs/_includes/head.html b/MOP1.1/docs/_includes/head.html new file mode 100644 index 0000000..0e16731 --- /dev/null +++ b/MOP1.1/docs/_includes/head.html @@ -0,0 +1,13 @@ + + + + + +
+
+
+
+
+* **index_reference** index the reference file for Epinano
+* **call_variants** uses Samtools for calling the variants for Epinano
+* **calc_var_frequencies** it uses TSV_to_Variants_Freq.py3 for calculating the frequencies of each variants for Epinano
+* **predict_with_EPInano** It predicts the modifications with Epinano in parallel splitting the input file in 1 million rows
+* **combineEpinanoPred** It combine the results from Epinano
+* **resquiggling** resquiggle fast5 files for Tombo
+* **getModifications** it estimates the modifications using Tombo comparing WT vs KO
+
+## Input Parameters
+1. **input_path** path to the folders produced by NanoPreprocessing step.
+1. **comparison** tab separated text file containing the list of comparison. Here an example:
+
+```bash
+WT1 KO1
+WT2 KO2
+WT3 KO3
+```
+
+1. **reference** reference transcriptome
+1. **output** folder
+1. **coverage** read coverage threshold for prediction
+1. **tombo_opt** options for tombo
+1. **epinano_opt** options for epinano
+1. **email**
+
+## Results
+Three folders are produced by this module:
+
+1. Epinano, containing the results obtained with this method. You have a single file with putative modifications:
+
+```bash
+#Kmer,Window,Ref,Coverage,q1,q2,q3,q4,q5,mis1,mis2,mis3,mis4,mis5,ins1,ins2,ins3,ins4,ins5,del1,del2,del3,del4,del5,prediction,dist,ProbM,Pro
+bU
+AGTGG,394404:394405:394406:394407:394408,chr2,8.0:8.0:7.0:7.0:7.0,21.5,21.25,19.857,23.0,16.285999999999998,0.0,0.0,0.0,0.0,0.0,0.0,0.062,0.0
+71,0.0,0.0,0.0,0.0,0.0,0.0,0.0,unm,19.26143361547619,3.00000089999998e-14,0.9999999999999699
+TTTTT,12150:12151:12152:12153:12154,chr8,3.0:3.0:3.0:3.0:3.0,0.0,16.5,18.5,16.0,16.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.3329999999
+9999996,0.33299999999999996,0.0,0.0,unm,2.5976484688977424,0.06071658133381308,0.9392834186661868
+ACATT,438165:438166:438167:438168:438169,chr13,67.0:67.0:67.0:68.0:68.0,13.635,13.446,9.323,9.6,12.127,0.03,0.045,0.015,0.147,0.0740000000000
+0001,0.0,0.0,0.0,0.0,0.0,0.06,0.03,0.075,0.11800000000000001,0.07400000000000001,unm,0.08435556637195174,0.519879422458087,0.4801205775419129
+5...
+```
+
+and three plots in pdf indicating possible events related to insertion, deletion and mismatches, see the example below.
+
+
+
+
+2. Tombo, containing the results obtained with this method in fasta format. You have one file for each comparison WT vs KO
+
+```bash
+>chr11:455562:- Est. Frac. Alternate: 0.98
+TGACA
+>chr12:1008723:- Est. Frac. Alternate: 0.98
+TATCT
+>chr15:491587:+ Est. Frac. Alternate: 0.96
+TATAT
+>chr10:425794:- Est. Frac. Alternate: 0.95
+ATGTT
+>chr13:510759:+ Est. Frac. Alternate: 0.95
+...
+```
+And for convenience a 6 bed files with the coordinates of the event.
diff --git a/MOP1.1/docs/nanomod_pl.png b/MOP1.1/docs/nanomod_pl.png
new file mode 100644
index 0000000..3fb0bed
Binary files /dev/null and b/MOP1.1/docs/nanomod_pl.png differ
diff --git a/MOP1.1/docs/nanopreprocess.md b/MOP1.1/docs/nanopreprocess.md
new file mode 100644
index 0000000..7411957
--- /dev/null
+++ b/MOP1.1/docs/nanopreprocess.md
@@ -0,0 +1,156 @@
+---
+layout: page
+title: NanoPreprocess
+navigation: 3
+---
+
+# NanoPreprocess
+
+This module takes as input the raw fast5 reads - single or multi - and produces a number of outputs (basecalled fast5, sequences in fastq format, aligned reads in BAM format etc). The pre-processing module performs base-calling, demultiplexing (optional), filtering, quality control, mapping to a genome / transcriptome reference, feature counting and it generates a final report of the performance and results of each of the steps performed. It automatically detects the kind of input fast5 file (single or multi sequence).
+
+
+## Workflow
+
+
+
+| Process name | Description |
+| ------------- | ------------- |
+|**testInput**|Detection of kind of fast5 (multi or single)|
+|**baseCalling**|Basecalling with *Albacore* or *Guppy* (up to guppy 4.0)|
+|**demultiplexing**|Demultiplexing (optional)
+|**concatenateFastQFiles**|This process concatenates the fastq files produces for each single basecalling |
+|**QC**|Performed with *MinIONQC*|
+|**fastQC**|Executed on fastq files|
+|**mapping**|Mapping to genome / transcriptome with either *minimap2*, *graphmap* or*graphmap2*|
+|**counting**|If mapping to the genome, it obtains counts per gene with *htseq-count*. Otherwise, if mapping to the transcriptome, transcript counts are generated with *NanoCount*. Reads are also assigned to a gene or to a transcript if they are uniquely mapping. A report file is also generated.|
+|**alnQC2**|QC of aligned reads with *NanoPlot*. The plots PercentIdentityvsAverageBaseQuality_kde, LengthvsQualityScatterPlot_dot, HistogramReadlength and Weighted_HistogramReadlength are then merged together in a single picture.|
+|**alnQC**|QC of aligned reads with *bam2stats*.|
+|**cram_conversion**|Generating cram file from alignment.|
+|**joinAlnQCs**|Merging the QC files generated by the alnQC step.|
+|**joinCountQCs**|Merging the report files generated by the counting step.|
+|**multiQC**|Final report generation - enventually sent by mail to the user too.|
+
+
+## Input Parameters
+
+| Parameter name | Description |
+| ------------- | --------------|
+|**fast5 files**|Path to fast5 input files (single or multi-fast5 files). They should be inside a folder that will be used as sample name. **[/Path/sample_name/*.fast5]**|
+|**reference**|File in fasta format. **[Reference_file.fa]**|
+|**ref_type**| Specify if the reference is a genome or a transcriptome. **[genome / transcriptome]** |
+|**kit**|Kit used in library prep - required for basecalling.|
+|**flowcell**|Flowcell used in sequencing - required for basecalling. |
+|**annotation**|Annotation file in GTF format. It is optional and needed only in case of mapping to the genome and when interested in gene counts. **[Annotation_file.gtf]** |
+|**seq_type**| Sequence type. **[RNA / DNA]** |
+|**output**|Output folder name. **[/Path/to_output_folder]**|
+|**granularity**|indicates the number of input fast5 files analyzed in a single process. It is by default 4000 for single-sequence fast5 files and 1 for multi-sequence fast5 files. In case **GPU** option is turned on this value is not needed since every file will be analyzed sequentially.|
+|**basecaller**|Algorithm to perform the basecalling. guppy or albacore are supported. **[albacore / guppy]**|
+|**basecaller_opt**| Command line options for basecaller program. Check available options in respective basecaller repository.|
+|**GPU**| Allow the pipeline to run with GPU. **[OFF / ON]**|
+|**demultiplexing**| Demultiplexing algorithm to be used. **[OFF / deeplexicon / guppy / guppy-readucks]**|
+|**demultiplexing_opt**|Command line options for the demultiplexing software. |
+|**demulti_fast5**| If performing demultiplexing, also generate demultiplexed multifast5 files. **[OFF / ON]**|
+|**filter**| Program to filter fastq files. **[nanofilt / OFF]**|
+|**filter_opt**| Command line options of the filtering program. |
+|**mapper**| Mapping algorithm. **[minimap2 / graphmap / graphmap2]** |
+|**mapper_opt**| Command line options of the mapping algorithm. |
+|**map_type**|Spliced - recommended for genome mapping - or unspliced - recommended for transcriptome mapping. **[spliced / unspliced]**|
+|**counter**| Generating gene counts (genome mapping) or transcript counts (transcriptome mapping). **[YES / ""]**|
+|**counter_opt**|Command line options of the counter program: NanoCount for transcripts and Htseq-count for genes.|
+|**email**| Users email for receving the final report when the pipeline is finished. **[user_email]**|
+
+
+You can change them by editing the **params.config** file or using the command line - please, see next section.
+
+## How to run the pipeline
+Before launching the pipeline, user should decide which containers to use - either docker or singularity **[-with-docker / -with-singularity]**.
+Then, to launch the pipeline, please use the following command:
+```bash
+nextflow run nanopreprocess.nf -with-singularity > log.txt
+```
+
+
+ 1. **check_reference** It verifies whether the reference is zipped and eventually unzip it
+ 1. **tailfindr** it runs *tailfindr* tool in parallel.
+ 1. **collect_tailfindr_results** It collects the results of tailfindr.
+ 1. **filter_bam** Bam files are filtered with *samtools* to keep only mapped reads and remove secondary alignments
+ 1. **tail_nanopolish** It runs *nanopolish* in parallel.
+ 1. **collect_nanopolish_results** It collects the results of tail_nanopolish.
+ 1. **join_results** It merges the results from the two algorithms and make a plot of the correlation.
+
+
+## Input Parameters
+
+1. **input_folders** path to the folders produced by NanoPreprocessing step.
+1. **nanopolish_opt** options for the nanopolish program
+1. **tailfindr_opt** options for the tailfindr program
+1. **reference** reference genome / transcriptome
+1. **output** folder
+1. **email**
+
+
+## Results
+Three folders are created by the pipeline within the output folder:
+1. NanoPolish: contains the output of *nanopolish* tool.
+1. Tailfindr: contains the output of *tailfindr* tool.
+1. PolyA_final: contains the txt files with the combined results (i.e. predicted polyA sizes). Here an example of a test:
+
+```bash
+"Read name" "Tailfindr" "Nanopolish" "Gene Name"
+"013a5dde-9c52-4de1-83eb-db70fb2cd130" 52.16 49.39 "YKR072C"
+"01119f62-ca68-458d-aa1f-cf8c8c04cd3b" 231.64 274.28 "YDR133C"
+"0154ce9c-fe6b-4ebc-bbb1-517fdc524207" 24.05 24.24 "YFL044C"
+"020cde28-970d-4710-90a5-977e4b4bbc46" 41.27 56.79 "YGL238W"
+```
+A plot is also produced for showing the correlation between the two methods.
diff --git a/MOP1.1/docs/output.md b/MOP1.1/docs/output.md
new file mode 100644
index 0000000..1e4cfff
--- /dev/null
+++ b/MOP1.1/docs/output.md
@@ -0,0 +1,21 @@
+## Output generated by the MasterOfPores pipeline
+The final outputs of the pipeline include:
+
+From NanoPreprocess (Module 1):
+- Basecalled fast5 files within the “fast5_files” folder.
+- Filtered fastq files within “fastq_files” folder.
+- QC reports within “QC” folder.
+- Final report within “report” folder.
+- Aligned reads in BAM files within the “aln” folder.
+- Read counts within the "count" folder.
+
+
+From NanoMod (Module 2):
+- Plain TXT files with predicted RNA modifications in the “rna_mods” folder
+
+
+From NanoTail (Module 3):
+- Plain TXT files with polyA tail length estimations in the “polyA_length” folder
+
+
+
diff --git a/MOP1.1/docs/running.md b/MOP1.1/docs/running.md
new file mode 100644
index 0000000..5cbe62c
--- /dev/null
+++ b/MOP1.1/docs/running.md
@@ -0,0 +1,197 @@
+# Running the pipeline
+The pipeline is composed of three different modules within three different folders:
+
+1. **NanoPreprocess** containing the preprocessing steps that will extract the data from raw fast5 files
+1. **NanoTail** that will use the results produced by NanoPreprocess for estimating polyA sizes.
+1. **NanoMod** that will use the results produced by NanoPreprocess for estimating checmically modified sites.
+
+Each module relies on tools installed within linux containers which recipes are stored within the **docker** folder. In case the *singularity* engine is chosen a number of images will be automatically generated and placed within the **singularity** folder.
+
+## NanoPreprocess
+
+Input files are either multifast5 or single fast5 files containing reads from direct RNA sequencing.
+They will be basecalled and eventually demultiplexed and aligned to a reference sequence (genome or transcriptome).
+
+### Steps
+ 1. **testInput** Detection of kind of fast5 (multi or single)
+ 1. **baseCalling** Basecalling with *Albacore* or *Guppy*
+ 1. **demultiplexing_with_deeplexicon** Demultiplexing (optional) with **DeePlexiCon**
+ 1. **concatenateFastQFiles** This process concatenates the fastq files produces for each single basecalling
+ 1. **QC** performed with *MinIONQC*
+ 1. **fastQC** on fastq files
+ 1. **mapping** to the genome / transcriptome with either *minimap2* or *graphmap2*
+ 1. **counting** counts per gene, if mapping to the genome with *htseq-count*, or per transcript if mapping to the transcriptome with *NanoCount*. Reads are also assigned to a gene or to a transcript if they are uniquely mapping. A report file is also generated
+ 1. **alnQC** QC of aligned reads with *bam2stats*.
+ 1. **joinCountQCs** This process is for merging the report files generated by the counting step.
+ 1. **joinAlnQCs** This process is for merging the QC files generated by the alnQC step.
+ 1. **alnQC2** QC of aligned reads with *NanoPlot*. The plots PercentIdentityvsAverageBaseQuality_kde, LengthvsQualityScatterPlot_dot, HistogramReadlength and Weighted_HistogramReadlength are then merged together in a single picture.
+ 1. **multiQC**. Final repor enventually sent by mail too.
+
+
+You can launch the pipeline choosing either the parameter **-with-singularity** or **-with-docker** depending on which containers you want to use:
+
+### Input Parameters
+
+1. **fast5 files**. Path to fast5 input files. They can contain either a single sequence or multiple ones. They should be inside a folder that will be used as sample name.
+1. **reference** file in fasta format. It can be either a genome or a transcriptome. this must be specified via **ref_type** parameter.
+1. **kit** and **flowcell** parameters needed for basecalling.
+1. **annotation** in GTF format. It is optional and needed only in case of mapping to the genome and when interested in gene counts.
+1. **seq_type**, It can be either RNA or DNA.
+1. **output** output folder name
+1. **granularity** indicates the number of input fast5 files analyzed in a single process. It is by default 4000 for single-sequence fast5 files and 1 for multi-sequence fast5 files. In case **GPU** option is turned on this value is not needed since every file will be analyzed sequentially.
+1. **basecaller** program. guppy or albacore are supported.
+1. **basecaller_opt** command line options for basecaller program
+1. **GPU** it allows using GPU or not. I can be either OFF or NO
+1. **demultiplexing** program. It is supported only deeplexicon. It can be turned off by specifying "OFF"
+1. **demultiplexing_opt** options for the demultiplexing program.
+1. **filter** it can be NanoFilt or OFF is filtering is needed.
+1. **filter_opt** options of the filtering program.
+1. **mapper** it can be either minimap2 or graphmap2
+1. **mapper_opt** options of the mapping program.
+1. **map_type** it can be either spliced or not. In case the alignment is to a eukaryotic genome it should be spliced.
+1. **count** this parameter can be YES for counting the number of tags per gene (in case of mapping to the genome) or per transcript (in case of mapping to the transcriptome). An annotation file is needed in case of mapping to the genome.
+1. **counter_opt** options of the counter program: NanoCount for transcripts and Htseq-count for genes.
+1. **email** for receving a mail with the final report when the pipeline is finished
+
+
+You can change them by editing the **params.config** file or using the command line (each param name needs to have the characters **--** before):
+
+```bash
+nextflow run nanopreprocess.nf -with-singularity -bg --output test2 > log.txt
+```
+
+To resume a previous execution that failed at a certain step or if you change a parameter that affects only some steps you can use the **Netxtlow** parameter **-resume** (only one dash!):
+
+
+```bash
+nextflow run nanopreprocess.nf -with-singularity -bg -resume > log.txt
+
+...
+
+[warm up] executor > crg
+[e8/2e64bd] Cached process > baseCalling (RNA081120181_1)
+[b2/21f680] Cached process > QC (RNA081120181_1)
+[c8/3f5d17] Cached process > mapping (RNA081120181_1)
+...
+
+```
+
+### Results:
+
+Seven folders are created by the pipeline within the output folder specified by the **output** parameter:
+
+1. fast5_files: contains the basecalled multisequence fast5 files. Each batch contains 4000 sequences.
+1. fastq_files: contains one or, in case of demultiplexing, more fastq files.
+1. QC_files: contains each single QC produced by the pipeline.
+1. alignment: contains the bam file(s)
+1. counts: contains read counts per gene / transcript. It is optional.
+1. assigned: contains assignment of each read to a given gene / transcript. It is optional.
+1. report: contains the final multiqc report.
+
+-----------------------------------------------------
+
+
+## NanoTail
+This module allows to estimates polyA sizes by using two different methods. Data produced by NanoPreprocess are needed and in particular the read counts / assignment must be given.
+
+### Steps
+
+ 1. **check_reference** It verifies whether the reference is zipped and eventually unzip it
+ 1. **tailfindr** it runs *tailfindr* tool in parallel.
+ 1. **collect_tailfindr_results** It collects the results of tailfindr.
+ 1. **filter_bam** Bam files are filtered with *samtools* to keep only mapped reads and remove secondary alignments
+ 1. **tail_nanopolish** It runs *nanopolish* in parallel.
+ 1. **collect_nanopolish_results** It collects the results of tail_nanopolish.
+ 1. **join_results** It merges the results from the two algorithms and make a plot of the correlation.
+
+
+### Input Parameters
+
+1. **input_folders** path to the folders produced by NanoPreprocessing step.
+1. **nanopolish_opt** options for the nanopolish program
+1. **tailfindr_opt** options for the tailfindr program
+1. **reference** reference genome / transcriptome
+1. **output** folder
+1. **email**
+
+
+### Results
+Three folders are created by the pipeline within the output folder:
+1. NanoPolish: contains the output of *nanopolish* tool.
+1. Tailfindr: contains the output of *tailfindr* tool.
+1. PolyA_final: contains the txt files with the combined results (i.e. predicted polyA sizes). Here an example of a test:
+
+```bash
+"Read name" "Tailfindr" "Nanopolish" "Gene Name"
+"013a5dde-9c52-4de1-83eb-db70fb2cd130" 52.16 49.39 "YKR072C"
+"01119f62-ca68-458d-aa1f-cf8c8c04cd3b" 231.64 274.28 "YDR133C"
+"0154ce9c-fe6b-4ebc-bbb1-517fdc524207" 24.05 24.24 "YFL044C"
+"020cde28-970d-4710-90a5-977e4b4bbc46" 41.27 56.79 "YGL238W"
+```
+A plot is also produced for showing the correlation between the two methods.
+
+## NanoMod
+This module allows to predict the loci with RNA modifications. Data produced by NanoPreprocess are needed and in particular the reads must be aligned to the transcriptome.
+
+### Steps
+1. **index_reference** index the reference file for Epinano
+1. **call_variants** uses Samtools for calling the variants for Epinano
+1. **calc_var_frequencies** it uses TSV_to_Variants_Freq.py3 for calculating the frequencies of each variants for Epinano
+1. **predict_with_EPInano** It predicts the modifications with Epinano
+1. **filter_EPInano_pred** It filers the results from Epinano using replicates if avialable
+1. **resquiggling** resquiggle fast5 files for Tombo
+1. **getModifications** it estimates the modifications using Tombo comparing WT vs KO
+1. **cross_tombo_pred** it gets the intersection between differen replicates
+1. **join_results** it gets the intersection between the Epinano and Tombo predictions
+
+### Input Parameters
+1. **input_folders** path to the folders produced by NanoPreprocessing step.
+1. **comparison** tab separated text file containing the list of comparison. Here an example:
+```bash
+WT1 KO1
+WT2 KO2
+WT3 KO3
+```
+1. **reference** reference transcriptome
+1. **output** folder
+1. **tombo_opt** options for tombo
+1. **epinano_opt** options for epinano
+1. **tombo_score** score for filtering reliable modifications (from 0.5 to 1)
+1. **epinano_score** coverage score for epinano for filtering reliable modifications (integer)
+1. **email**
+
+### Results
+Three folders are produced by this module:
+
+1. Epinano, containing the results obtained with this method. You have a single file with already filtered modifications.
+
+```bash
+geneA,126771,GAACT,5.0,0.7865299392210111,6.0,7.650999662942007e-06,YES
+geneA,139467,AGACA,26.0,1.2631007354786662e-05,34.0,9.22202894828834e-14,YES
+geneA,139625,AGACA,17.0,0.012299404049052885,20.0,4.64573431891912e-06,YES
+geneA,192033,AGACC,11.0,1.849874054901369e-12,11.0,3.00000089999998e-14,YES
+geneA,192201,AGACA,14.0,0.01469732206992497,16.0,3.00000089999998e-14,YES
+...
+```
+2. Tombo, containing the results obtained with this method. You have one file for each comparison WT vs KO and a final one, **tombo_all.txt**, with the intersection after filtering per score. Here an example of tombo_all.txt file:
+
+```bash
+>geneA:549289:+
+CTGAC
+>geneA:478105:+
+GAGCT
+>geneA:426607:-
+TTTTT
+...
+```
+
+3. Comb_mod, containing the all the modifications found in Epinano and Tombo called **RNA_modifications.txt** and a Venn Diagram. Here an example of RNA_modifications.txt file:
+
+
+```bash
+"positions" "epinano" "tombo"
+"chrIX-290356" "1" "1"
+"chrX-274861" "1" "1"
+"chrXV-513509" "1" "1"
+"chrI-126771" "1" "0"
+```