show.Rmd

---
title: "<small>Towards</small> a reproducible and reusable publication and analysis workflow"
author:
- Rad Suchecki
- Nathan Watson-Haigh
- Stuart Stephen
- Alex Whan
#subtitle: "Subtitle"
date: "<center> `r format(Sys.time(), '%A, %d %B, %Y')` </center>"
#tags: [nothing, nothingness]
output:
  #pdf_document:
    #default
  html_document:
    toc: true
    toc_float: true
    number_sections: true
    theme: yeti
    highlight: tango
  beamer_presentation:
    highlight: default #specifies the syntax highlighting style. Supported styles include “default”, “tango”, “pygments”, “kate”, “monochrome”, “espresso”, “zenburn”, and “haddock” (specify null to prevent syntax highlighting)
    toc: true
    slide_level: 2
  revealjs::revealjs_presentation:
    theme: beige #“default”, “simple”, “sky”, “beige”, “serif”, “solarized”, “blood”, “moon”, “night”, “black”, “league” or “white”
    highlight: default # “default”, “tango”, “pygments”, “kate”, “monochrome”, “espresso”, “zenburn”, and “haddock”. Pass null to prevent syntax highlighting.
    center: false #specifies whether you want to vertically center content on slides (this defaults to false).
    transition: slide #"default”, “fade”, “slide”, “convex”, “concave”, “zoom” or “none”.
    incremental: false
    css: reveal.css
    slide_level: 1 #default=2
    self_contained: true
    reveal_options:
      slideNumber: true
      margin: 0.1
      width: 1280
      height: 800
      preview_links: true
  slidy_presentation:
    font_adjustment: -1
  ioslides_presentation:
    widescreen: true
    transition: faster
---

## Reproducible manuscript - why?

<div class="column-left">

[![https://xkcd.com/242/](https://imgs.xkcd.com/comics/the_difference.png){ width=80% }](https://xkcd.com/242/)

<small> [https://xkcd.com/242/](https://xkcd.com/242/) </small>
</div>

<div class="column-right">

> * To avoid errors
>   * Widely reported inconsistencies between results and the methodology reported
> * To promote computational reproducibility
>   * Other people (and you!) can take your data and get the same numbers that are in your paper
>   * Document must specify where ALL the numbers come from
>   * Otherwise numbers hard to recover even in absence of errors
> * To create documents which can be revised easily
>   * New data, updated software or requests from reviewers can be incorporated much more easily

<!--[inspiration](https://github.com/libscie/rmarkdown-workshop)-->

</div>

----


## Ingredients for reproducibility in data science

* data

```{bash, comment='', echo=FALSE}
#curl -s ftp://ftp.ensemblgenomes.org/pub/plants/release-38/fasta/arabidopsis_thaliana/dna/Arabidopsis_thaliana.TAIR10.dna.toplevel.fa.gz | gunzip --stdout | head
head data/At.head.fa
```

* code

```{bash, comment='', echo=FALSE}
head bin/extractStatsFromBAM.sh
```

* compute environment 

```{R, comment='', echo=FALSE}
sessionInfo()
```


----

## BioKanga {incremental=false}

- A suite of bioinformatics tools developed at CSIRO
- [https://github.com/csiro-crop-informatics/biokanga](https://github.com/csiro-crop-informatics/biokanga)


### Aims

  * evaluation of BioKanga's sequence alignment module vs state-of-the-art tools
  * turn-key reproducibility
  * re-usability

<div class="column-right">
<!--![](https://openclipart.org/download/192541/pushthebutton.svg){width=33%}-->
![](figs/pushthebutton.svg){width=33%}
</div>

----

## Means

* open source and version control from day 1
  * [https://github.com/csiro-crop-informatics/reproducible_poc](https://github.com/csiro-crop-informatics/reproducible_poc)
* focus on design, not immediate result
* workflow management
* containers

### Ongoing

* automated manuscript content generation
  * tables, figures... not AI for writing text

### Planned

  * CI
  * automated builds 

----

## Plans vs reality

- we have developed a modular workflow using ~~two popular automation frameworks, Snakemake and~~ [Nextflow](https://www.nextflow.io/)
- Individual tasks are executed in Cloud or HPC environment using dedicated, ~~lightweight~~ (Docker/Singularity) containers
- Raw data is acquired from public sources, processed and analysed
- Final document including dynamically generated figures and tables is compiled from R Markdown <strike> or LaTeX/knitr </strike>  (minimal content at the moment, but works in principle)

----

## [Nextflow](https://www.nextflow.io/) 

### A Workflow management system which promises 

  * Portability
  * Scalability
  * Reproducibility

### Features 

  * Separation of pipeline logic from compute environment definition(s)
  * Syntax is a superset of Groovy, but
  * Polyglot
    * mixing scripting languages, as long as interpreter available on system
    
    
  * Multiple _executors_
    * SGE, LSF, SLURM, PBS/Torque, NQSII HTCondor, Ignite, Kubernets, AWS Batch

<small>[https://www.nature.com/articles/nbt.3820/](https://www.nature.com/articles/nbt.3820)</small>


----

## Pipeline flow

```{bash, echo = FALSE}
dot -Tpng -odoc/flowchart.png doc/flowchart.dot
dot -Tpng -Gsize=9,7\! -Gdpi=100 -odoc/flowchart_small.png doc/flowchart.dot
```

[![Flowchart](doc/flowchart_small.png)](flowchart.png)

----

## Pipeline syntax

* Primarily definitions of processes, each with well defined input and output channels

```{bash, comment=''}
cat main.nf
```

----

## Compute profiles 

### Implemented

- `standard` (i.e. local/interactive assuming all software available)
- `slurm`
- `modules`
- `docker`
- `singularity`
- `ec2`

```{bash, eval = FALSE}
# some of the profiles can, and others are intended to be used in concert, e.g. 
nextflow run main.nf -profile slurm,modules
# or 
nextflow run main.nf -profile slurm,singularity,singularitymodule
```

### Upcoming:

* `k8s`, `awsbatch`

----


## Modules

* Required modules are defined separately and explicitly for each process (or associated with labels)

```{bash, comment=''}
cat conf/modules.config
```

* Unfortunately, these are compute environment specific

----

## Containers

* Required containers are defined separately and explicitly for each process (or associated with labels)
* Available on docker hub
  * Used both by `docker` and `singularity`
  * Always specify version tag (avoid `':latest'`)

```{bash, comment=''}
cat conf/containers.config
```

----

## Generating reports/publications/slides from R Markdown 

![Tool-chain](https://d33wubrfki0l68.cloudfront.net/61d189fd9cdf955058415d3e1b28dd60e1bd7c9b/9791d/images/rmarkdownflow.png)

### Some of the supported output formats:

* HTML* documents and slide-shows using 
  * Slidy, ioslides, **reveal.js**
* PDF
  * LaTeX
  * *LaTeX Beamer slides*
* MS Word, ODT, RTF 
* Markdown, and Github flavoured Markdown documents

----

## [These slides](https://github.com/csiro-crop-informatics/reproducible_poc/blob/develop/show.Rmd)

* Source: [show.Rmd](https://github.com/csiro-crop-informatics/reproducible_poc/blob/develop/show.Rmd)
* Tools and packages:
  * R, Pandoc, `rmarkdown`, `revealjs`, `kableExtra`
* Use R, RStudio or a simple `RScript` to render from the command line


```{bash, eval = FALSE}
./render.R show.Rmd "revealjs::revealjs_presentation"
```

```{bash, comment='', echo = FALSE}
cat render.R
```

* Output: [https://rsuchecki.github.io/reproducible.github.io/show.html](https://rsuchecki.github.io/reproducible.github.io/show.html)

----

## Report generation

* ### Design conundrum
  * ~~Pipeline to render the document~~
  * Document rendering to execute pipeline
  * Run pipeline, then render the document
* ### Detail 
  * output from the tested tools evaluated against ground truth
  * speed benchmarking
* ### Workflow introspection 
  * Nextflow offers detailed execution [reports](https://rsuchecki.github.io/reproducible.github.io/report.html) and [timelines](https://rsuchecki.github.io/reproducible.github.io/timeline.html), which could be incorporated or linked to in the generated report/manuscript

----

## Tables and figures dynamically generated from collected results

```{R}
dt <- read.table("results/stats/allStats", header=TRUE, sep="\t")
#kable(dt[with(dt, order(MatchRate, decreasing=TRUE)), ]) %>% 
kable(dt) %>% 
scroll_box(height = "400px") #add width for horizontal scroll, can use % as well
```

----

## Further Work (I)

### Document generation refinement 

* R Markdown rendering not as flexible as hoped
* Generation of output in different formats somewhat limited, for more complex content where pure (R)Markdown is insufficient have to decide on either 
  * (R)Markdown with LaTeX 
  * (R)Markdown with HTML

### Workflow 

* Comparison of Nextflow and Snakemake (limited, in context)
* Tagged containers for 
  * each release of the pipeline
  * submitted/revised/final publication

----

## Further Work (II)

* More systematic evaluation of BioKanga alignment module vs other tools
  * At default settings
  * Exploration of parameter space, optimisation
  * More alignment tools to be included


----

[![Flowchart](doc/flowchart_small.png)](flowchart.png)


----

## (Some of) the good

* [https://gitlab.com/makhlaghi/reproducible-paper](https://gitlab.com/makhlaghi/reproducible-paper) (`make` and `LaTeX`)
* [https://github.com/Robinlovelace/rmarkdown-paper-repo](https://github.com/Robinlovelace/rmarkdown-paper-repo)
* [https://github.com/libscie/rmarkdown-workshop](https://github.com/libscie/rmarkdown-workshop)
* [https://andrewgoldstone.com/blog/2015/05/27/rmd-slides/](https://andrewgoldstone.com/blog/2015/05/27/rmd-slides/)

## The bad & the ugly

<div class="column-right">

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">4 pages of this. Images pasted into a word document! <a href="https://t.co/CkVpLutZZV">pic.twitter.com/CkVpLutZZV</a></p>&mdash; Paul Gardner (@ppgardne) <a href="https://twitter.com/ppgardne/status/1000582103141007361?ref_src=twsrc%5Etfw">May 27, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

</div>

<!--<div class="column-farright">-->

<!--<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">The &quot;Journal of Irreproducible Results&quot; was a joke, but actually that&#39;s what most journals really are. <a href="http://t.co/TiLh2f5JZH">http://t.co/TiLh2f5JZH</a></p>&mdash; Sean Carroll (@seanmcarroll) <a href="https://twitter.com/seanmcarroll/status/424596260893364225?ref_src=twsrc%5Etfw">January 18, 2014</a></blockquote>-->
<!--<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>-->

<!--</div>-->





<!--Credit: https://markdownmonster.west-wind.com/docs/_4sf0may9m.htm#markdown-code-output-->
<!--<link href="scripts/highlightjs/styles/github-gist.css" rel="stylesheet" />-->
<!--<script src="scripts/highlightjs/highlight.pack.js"></script>-->

<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.11.0/styles/default.min.css">
<script src="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.11.0/highlight.min.js"></script>

<script>
function highlightCode() {
    var pres = document.querySelectorAll("pre>code");
    for (var i = 0; i < pres.length; i++) {
        hljs.highlightBlock(pres[i]);
    }
}
highlightCode();
</script>