non quickstart users

This is the full set of documentation that will walk you through the full installation process and the various options you have for running tools/workflows.

Getting Started

Download the Consonance Bash script from github. Put it on your PATH and run it to automatically download the client-jar (requirement of Java 8) and set up a config file.

Basic documentation is available through the command-line.

ubuntu@consonance-user:~$ consonance

Usage: consonance [<flag>]
  consonance <command> [--help]

Commands:
  run           Schedule a job
  status        Get the status of a job

Flags:
  --help        Print help out
  --debug       Print debugging information
  --version     Print Consonance's version
  --metadata    Print metadata environment

Config File Format

The Consonance command-line has a small config file indicating which Consonance web service it connects with and what files to instantiate on every worker host that is created.

[webservice]
base_path = http://foo.bar:8080
token = foobar 

extra_files =/root/.aws/credentials=/home/ubuntu/.aws/config=false,node-engine.cwl=node-engine.cwl=true

Scheduling a Workflow

consonance run  --flavour m1.xlarge \
    --image-descriptor collab.cwl \
    --run-descriptor collab-cwl-job-pre.json \
    --extra-file node-engine.cwl=node-engine.cwl=true \
    --extra-file /root/.aws/config=/home/ubuntu/.aws/config=false

This schedules a workflow to run on an instance-type of m1.xlarge, the workflow that we will run is described by collab.cwl, the workflow run (i.e. inputs and outputs) is described by collab-cwl-job-pre.json, we will also be loading a file called node-engine.cwl into the working directory, and an AWS credential file at ~root/.aws/config (necessary for uploading files to S3).

For the extra files, note that the format is destination=source=(whether we should retain this information).

See the format for CWL on github.

Insert --quiet after consonance to run in quiet mode.

Note that if you use the same extra files over and over again (e.g. all your workflows require provisioning to S3) you can describe them globally for all workflow runs in your ~/.consonance/config. The format of this file is above

Finally, Consonance relies upon a standalone launcher for SeqWare and CWL tools which can be run independently. Some configuration for this tool is possible and the config file format for it is defined here

To override, simply send a custom config file to the working directory. For example, to send a custom config file in order to send results to an alternate S3 api, you will need the following in your global config:

extra_files = cwl-launcher.config=/home/ubuntu/cwl-launcher.config=true

Check on the status of a workflow

consonance status --job_uuid foobar-foobar-foobar

This checks on the status of a workflow.

Extract individual fields (such as the stderr) with the following one-liner:

consonance status --job_uuid foobar-foobar-foobar | python -c 'import sys, json; print json.load(sys.stdin)["stderr"]'

File Provisioning

When specifying input files in the run descriptor ( collab-cwl-job-pre.json above ) ftp, http, https, local files can all be used out of the box with no credentials (as long as they are not protected). However, special credentials and config files are required for file provisioning from more exotic locations like those listed below.

S3 supports both input and output files.

DCC-storage supports only input files.

S3 on AWS

Input files from AWS S3 and and output files from AWS S3 can be provisioned if you provide a ~/.aws/config with an access and secret key. The format for this file is provided in detail at on AWS but an example would be as follows:

ubuntu@consonance-user:~/test_s3_on_collab$ cat ~/.aws/config 
[default]                                                                                    
aws_access_key_id     = foobar
aws_secret_access_key = foobar                             
region=us-east-1  

You can attach them on a per-job basis as shown below or define them globally in your config file.

--extra-file /root/.aws/config=aws_config=false

Entries in the json look as follows:

{
  "bam_input": {
        "class": "File",
        "path": "ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/NA12878/alignment/NA12878.chrom20.ILLUMINA.bwa.CEU.low_coverage.20121211.bam"
    },
    "bamstats_report": {
        "class": "File",
        "path": "s3://oicr.temp/testing-launcher/bamstats_report.zip"
    }
}

S3 Api on Ceph (cancercollaboratory.org)

For alternative endpoints that support the S3 API, you can configure the endpoint you wish to use in the dockstore descriptor launcher config.

--extra-file /root/.aws/config=aws_config=false --extra-file cwl-launcher.config=cwl-launcher.config=true

cwl-launcher will look like the following

[s3]
endpoint = https://www.cancercollaboratory.org:9080

The format of the AWS credentials matches AWS S3, but get an access code and secret key from your system administrator rather than from AWS.

DCC-Storage on Ceph (cancercollaboratory.org)

For the DCC-Storage system, you will need to provide a properties file for the DCC:

--extra-file /icgc/dcc-storage/conf/application-amazon.properties=application-amazon.properties=false

application-amazon.properties looks like the following:

logging.level.org.icgc.dcc.storage.client=DEBUG
logging.level.org.springframework.web.client.RestTemplate=DEBUG
client.upload.serviceHostname=storage.cancercollaboratory.org
client.ssl.trustStore=classpath:client.jks
accessToken=<your token here>

Entries in the json look as follows:

{
  "bam_input": {
        "class": "File",
        "path": "icgc:1675bfff-4494-5b15-a96e-c97169438715 "
    },
    "bamstats_report": {
        "class": "File",
        "path": "s3://oicr.temp/testing-launcher/bamstats_report.zip"
    }
}

DCC-Storage on AWS

For this, simply replace the following line of the above properties file.

client.upload.serviceHostname=objectstore.cancercollaboratory.org