IDE for WDL

This project aims to provide a "batteries-included" environment for developing WDL workflows.

Please note: it is currently under active development, and is not yet feature-complete.

Editor-agnostic WDL support

We will provide a WDL Language Server plugin, based on Language Server Protocol (LSP), cromwell-tools, pygls, and miniwdl.

This protocol is supported by many code editors, and enables universal support for language features.

More specifically, our plugin will enable you to:

• check the syntax of a workflow
• submit a workflow for execution to Cromwell API
• watch for completion/failure of each workflow (or cancel it)
• highlight task-specific failures
• get feedback on runtime resource management
• enjoy rich editor support (jump to definition etc.)

You can find the following published extensions based on this plugin:

• WDL DevTools for VS Code
• WDL DevTools for IntelliJ

Other editors are supported via their respective LSP plugins. Examples on how to connect them are in the client directory.

Browser IDE

Additionally, we provide an Integrated Development Environment (IDE), which runs in a web browser and is based on Theia.

It bundles WDL extensions for Visual Studio Code - WDL DevTools and WDL Syntax Highlighter - along with a "local" instance of Cromwell.

The bundle consists of Docker containers, which you can set up with a single Docker Compose command.

This approach is used to

• develop workflows locally, with an ultra-fast feedback loop
• run workflows in the cloud from developer machine - no need for a Cromwell server
• create reproducible setup - it works on any OS with Docker Compose
• run the same setup on a remote server - think Notebooks, but for WDL!
• simplify local development - it just works™

Deployment

To deploy the IDE:

• clone or download this repo into a local folder
• install Docker Compose
• for local-only development, run this command in the cloned folder:

  docker-compose pull && docker-compose up
  

• for local and Google Cloud development, do the following instead [*]:
  • create a project on Google Cloud Platform
  • create a service account with project-wide Genomics Pipelines Runner role, and download its key in JSON format
  • grant it Service Account User permission on Compute Engine default service account in that project
  • create a bucket for Cromwell executions
  • grant Storage Object Admin permission on the Cromwell executions bucket
  • run this command, replacing <...> with your values:

    docker-compose pull && \
    GOOGLE_APPLICATION_CREDENTIALS=./<your-service-account-key>.json \
    GOOGLE_AUTH_MODE=service-account \
    GOOGLE_CLOUD_PROJECT=<your-project-name> \
    GOOGLE_EXECUTIONS_BUCKET=<executions-bucket-name> \
    docker-compose up
    

  [*] In the future, we may provide a simplified script to do most of the above. Additionally, we may add settings for other cloud providers (PRs are welcome!).

The first time you run the Docker compose command, it will take ~5 minutes to compile the IDE from sources and bring up the environment. Later on, we will provide a Docker image to speed that up.

When you no longer see the log messages, the IDE is running and you can navigate to it in a browser at this address: localhost:3000.