Updated API

Updated API

Update to Airflow 2.1.4

Updated dependencies

1. airflow 2.1.4
2. cwltool 3.1.20210816212154

Fixed bugs in docker-compose file

Attached python_3.8.12_cwl_airflow_master_linux.tar.gz is a relocatable python 3.8.12 extracted from the manylinux2014 image with pre-installed CWL-Airflow (ca14232bbb78df242c841047318bcb67c775379f) built in Centos:7 docker container.

Attached python_3.8_cwl_airflow_master_macos.tar.gz is a relocatable python 3.8 from Briefcase Suite with pre-installed CWL-Airflow (ca14232bbb78df242c841047318bcb67c775379f) built in macOS 11.4.

Bug fixes. Statistics report. Initial support of conditionals

• Proper handling of not delivered progress reports and workflow execution results
• Initial support of conditional workflow step execution (CWL v1.2)
• Bug fixes

All relocatable tar.gz are built from dd78057

Supports Airflow 2.0.0

• Updated to Airflow 2.0.0
• Fixed bug in triggering DAGs from API
• Added DAG to optionally resend all undelivered progress reports and results

Note
Due to changes in airflow.cfg in the latest Airflow 2.0.0, when updating CWL-Airflow from the version 1.2.8 the new airflow.cfg will be created. The old one will be backed up. Please make sure that all custom properties of the original airflow.cfg have been properly transferred.

Portable versions are build from this commit

Bug fixes, added docker-compose

• Added docker-compose with LocalExecutor
• Switched from zlib to gzip for workflow compressing/decompressing (not breaking changes)
• Updated packaging script for portable version for Ubuntu (runs LocalExecutor with PostgreSQL)

API updates

• Added API endpoint to create and trigger DAGs in the same call
• Added error categories reports
• Updated macOS packaging script to use Python 3.8 by default
• Updated clean_dag_run.py to remove outdated DAGs

removes the DAGs for the previous versions of the same workflows if they are not any running DagRuns

• Updated docker-compose (work in progress)

use latest ubuntu 20.04 and Python 3.8
add cwl_inputs_folder volume mapping to the docker-compose
start services in the order that guaranties all databases and tables are available

All tar.gz are built from the master branch (https://github.com/Barski-lab/cwl-airflow/tree/0a80928daf55ba306805ff2f3e8eaad5e6c7397c)

Bug fixes

Fixed bugs:

• broken dependencies (add --constraint parameter for installation)
• workflow fails to execute if section in of the workflow step is empty
• workflow fails to execute if workflow input has loadListing section
• API fails to trigger multiple DAG-runs when POSTing multiple requests a once
• init command fails to patch Airflow config file if the path to it includes spaces

Other changes:

• refactored progress reports posting
• added crypto installation dependency

All tar.gz are built from master (https://github.com/Barski-lab/cwl-airflow/tree/a2b9512cb3480ab005a63efc23c0421c9537915e)

Bug fixes

Updates:

• uses apache-airflow==1.10.12 & cwltool==3.0.20200710214758
• outputs CWL related logs into separate files
• supports simple scatter
• supports literal file inputs

Tested on cwl-v1.1 conformance test from Oct 7, 2020

• total: 253
• excluded: 2 (due to common-workflow-language/cwl-v1.2#51)
• failed with known reason:
  • 2 (due to common-workflow-language/cwl-v1.2#52)
  • 5 (due to #43)
  • 12 (due to #44)
• success rate: 90%

Reference: SDDA Sprint 1

Useful features update + portable versions for Ubuntu and macOS

• Add extras_require into setup.py for mysql, statsd, celery dependencies
• Support workflow parameter as zlib compressed base64 encoded string
• Update api and test to support embedded in DAG zlib compressed workflows
• Add --upgrade argument to init command to update old formatted dags
• Update on_kill function of CWLStepOperator to stop running docker container

Refactored CWL-Airflow (latest Airflow 1.10.11)

1. Faster CWL parsing and validation
2. Simplified syntax for new DAG creation

#!/usr/bin/env python3
from cwl_airflow.extensions.cwldag import CWLDAG
dag = CWLDAG(workflow="my_workflow.cwl", dag_id="my_dag")