RFC: Workflows

[thewilkybarkid]

Problem

Libero needs to allow for services to perform groups of related tasks based on incoming data in the form of workflows.

The key criteria for a potential workflow system is:

• avoid dependency on a specific service provider
• minimal required system configuration
• easy to deploy
• has a developer friendly API to create workflows and tasks

Suggestion

• Implement workflows using Airflow DAGs
• Use Airflow as a standard solution for executing DAGs and their tasks
• An Airflow instance must have a parent service and will be owned by this service
• An Airflow instance will only have its DAGs triggered by its parent service
• Airflow will emit notifications to the event bus

Concerns

• Airflow is an opinionated choice, but far less so than other existing options which in addition, fail to meet our requirements criteria in some way
• Can the communication method between a service and Airflow be separately specified, allowing for Airflow to be replaced with other implementations?
• Airflow is primarily geared towards scheduled workflows and currently only has an experimental rest - API for external interaction, though this can be extended via Airflow’s extensible plugin system
• An understanding of Airflow’s key concepts is required to actually create and manage workflows

[seanwiseman]

Additional context may be provided in the following resources composed during the investigatory period:

Airflow investigation notes:
https://docs.google.com/document/d/1_LUa9lVjNTleJfa9s7l3jq-craljiVG5pjoj-4sv588/edit

Higher level concept note on the required components:
https://docs.google.com/document/d/1yQbViVEwfXckS8bnp1G1vrl7q5AMb8tiSpPDlhCsGm8/edit

Initial ADR:
libero/walking-skeleton@e5072d9?short_path=dfb35fb#diff-dfb35fb00256fdfad4f24c3fd871002a