Elegant Echo

The fifth official release focuses on reduction of complexity and increase of maintainability, further separating stack resources and setup scripts, plus a few security improvements.

Features:

• Use the AWS Secrets Manager to create and store the RDS password
• Use a KDF to derive the Fernet key instead of generating it randomly
• Control RDS access using source security groups instead of CIDR blocks
• Publish no metrics if the there are no workers and the queue is empty

Enhancements:

• Use proper shell scripts to set up instances checked with shellcheck
• Convert load metric to python and check with pylint, flake8, black

Bugfixes:

• Fix pycurl dependency mismatch after its version was pinned by airflow
• Fix connection failure if database password contained special characters
• Fix race condition between instances and codedeploy causing setup to fail

Misc:

• Upgraded to apache-airflow==1.10.9
• Remove non-GPL airflow install flag used in previous airflow versions
• Add tests for guidelines like consistent parameter documentation
• Formally document the load estimation technique

Decisive Dice

The fourth official release focuses on a more mature cloud architecture following AWS recommendations for CloudFormation templates.

New features:

• Multi-AZ ASG for the Airflow EC2 instances, towards fault tolerance
• Split template into nested stacks allowing deployment on existing VPCs
• Store the Fernet key in the AWS Parameter Store instead of EBS

Stack improvements

• Removed hardcoded bucket names and trust CFN to create them
• Removed tenacity version pin for py3.7 because it was fixed on airflow
• Removed celery version pin for thread safety because it was fixed on airflow
• Changed default mininum worker count to 1, a more sensible default for demos
• Multiple improvements targeting compliance with AWS QuickStart guidelines

Fixes:

• Added cfn-signal missing relevant parameters
• Removed hardcoded AZ options that failed in specific regions
• Added EFS mount to fstab guaranteeing it's there after a reboot

Misc:

• Upgraded to apache-airflow==1.10.6
• Added cfn-lint as GitHub Action
• Automated testing with taskcat
• Merged on upstream Airflow a higher limit for queue names

Creative Cell

The third official release focuses on more robust deployments and safer auto scaling with graceful shutdown.

New features:

• Deployment with CodeDeploy, giving more control and speed
• Airflow logs now are stored in a dedicated S3 bucket
• Parameter allowing to opt out of loading example DAGs
• Parameter allowing to opt out of loading default connections

Stack improvements

• EC2 instances now have an attached IAM role, making it easier to attach policies
• Celery now uses that role with narrower queue permissions
• EC2 instances now have an env var specifying its kind (worker, scheduler, webserver)
• Celery broker transport options are now enforced, no need to specify on airflow.cfg

Fixes:

• No more race conditions running initdb
• No more terminating instances killing tasks

Misc:

• Upgraded to apache-airflow==1.10.2

Blazing Beat

The second official release focuses on more flexible parameters and a better auto scaling attempt.

• New parameters: scheduler and webserver instance types
• New parameters: webserver port (enforced in the environment)
• New parameters: CIDR block for opening SSH access
• Worker setup: tries to install from AIRFLOW_HOME/requirements.txt
• Worker load: metric now discounts an estimation of time spent on boot
• Stack updates: machines watch for metadata updates and restart airflow

Amazonic Art

The first official release! Featuring many improvements over the unofficial previous release:

• Airflow 1.10.1 with official Sagemaker support!
• Managed systemd: switched from the old supervisord 2
• Parametrized region: not restricted to us-east-1 anymore
• Upgraded AMI: now using Amazon Linux 2 with Python 3.7
• Guessed AMI IDs: picks the latest GP2 AMI for that region
• EFS on fstab: airflow home directory now survive reboots
• Prefixed tags: using the stack name on tags