Add Azure Container Instance as Execution Mode

[danielvdende]

Description

This PR adds support for Azure Container Instance as an Execution Mode. In this mode, a container is run on Azure to execute the dbt commands.

Related Issue(s)

This closes #635

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	250d5d3
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/6585b0ff17ca1f00088ee885

[danielvdende]

Note: I haven't added the step-by-step documentation yet. This is deliberate, as it'd be great to first get feedback on the implementation so I can make sure to align implementation with docs.

[codecov]

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (7a9b697) 94.81% compared to head (0a8913d) 94.40%.

Files	Patch %	Lines
cosmos/__init__.py	22.22%	7 Missing ⚠️
cosmos/operators/azure_container_instance.py	84.21%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #771      +/-   ##
==========================================
- Coverage   94.81%   94.40%   -0.42%     
==========================================
  Files          55       56       +1     
  Lines        2470     2520      +50     
==========================================
+ Hits         2342     2379      +37     
- Misses        128      141      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jbandoro]

Thanks for your contribution, this looks awesome! Minor comments below from me, I'm not familiar with the ACI, so it might be good to have another set of 👀 here for review.

You'll have to add the new operator imports to the init.py file below following like what's done for Docker and Kubernetes:

astronomer-cosmos/cosmos/__init__.py

Lines 34 to 52 in b58b018

	try:
	from cosmos.operators.docker import (
	DbtLSDockerOperator,
	DbtRunDockerOperator,
	DbtRunOperationDockerOperator,
	DbtSeedDockerOperator,
	DbtSnapshotDockerOperator,
	DbtTestDockerOperator,
	)
	except ImportError:
	DbtLSDockerOperator = MissingPackage("cosmos.operators.docker.DbtLSDockerOperator", "docker")
	DbtRunDockerOperator = MissingPackage("cosmos.operators.docker.DbtRunDockerOperator", "docker")
	DbtRunOperationDockerOperator = MissingPackage(
	"cosmos.operators.docker.DbtRunOperationDockerOperator",
	"docker",
	)
	DbtSeedDockerOperator = MissingPackage("cosmos.operators.docker.DbtSeedDockerOperator", "docker")
	DbtSnapshotDockerOperator = MissingPackage("cosmos.operators.docker.DbtSnapshotDockerOperator", "docker")
	DbtTestDockerOperator = MissingPackage("cosmos.operators.docker.DbtTestDockerOperator", "docker")

[jbandoro]

Thanks for addressing all the comments! This looks great to me.

I'd like to get another person from the team to review for adding in a new execution mode. Feel free to add in the step-by-step instructions in docs/getting_started/azure-container-instance.rst in the meantime since the implementation looks 👍.

[danielvdende]

@jbandoro I've now added a step-by-step-guide. It includes a reference to an example I've put together in the cosmos-example repo. I haven't opened the PR for it yet, because I first fixed the existing examples in there for which a PR is now open here. Once that one is merged, I can open the one with the example DAG for Azure Container Instance.

[tatiana]

Hi @danielvdende - this feature is very handy for users running in Azure. Thanks for working on this!

We recently merged @jbandoro 's #774 with many improvements to the existing operators/execution modes. Please, could you rebase and follow the new pattern? It will improve our code maintainability :)

[danielvdende]

@jbandoro @tatiana I see you have been merging main with this branch a few times (perfectly fine of course). Anything you would like me to do at this point?

[jbandoro]

@danielvdende we're trying to get an alpha release of v1.4 out soon, and I was hoping to get this work merged, but the unit tests are failing after merging in the latest commits from main. I checked out this branch and can't reproduce the failing unit tests locally. When I find a solution I'll open up a PR in your fork so that we can we can merge it after all the tests pass, apologies that this is taking some time.

edit: @danielvdende I was able to get the tests working in this PR in your fork: https://github.com/danielvdende/astronomer-cosmos/pull/1/files

[danielvdende]

Great @jbandoro! I just merged the PR, and see pipelines running here again now too.