airflow-dbt-python v1.0.0

[tomasfarias]

Over the last few months I've been working on a heavy refactoring of the internals of airflow-dbt-python. Life has consistently gotten in the way, but the finish line is now clear. The main goal of the refactor was formalize the responsibilities of operators, hooks, and utilities to allow me to commit to a v1.0.0 release:

airflow-dbt-python operators:

• Define what dbt command/s to execute.
• Expose configuration parameters.
• Push/return execution results.

airflow-dbt-python hooks:

• Define how to execute a dbt command.
• Setup dbt for execution by interacting with remotes.
• Enable Airflow features, like connections, to be used with dbt.

airflow-dbt-python utilities:

• Misc. utilities
• Configurations to map arguments exposed by operators to configuration values used by dbt.

Moreover, besides the items in the Roadmap (see the bottom), the release of v1.0.0 signifies I have no more ideas for features to implement in airflow-dbt-python. I'll be happy to resume maintenance of the issue board which has gone a bit stale at this point, and I'll rely on the community to propose new ideas for features to develop.

Due to the sheer size of this refactor, once merged I'll make a beta release for version 1.0.0. I encourage folks to try out the beta version and report any issues if they can.

Changes:

• airflow-dbt-python operators no longer handle temporary directories.
  • airflow-dbt-python operators should be about defining what to execute, not how.
  • airflow-dbt-python hooks are the ones that should worry about setting up directories.
  • This significantly reduces the complexity in operators.
• Refactored the dbt remote interface (previously dbt backends).
  • The interface has been simplified to two methods: upload and download.
  • Now utilizes a more specific URL class for all URL-like arguments.
  • All dbt remotes are now hooks too, which means they can use Airflow Connections.
• Implemented a new DbtRemoteGitHook to utilize git repositories as remotes.
• tar file is a new supported archive format for dbt projects.
• Moved dbt configurations to the utilities module.
  • dbt configurations are a wrapper to dbt. They exist purely because dbt doesn't have a formalized API outside the CLI.
  • Once dbt as a library moves along (see [CT-1581] [Epic] dbt-core as a library: first steps dbt-labs/dbt-core#6356), we may drop all of these configurations.
• Much better understanding of how dbt logs things (as in, we now don't log things multiple times).
• Support for Python 3.11.
• Updated documentation.

Breaking changes:

• Dropped support for Airflow major version 1.
  • It required too much work to maintain, and AWS MWAA has offered Airflow >= 2.x for a while now.

Roadmap (things that I didn't get to do but may do in the future):

• Deferrable operators: dbt workflows are, by definition, heavily I/O bound. Airflow released deferrable operators to better utilize resources in tasks that do a lot of waiting, like heavy I/O bound tasks. These operators utilize Python's asyncio which unfortunately dbt does not currently use (dbt is thread-based). It may be possible to use deferrable operators with an asyncio dbt adapter, but I have not had the time to turn these ideas into anything concrete.
• Airflow provider packages implement a set of interfaces that cover mostly hook attributes and methods. Although we have covered some, we don't cover all of the requirements yet.