Add on_kill to Databricks Workflow Operator

[R7L208]

The Databricks Provider did not implement on_kill to cancel tasks generated by _CreateDatabricksWorkflowOperator. This led to data quality issues, where Airflow would report a cancellation due to timeout; however, the corresponding workflow task would continue to run on Databricks.

This PR implements on_kill for _CreateDatabricksWorkflowOperator.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: [email protected]
  Slack: https://s.apache.org/airflow-slack

[R7L208]

hey @pankajkoti - Any ETA on when you'd be able to review this PR? 🙏

[pankajkoti]

hi @R7L208 Apologies, I missed reviewing it so far.

I am happy to review the changes for databricks_workflow.py but not so comfortable yet reviewing the changes in databricks_sql.py (operators & hooks) as I do not have enough expertise on the changes made there. Would you please to separate out the two changes(move out databricks_sql.py related changes to another PR) and then we can invite some expert to review the other PR?

[R7L208]

Opened #42668

@pankajkoti @eladkal - can you please assign someone to review?

[pankajkoti]

@R7L208 Can you please update the PR description based on the altered scope of this PR?

[R7L208]

@pankajkoti - it's been updated

[pankajkoti]

@pankajkoti - it's been updated

@R7L208 it's not. It mentions about DatabricksSqlHook queries ("SQL queries submitted by DatabricksSqlHook") & using threading ("uses threading to cancel SQL queries submitted by DatabricksSqlHook.run()") which we no longer have in this PR. Could you please re-read the description and update to what's limited to the scope of this PR.

Also, _CreateDatabricksWorkflowOperator does not rely on DatabricksSqlHook, but leverages DatabricksHook just in case you missed checking that :)

[R7L208]

@pankajkoti - Apologies! I read PR title instead of PR description 🤦

PR description is now updated

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.