Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add DatabricksOauthProfileMapping profile #1091

Merged

Conversation

CorsettiS
Copy link
Contributor

@CorsettiS CorsettiS commented Jul 13, 2024

Description

As of now, Token-based auth is the only possible way to access databricks from cosmos, even though dbt-databricks support OAuth-based access as can be seen here .

This PR aims to enhance the support offered by cosmos to databricks. The final goal is to close #1085

I have tested locally and got a positive result.

Breaking Change?

No

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jul 13, 2024
@CorsettiS CorsettiS closed this Jul 13, 2024
Copy link

netlify bot commented Jul 13, 2024

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 02cad9f
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6692e0e014b64a000894d8cd

Copy link

netlify bot commented Jul 13, 2024

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 998908b
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6694d9a6f96f9c0008f69055

@dosubot dosubot bot added area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc profile:databricks Related to Databricks ProfileConfig labels Jul 13, 2024
@CorsettiS CorsettiS reopened this Jul 13, 2024
@CorsettiS CorsettiS changed the title Rename to databricks oauth profile mapping add DatabricksOauthProfileMapping profile Jul 13, 2024
Copy link

codecov bot commented Jul 13, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.25%. Comparing base (1f27638) to head (998908b).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1091      +/-   ##
==========================================
+ Coverage   96.23%   96.25%   +0.01%     
==========================================
  Files          63       64       +1     
  Lines        3270     3284      +14     
==========================================
+ Hits         3147     3161      +14     
  Misses        123      123              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution! It looks great. I just posted a few minor comments

cosmos/profiles/databricks/client.py Outdated Show resolved Hide resolved
cosmos/profiles/databricks/client.py Outdated Show resolved Hide resolved
tests/profiles/databricks/test_dbr_client.py Outdated Show resolved Hide resolved
tests/profiles/databricks/test_dbr_client.py Outdated Show resolved Hide resolved
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Jul 15, 2024
@pankajastro pankajastro added this to the Cosmos 1.6.0 milestone Jul 17, 2024
@pankajastro pankajastro merged commit 954fa34 into astronomer:main Jul 18, 2024
64 checks passed
@pankajkoti pankajkoti mentioned this pull request Jul 31, 2024
pankajkoti added a commit that referenced this pull request Aug 20, 2024
New Features

* Add support for loading manifest from cloud stores using Airflow
Object Storage by @pankajkoti in #1109
* Cache ``package-lock.yml`` file by @pankajastro in #1086
* Support persisting the ``LoadMode.VIRTUALENV`` directory by @tatiana
in #1079
* Add support to store and fetch ``dbt ls`` cache in remote stores by
@pankajkoti in #1147
* Add default source nodes rendering by @arojasb3 in #1107
* Add Teradata ``ProfileMapping`` by @sc250072 in #1077

Enhancements

* Add ``DatabricksOauthProfileMapping`` profile by @CorsettiS in #1091
* Use ``dbt ls`` as the default parser when ``profile_config`` is
provided by @pankajastro in #1101
* Add task owner to dbt operators by @wornjs in #1082
* Extend Cosmos custom selector to support + when using paths and tags
by @mvictoria in #1150
* Simplify logging by @dwreeves in #1108

Bug fixes

* Fix Teradata ``ProfileMapping`` target invalid issue by @sc250072 in
#1088
* Fix empty tag in case of custom parser by @pankajastro in #1100
* Fix ``dbt deps`` of ``LoadMode.DBT_LS`` should use
``ProjectConfig.dbt_vars`` by @tatiana in #1114
* Fix import handling by lazy loading hooks introduced in PR #1109 by
@dwreeves in #1132
* Fix Airflow 2.10 regression and add Airflow 2.10 in test matrix by
@pankajastro in #1162

Docs

* Fix typo in azure-container-instance docs by @pankajastro in #1106
* Use Airflow trademark as it has been registered by @pankajastro in
#1105

Others

* Run some example DAGs in Kubernetes execution mode in CI by
@pankajastro in #1127
* Install requirements.txt by default during dev env spin up by
@@CorsettiS in #1099
* Remove ``DbtGraph.current_version`` dead code by @tatiana in #1111
* Disable test for Airflow-2.5 and Python-3.11 combination in CI by
@pankajastro in #1124
* Pre-commit hook updates in #1074, #1113, #1125, #1144, #1154,  #1167

---------

Co-authored-by: Pankaj Koti <[email protected]>
Co-authored-by: Pankaj Singh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc lgtm This PR has been approved by a maintainer profile:databricks Related to Databricks ProfileConfig size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Databricks - Allow each dbt node to run in its own configurable cluster
3 participants