Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Databricks and Unity catalog connections #354

Open
1 task done
mitches-got-glitches opened this issue Dec 18, 2024 · 1 comment
Open
1 task done

Support Databricks and Unity catalog connections #354

mitches-got-glitches opened this issue Dec 18, 2024 · 1 comment

Comments

@mitches-got-glitches
Copy link

mitches-got-glitches commented Dec 18, 2024

Things to check first

  • I have searched the existing issues and didn't find my feature already requested there

Feature description

I'm putting this in as a feature because I'm not sure it is supported yet. At least it broke in a way that indicated it may not have been considered yet when I tried it.

I ran:

sqlacodegen --generator sqlmodels "databricks://my_connection_string"

Where the URL/connection string was defined in the format below:

# How my connection string is constructed
# From https://docs.databricks.com/en/dev-tools/sqlalchemy.html#authentication
url = (
    f"databricks://token:{ACCESS_TOKEN}@{SERVER_HOSTNAME}?"
    + f"http_path={HTTP_PATH}&catalog={CATALOG}&schema={SCHEMA}"
)

Which resulted in the following errors:

databricks.sql.exc.ServerOperationError: [NO_SUCH_CATALOG_EXCEPTION] Catalog 'none' was not found.
Please verify the catalog name and then retry the query or command again.databricks.sql.exc.ServerOperationError:
[NO_SUCH_CATALOG_EXCEPTION] Catalog 'none' was not found. Please verify the catalog name and then retry the query or command again.

sqlalchemy.exc.DatabaseError: (databricks.sql.exc.ServerOperationError)
[NO_SUCH_CATALOG_EXCEPTION] Catalog 'none' was not found. Please verify the catalog name and then retry the query or command again.
[SQL: SHOW TABLES FROM `None`.`None`]

This suggests that the catalog and schema haven't been parsed correctly from the databricks connection string. It may be as simple as handling this specific type of URL, but there may be more to it that I'm currently ignorant of.

Use case

I want to generate SQLModels from an existing schema on Databricks where tables have been created directly from Spark dataframes on ingest process.

@agronholm
Copy link
Owner

But sqlacodegen doesn't do any parsing of the connection string. It's SQLAlchemy that does that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants