Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(backends): support creation from a DB-API con #9603

Merged
merged 23 commits into from
Jul 22, 2024
Merged

feat(backends): support creation from a DB-API con #9603

merged 23 commits into from
Jul 22, 2024

Conversation

deepyaman
Copy link
Contributor

Description of changes

Enable backend creation from an existing DB-API connection

Issues closed

Resolves #8877

@github-actions GitHub Actions
Copy link
Contributor

ACTION NEEDED

Ibis follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message.

Please update your PR title and description to match the specification.

@deepyaman deepyaman force-pushed the feat/from-dbapi-con branch from 6fa7d34 to b359aaf Compare July 16, 2024 15:03
cpcloud
cpcloud reviewed Jul 16, 2024
View reviewed changes
ibis/backends/tests/test_client.py Outdated
@@ -1764,3 +1764,8 @@ def test_insert_using_col_name_not_position(con, first_row, second_row, monkeypa
# Ideally we'd use a temp table for this test, but several backends don't
# support them and it's nice to know that data are being inserted correctly.
con.drop_table(table_name)


def test_from_dbapi_connection(con):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we can run the entire test suite using a backend constructed this way, which will give us more confidence that the API works.

That can be done in a follow up though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you envision doing this without running the whole test suite again? Or it's fine to run it again?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can (somewhat arbitrarily) pick a backend and modify its conftest.py to use this way of initialization.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But then we wouldn't have the other way of initialization, right? Or, because there's ddl_con and con fixtures both, it would be OK?

Will look to explore this in a followup then; will just get the rest of the backends added and tests passing here first.

jcrist
jcrist reviewed Jul 16, 2024
View reviewed changes
ibis/backends/__init__.py Outdated Show resolved Hide resolved
jcrist
jcrist reviewed Jul 16, 2024
View reviewed changes
Copy link
Member

@jcrist jcrist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be nice to make this available as ibis.<backend>.from_connection as well. The code that does this is gross, but lives here. Could add it with a hasattr check here, or with adding from_connection to _top_level_methods on base SQL backend (and the snowflake backend, since it appends). No strong thoughts.

deepyaman
deepyaman commented Jul 16, 2024
View reviewed changes
ibis/backends/clickhouse/__init__.py Outdated
@@ -163,6 +163,20 @@ def do_connect(
**kwargs,
)

@classmethod
def from_dbapi_connection(cls, con: cc.driver.Client) -> Backend:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest request from this functionality comes from @sh-rp and the dltHub team: #8877 (comment)

However, for a case like ClickHouse, I'm not 100% sure we can bridge the gap, because dltHub uses clickhouse-driver (community package, longer history, potentially better at scale?) and Ibis uses clickhouse-connect (official ClickHouse package, better HTTP support). From ClickHouse/clickhouse-connect#91, it looks like there was a lot of consideration around unification, but, in the end, doesn't seem likely.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scale seems like a non-issue (e.g., ClickHouse/clickhouse-connect#91 (comment)).

We switched to clickhouse-connect due to a bunch of data type issues and choices that didn't align well with Ibis (ClickHouse enums were completely broken due to the refusal to play nice with Python's built in Enum types), and at the time the maintainer was unresponsive and hadn't made a release in many months.

We're definitely not prepared to support multiple ways to connect to ClickHouse right now and it's unlikely we will do so in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense! I just wanted to note this, and appreciate the explanation of why Ibis moved to clickhouse-connect.

@deepyaman deepyaman changed the title [WIP] feat(backends): support creation from a DB-API con feat(backends): support creation from a DB-API con Jul 16, 2024
@deepyaman deepyaman force-pushed the feat/from-dbapi-con branch 3 times, most recently from 1bd470a to 9f380c9 Compare July 17, 2024 06:20
@sh-rp sh-rp mentioned this pull request Jul 17, 2024
Closed
@deepyaman deepyaman force-pushed the feat/from-dbapi-con branch 3 times, most recently from 18fdb69 to 79db1fe Compare July 18, 2024 21:16
@deepyaman deepyaman mentioned this pull request Jul 18, 2024
Merged
@deepyaman deepyaman force-pushed the feat/from-dbapi-con branch 10 times, most recently from e892fb8 to fa7de69 Compare July 20, 2024 04:55
@deepyaman deepyaman marked this pull request as ready for review July 20, 2024 05:48
@deepyaman deepyaman force-pushed the feat/from-dbapi-con branch from fa7de69 to dfabfb3 Compare July 20, 2024 05:48
@deepyaman
Copy link
Contributor Author

deepyaman commented Jul 20, 2024 •
edited
Loading

Oh, I forgot about BigQuery and Snowflake...

Done.

@deepyaman deepyaman force-pushed the feat/from-dbapi-con branch from dfabfb3 to a21de5b Compare July 20, 2024 15:40
@deepyaman deepyaman force-pushed the feat/from-dbapi-con branch from 0633643 to c8267e6 Compare July 22, 2024 17:59
cpcloud
cpcloud approved these changes Jul 22, 2024
View reviewed changes
Copy link
Member

@cpcloud cpcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work!

@cpcloud cpcloud added the backends Issues related to all backends label Jul 22, 2024
@cpcloud
Copy link
Member

cpcloud commented Jul 22, 2024

Gonna run BQ and Snowflake locally real quick...

@cpcloud
Copy link
Member

cpcloud commented Jul 22, 2024 •
edited
Loading

Found a small bug in BQ, fixing it now.

@jcrist jcrist mentioned this pull request Jul 22, 2024
Closed
1 task
@cpcloud cpcloud force-pushed the feat/from-dbapi-con branch from 4bd25c5 to ce48b22 Compare July 22, 2024 20:01
@cpcloud cpcloud force-pushed the feat/from-dbapi-con branch from ce48b22 to 8556c0c Compare July 22, 2024 20:14
@cpcloud cpcloud added the ci-run-cloud Add this label to trigger a run of BigQuery, Snowflake, and Databricks backends in CI label Jul 22, 2024
@cpcloud
Copy link
Member

cpcloud commented Jul 22, 2024

Tested the clouds locally, but running here for completeness. Will merge on green and then cut 9.2!

@ibis-docs-bot ibis-docs-bot bot removed the ci-run-cloud Add this label to trigger a run of BigQuery, Snowflake, and Databricks backends in CI label Jul 22, 2024
@cpcloud cpcloud force-pushed the feat/from-dbapi-con branch from 8556c0c to 2184b77 Compare July 22, 2024 21:04
@cpcloud cpcloud added the ci-run-cloud Add this label to trigger a run of BigQuery, Snowflake, and Databricks backends in CI label Jul 22, 2024
@ibis-docs-bot ibis-docs-bot bot removed the ci-run-cloud Add this label to trigger a run of BigQuery, Snowflake, and Databricks backends in CI label Jul 22, 2024
@cpcloud cpcloud enabled auto-merge (squash) July 22, 2024 21:37
@cpcloud cpcloud merged commit fc4d1e3 into ibis-project:main Jul 22, 2024
88 checks passed
@deepyaman deepyaman deleted the feat/from-dbapi-con branch July 22, 2024 22:33
@jcrist jcrist mentioned this pull request Jul 23, 2024
Closed
1 task
This was referenced Jul 23, 2024
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jcrist jcrist jcrist left review comments

@cpcloud cpcloud cpcloud approved these changes

Assignees
No one assigned
Labels
backends Issues related to all backends
Projects
None yet
Milestone
9.2
Development

Successfully merging this pull request may close these issues.

feat: Reuse the existing DB connection while creating backend
3 participants
@deepyaman @cpcloud @jcrist