chore(deps): Migrate from `crate[sqlalchemy]` to `sqlalchemy-cratedb` #29243

amotl · 2024-06-13T18:30:34Z

SUMMARY

The CrateDB SQLAlchemy dialect needs more love, so it was separated from the DBAPI HTTP driver.
This patch concludes the update to the new designated CrateDB SQLAlchemy dialect package, sqlalchemy-cratedb, which has been validated on a few other downstream packages already.
The patch also updates the documentation section about configuring the CrateDB connector driver (preview).

Personal Message

Dear Superset maintainers, first things first: Thanks a stack for conceiving and maintaining Apache Superset, you know who you are. Hereby, we are submitting a little update about the database adapter of CrateDB. With kind regards, Andreas.

May we ask you to approve running the software tests on this draft PR, so we can get an impression if anything breaks?

/cc @hammerhead, @hlcianfagna, @surister

github-actions

Congrats on making your first PR and thank you for contributing to Superset! 🎉 ❤️

We hope to see you in our Slack community too! Not signed up? Use our Slack App to self-register.

rusackas · 2024-06-14T16:02:49Z

Thanks for the PR! We very much appreciate it :) Running CI. 🤞

john-bodley · 2024-06-14T17:23:11Z

@amotl it seems likely you've contributed to both the crate SQLAlchemy dialect and sqlalchemy-cratedb—serving as the main author for the later. What was the main reason for splitting this work out, i.e., why isn't it viable to simply update the SQLAlchemy dialect for the former?

amotl · 2024-06-14T20:46:21Z

Hi @john-bodley. Thanks for asking.

What was the main reason for splitting this work out, i.e., why isn't it viable to simply update the SQLAlchemy dialect for the former?

The Python DB API HTTP driver is stable, and does not receive many changes any longer, actually mostly none. This is by far not the case with the SQLAlchemy dialect. To get an idea about the relevant traffic, let us enumerate the whole stack of requests and improvements already queuing up.

In particular, that patch demonstrates that the HTTP driver is not necessarily tied to the SQLAlchemy dialect, and vice versa.

Dialect: Add support for asyncpg and psycopg3 drivers crate/sqlalchemy-cratedb#11

In this spirit, we decided to split concerns between those two artefacts, by separating them into two different repositories.
Currently, the SQLAlchemy dialect still exclusively uses the HTTP driver, but it will give users a choice after the aforementioned patch has been integrated.

Let us know about any further questions you may have on this topic.

The CrateDB SQLAlchemy dialect needs more love, so it was separated from the Python DB API HTTP driver.

amotl

Hi. We just refreshed this PR, bumping the dependency spec to use the most recent sqlalchemy-cratedb package, and updating the corresponding documentation section about CrateDB.

amotl · 2024-11-15T16:29:55Z

docs/docs/configuration/databases.mdx

 #### CrateDB

-The recommended connector library for CrateDB is
-[crate](https://pypi.org/project/crate/).
-You need to install the extras as well for this library.
-We recommend adding something like the following
-text to your requirements file:
-
+The connector library for CrateDB is [sqlalchemy-cratedb].
+We recommend to add the following item to your `requirements.txt` file:
 ```
-crate[sqlalchemy]==0.26.0
+sqlalchemy-cratedb>=0.40.1,<1
 ```

-The expected connection string is formatted as follows:
-
+An SQLAlchemy connection string for [CrateDB Self-Managed] on localhost,
+for evaluation purposes, looks like this:
 ```
 crate://[email protected]:4200
 ```
+An SQLAlchemy connection string for connecting to [CrateDB Cloud] looks like
+this:
+```
+crate://<username>:<password>@<clustername>.cratedb.net:4200/?ssl=true
+```
+
+Follow the steps [here](/docs/configuration/databases#installing-database-drivers)
+to install the CrateDB connector package when setting up Superset locally using
+Docker Compose.
+```
+echo "sqlalchemy-cratedb" >> ./docker/requirements-local.txt
+```
+
+[CrateDB Cloud]: https://cratedb.com/product/cloud
+[CrateDB Self-Managed]: https://cratedb.com/product/self-managed
+[sqlalchemy-cratedb]: https://pypi.org/project/sqlalchemy-cratedb/


Hi. While being at it, refreshing the canonical dependencies for the CrateDB connector, we took the chance to also update the relevant documentation section. We hope you agree with those minor updates being slapped into the same PR per aef7f2d.

@hlcianfagna, @proddata, @simonprickett, @kneth: Please also have a look at those updates, and suggest any improvements that come to mind. See GitHub-rendered preview.

villebro

LGTM, thanks for the detailed context on the evolution of the driver/dialect.

amotl · 2024-11-15T21:27:19Z

Hi. Thanks for merging. We also just updated the canonical migration page on our documentation. This page educates about the package switchover, not specifically related to Apache Superset.

https://cratedb.com/docs/sqlalchemy-cratedb/migrate-from-crate-client.html

villebro · 2024-11-15T21:30:06Z

Thanks for all the work @amotl , this is definitely open source at it's finest ❤️

amotl · 2024-11-15T21:44:38Z

Thanks for all your excellent work on Superset! 💯

* fix: Exception handling for SQL Lab views (apache#30897) * fix: don't show metadata for embedded dashboards (apache#30875) * feat: add logging durations for screenshot async service (apache#30884) * refactor(Avatar): Migrate Avatar to Ant Design 5 (apache#30740) * refactor(input): Migrate Input component to Ant Design 5 (apache#30730) * fix(empty dashboards): Allow downloading a screenshot of an empty dashboard (apache#30767) Co-authored-by: Geido <[email protected]> * chore(ci): add tai and michael to helm owners (apache#30925) * fix(helm): use submodule on helm release action (apache#30924) * docs: add Free2Move to INTHEWILD.md (apache#30930) * fix: blocks UI elements on right side (apache#30886) Co-authored-by: Evan Rusackas <[email protected]> * chore(deps): Migrate from `crate[sqlalchemy]` to `sqlalchemy-cratedb` (apache#29243) * chore: update change log, UPDATING.md and bug-report.yml for 4.1 release (apache#30915) * chore(docs): Update list of supported databases to include CrateDB (apache#30946) * feat(trino,presto): add missing time grains (apache#30926) * fix(Dashboard): Exclude edit param in async screenshot (apache#30962) * fix(Card): Use correct class names for Ant Design 5 Card component (apache#30964) * chore(Accessibility): Fix accessibility for 'Show x entries' dropdown in tables (apache#30818) * build(deps): bump cross-spawn from 7.0.3 to 7.0.6 in /superset-frontend/cypress-base (apache#30969) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(release validation): scripts now support RSA and EDDSA keys. (apache#30967) * build(deps): bump cross-spawn from 7.0.3 to 7.0.6 in /docs (apache#30970) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: add performance information to tooltip (apache#30948) * fix(helm-chart): Fix broken PodDisruptionBudget due to introduction of extraLabels. (apache#30966) * chore(helm): bumping app version to 4.1.1 in helm chart (apache#30918) * chore: add unit tests for `is_mutating()` (apache#31021) * chore(🦾): bump python numexpr 2.10.0 -> 2.10.1 (apache#31006) Co-authored-by: GitHub Action <[email protected]> * chore(🦾): bump python cffi 1.16.0 -> 1.17.1 (apache#31002) Co-authored-by: GitHub Action <[email protected]> * chore(🦾): bump python flask-babel subpackage(s) (apache#31000) Co-authored-by: GitHub Action <[email protected]> * fix(Dashboard): Retain colors when color scheme not set (apache#30646) * fix(explore): verified props is not updated (apache#31008) * chore: publish wheels (apache#30981) * docs: Embedded sdk (apache#30972) * fix(imports): import query_context for imports with charts (apache#30887) * docs: Update doc about CSV upload (apache#30867) Co-authored-by: Evan Rusackas <[email protected]> * chore(🦾): bump python cattrs 23.2.3 -> 24.1.2 (apache#30998) Co-authored-by: GitHub Action <[email protected]> * fix(dataset): use sqlglot for DML check (apache#31024) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Michael S. Molina <[email protected]> Co-authored-by: Joe Li <[email protected]> Co-authored-by: Maxime Beauchemin <[email protected]> Co-authored-by: Mehmet Salih Yavuz <[email protected]> Co-authored-by: Geido <[email protected]> Co-authored-by: Ville Brofeldt <[email protected]> Co-authored-by: Paolo Terzi <[email protected]> Co-authored-by: Sukuna <[email protected]> Co-authored-by: Evan Rusackas <[email protected]> Co-authored-by: Andreas Motl <[email protected]> Co-authored-by: Andreas Motl <[email protected]> Co-authored-by: Levis Mbote <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Elizabeth Thompson <[email protected]> Co-authored-by: Christoph Keller <[email protected]> Co-authored-by: lodu <[email protected]> Co-authored-by: Beto Dealmeida <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: GitHub Action <[email protected]> Co-authored-by: JUST.in DO IT <[email protected]> Co-authored-by: David Hotham <[email protected]> Co-authored-by: Giampaolo Capelli <[email protected]> Co-authored-by: Linden <[email protected]> Co-authored-by: Seiya <[email protected]> Co-authored-by: Asher Manangan <[email protected]>

pull-request-size bot added the size/XS label Jun 13, 2024

github-actions bot added the doc Namespace | Anything related to documentation label Jun 13, 2024

github-actions bot reviewed Jun 13, 2024

View reviewed changes

amotl marked this pull request as ready for review June 14, 2024 07:50

dosubot bot added the data:connect Namespace | Anything related to db connections / integrations label Jun 14, 2024

john-bodley requested a review from betodealmeida June 14, 2024 17:19

amotl mentioned this pull request Jun 26, 2024

Return TIMESTAMP columns as native Python datetime objects to improve support for Apache Superset crate/crate-python#395

Closed

amotl mentioned this pull request Jul 22, 2024

Dependencies: Migrate from crate[sqlalchemy] to sqlalchemy-cratedb mindsdb/mindsdb#9345

Merged

chore(deps): Migrate from crate[sqlalchemy] to sqlalchemy-cratedb

5119132

The CrateDB SQLAlchemy dialect needs more love, so it was separated from the Python DB API HTTP driver.

amotl force-pushed the sqlalchemy-cratedb branch from 0935189 to afe9206 Compare November 15, 2024 16:24

pull-request-size bot added size/M and removed size/XS labels Nov 15, 2024

chore(docs): Update documentation section about CrateDB configuration

aef7f2d

amotl force-pushed the sqlalchemy-cratedb branch from afe9206 to aef7f2d Compare November 15, 2024 16:25

amotl commented Nov 15, 2024

View reviewed changes

villebro approved these changes Nov 15, 2024

View reviewed changes

villebro merged commit 1991d1f into apache:master Nov 15, 2024
37 checks passed

amotl mentioned this pull request Nov 15, 2024

chore(docs): Update list of supported databases to include CrateDB #30946

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(deps): Migrate from `crate[sqlalchemy]` to `sqlalchemy-cratedb` #29243

chore(deps): Migrate from `crate[sqlalchemy]` to `sqlalchemy-cratedb` #29243

amotl commented Jun 13, 2024 •

edited

Loading

github-actions bot left a comment

rusackas commented Jun 14, 2024

john-bodley commented Jun 14, 2024

amotl commented Jun 14, 2024 •

edited

Loading

amotl left a comment

amotl Nov 15, 2024 •

edited

Loading

villebro left a comment

amotl commented Nov 15, 2024 •

edited

Loading

villebro commented Nov 15, 2024

amotl commented Nov 15, 2024

chore(deps): Migrate from crate[sqlalchemy] to sqlalchemy-cratedb #29243

chore(deps): Migrate from crate[sqlalchemy] to sqlalchemy-cratedb #29243

Conversation

amotl commented Jun 13, 2024 • edited Loading

SUMMARY

Personal Message

github-actions bot left a comment

Choose a reason for hiding this comment

rusackas commented Jun 14, 2024

john-bodley commented Jun 14, 2024

amotl commented Jun 14, 2024 • edited Loading

amotl left a comment

Choose a reason for hiding this comment

amotl Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

amotl commented Nov 15, 2024 • edited Loading

villebro commented Nov 15, 2024

amotl commented Nov 15, 2024

chore(deps): Migrate from `crate[sqlalchemy]` to `sqlalchemy-cratedb` #29243

chore(deps): Migrate from `crate[sqlalchemy]` to `sqlalchemy-cratedb` #29243

amotl commented Jun 13, 2024 •

edited

Loading

amotl commented Jun 14, 2024 •

edited

Loading

amotl Nov 15, 2024 •

edited

Loading

amotl commented Nov 15, 2024 •

edited

Loading