feat(risingwave): init impl for Risingwave

[KeXiangWang]

Description of changes

This is the initial PR for ibis to support risingwave.

RisingWave is a distributed SQL streaming database engineered to provide the simplest and most cost-efficient approach for processing and managing streaming data with utmost reliability.

After a few weeks of investigation, here's my phasic results.

1. As Risingwave is largely compatible with Postgres, Ibis can be easily extended to support Risingwave. The main work is to add a new dialect for Risingwave, which is similar to the postgres dialect. I've almost finished this part in this PR. With this PR, Ibis can be used to connect to Risingwave and run some basic queries. I've manually tested some queries which works well and add some tests imitating Postgres Backend. I would appreciate if you can help test more queries.

2. As ibis relies on SQLalchemy to support Postgres, we follow its implementation to support Risingwave. However, there are also some differences in semantics between Risingwave and Postgres, which require some modification in either ibis or SQLalchemy. sqlalchemy-risingwave is designed to reduce this mismatch. So in this PR, I introduce the new dependencies sqlalchemy-risingwave to ibis.

3. Ibis has no support for Materialized View natively. However Materialized View is a core concept in RW, people use RW because of its convenient auto-updated Materialized View. Now, if a user wants to create a new MV, he needs to use a raw SQL. Adding DDLs like CreateMaterializedView, CreateSource and CreateSink in the base ddl file may help. We would appreciate it if you can help offer some suggestions.

Besides this, I also met some obstacle that may need your help.

1. Risingwave hasn't supported TEMPORARY VIEW yet, so I changed some implementations relying on TEMPORARY VIEW to a normal view. For example, for the _metadata() functions, RW backend's implementation is con.exec_driver_sql(f"CREATE VIEW IF NOT EXISTS {name} AS {query}"). While in PG it's con.exec_driver_sql(f"CREATE TEMPORARY VIEW {name} AS {query}"). Do you have any suggestions for other ways to work around this?
   Besides, I didn't quite get what _metadata() is doing here. I would appreciate it if you could explain it a bit.

2. There's some mismatch between the postgres dialect and risingwave dialect, which are still not fully tested in this PR. We'll continue to work on it.

3. This PR requires some new features of Risingwave v1.6.0 and sqlalchemy-risingwave v1.0.0 which are not released yet. They'll be released soon. Done. BTW, how should I indicate this backend is only for risingwave > 1.6?

4. I don't quite understand the test pipeline of Ibis. I copied the test cases from the postgres dialect and modified them to fit the risingwave dialect, and some of them are commented temporarily due to the lack of support. I also added an SQL script to help set up the test environment, which creates tables and loads data. But I don't know how to run it in the test pipeline. Any suggestions or guidance are welcomed. I suppose the test pipeline would require a docker image of Risingwave. We can provide one if needed.

5. I'm a newbie in the ibis community, this PR may not be perfect considering others. Any comments are welcomed and I sincerely appreciate your time and patience.

closes #8038

[deepyaman]

Just a couple views on your questions, didn't answer everything:

3. This PR requires some new features of Risingwave v1.6.0 and sqlalchemy-risingwave v1.0.0 which are not released yet. They'll be released soon. BTW, how should I indicate this backend is only for risingwave > 1.6?

I assume you could just include it in the backend docs, perhaps similar to how you have some notes for Oracle here: https://ibis-project.org/backends/oracle#connecting-to-older-oracle-databases

4. I don't quite understand the test pipeline of Ibis. I copied the test cases from the postgres dialect and modified them to fit the risingwave dialect, and some of them are commented temporarily due to the lack of support. I also added an SQL script to help set up the test environment, which creates tables and loads data. But I don't know how to run it in the test pipeline. Any suggestions or guidance are welcomed. I suppose the test pipeline would require a docker image of Risingwave. We can provide one if needed.

Yes, it would require relevant Docker Compose configuration be added to compose.yaml, and then the TestConf would be able to load the data (as you've already defined in ci/schema/risingwave.sql).

[KeXiangWang]

@deepyaman Hi Deepyaman, can you help invite more reviewers to push this PR forward. It seems that I have no access to invite.

[cpcloud]

@KeXiangWang Thanks for the PR and welcome to ibis 👋🏻, exciting work! I'll respond to your comments and questions inline.

I would appreciate if you can help test more queries.

We'll get into this a bit later, but ideally we don't need to crowd source testing because we have the backend test suite to help us out!

So in this PR, I introduce the new dependencies sqlalchemy-risingwave to ibis.

Sounds good! FYI we'll be moving away from sqlalchemy in two releases (version 9.0.0) and we'll be using sqlglot instead. In the case of risingwave I suspect that it will maintain a similar relationship to the postgres implementation, which is that a some of or a good chunk of the code can be shared.

Ibis has no support for Materialized View natively. ... We would appreciate it if you can help offer some suggestions.

Since RisingWave would be the first backend to support this, I suggest we avoid a new DDL object (these are in need of refactoring) for the time being and simply implement a create_materialize_view(self, ...) API directly on the RisingWave backend.

Risingwave hasn't supported TEMPORARY VIEW yet

Totally fine.

I didn't quite get what _metadata() is doing here.

_metadata accepts a query string and returns an iterator of 2-tuples, where the first element of the tuple is a column name and the second element is an ibis type. This method is used to discover the schema (names and types) of arbitrary queries. It's used in con.sql and TableExpr.sql.

how should I indicate this backend is only for risingwave > 1.6?

I think this is probably best communicated in our documentation.

I don't quite understand the test pipeline of Ibis.

Strap in 😂

Generally speaking the tests in ibis/backend/tests run for any and all backends. This is true regardless of backend architecture and deployment.

There are also usually backend-specific tests, those go in ibis/backends/$BACKEND/tests.

To run tests you'd write:

pytest -m risingwave

This will setup the data and find all the right tests (every test under ibis/backends/tests and every test under ibis/backends/risingwave/tests) and then run them.

The other piece of the puzzle is running RisingWave itself. I understand y'all publish a container, so that needs to be set up in our compose.yaml file.

When the PR is closer to ready (local tests all passing for at least one person 😅) we can roll it into CI, which in theory won't be that different from any of our other service-based backends like Trino or ClickHouse.

Excited to have you on board!

[chloeh13q]

Generally, from a maintainability perspective, I'm wondering if RisingWave intends to continue to maintain/increase compatibility with Postgres? @KeXiangWang If so, would it make sense to try to reuse functions and classes from the postgres backend as much as possible so that the same code doesn't have to be maintained in two places?

[KeXiangWang]

Hi @cpcloud . Thank you for the detailed explanation. I now understand how to enhance this PR and will make the necessary improvements ASAP.

[cpcloud]

😮 Extremely impressive, you've already got the backend working in CI (there's an XPASS, but that's generally good news and easily addressed by removing "risingwave" from the decorator's backend list).

[KeXiangWang]

All the tests have passed now. Can we first merge this PR? I prefer to continue on the critical features like create_materialized_view, create_source, and create_sink in new PRs. Let's carry it out step by step.

[gforsyth]

This looks good overall, @KeXiangWang !

I'd like to not merge in a bunch of commented-out tests. They should either be removed (especially if they've duplicative of the tests in the general backend test suite) or xfailed using the appropriate markers.

[lostmygithubaccount]

can we include this in the v8 release assuming it is merged today or tomorrow?

that would be better from a "we're announcing streaming support in Ibis" perspective, but also not critical (if it doesn't get in we'll just announce it's in progress for RisingWave)

[gforsyth]

I'm going to add the risingwave sqlalchemy dialect to conda-forge so we have the option of releasing a conda-forge version of this backend sooner rather than later

[KeXiangWang]

I'm going to add the risingwave sqlalchemy dialect to conda-forge.

Thanks! Anything we can help?

[gforsyth]

Hey @KeXiangWang -- I'm adding you as a feedstock maintainer, so if you can comment in conda-forge/staged-recipes#25146 saying that you agree to be a maintainer that will move this along.

[KeXiangWang]

Got it. It seems the name in conda-forge/staged-recipes#25146 is incorrect. My github handle is KeXiangWang instead of KeiXiangWang

[gforsyth]

Got it. It seems the name in conda-forge/staged-recipes#25146 is incorrect. My github handle is KeXiangWang instead of KeiXiangWang

Yeah, typo on my part, should be fixed now!

[cpcloud]

I'm able to run the backend test suite locally 🎉

I'm going to push up a few changes:

• I was able to reuse the minio configuration.
• Set the config file to a ro (readonly) volume mount
• Disable telemetry
• Reduce the # of lines of text for command
• Added a comment about using a release when posix_fs is launched
• Set restart to on-failure

I'll rebase on main first and then push up those changes.

[cpcloud]

@KeXiangWang Can you enable allowing maintainers to edit the pull request?

[cpcloud]

After removing the --config-path arguments and removing the /risingwave.toml config file volume mount the config file appears to be unnecessary.

[KeXiangWang]

enable allowing maintainers to edit the pull request?

Sorry I cannot find a button for that.

Alternatively, can you push the updates to a new branch? I can first rebase my branch on it, and then push the updates to this PR.

[deepyaman]

enable allowing maintainers to edit the pull request?

Sorry I cannot find a button for that.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests

You should see it on the right sidebar of the PR; I checked on one of mine.

[KeXiangWang]

You should see it on the right sidebar of the PR

I came across the same page. It's weird but there's no such a button for this PR.🙁

[cpcloud]

Actually, I think I'll just follow up with the changes in a separate PR.

@KeXiangWang Can you rebase on main and resolve the conflicts? Once that is done and CI is green I will merge!

[KeXiangWang]

Can you rebase on main and resolve the conflicts?

Done

[cpcloud]

Thanks @KeXiangWang great work!