discussion: Support creating lakehouse catalog in risingwave.

[liurenjie1024]

Is your feature request related to a problem? Please describe.

Currently, before risingwave can sink to iceberg, we need to rely on some external system(flink, spark, etc) to create table for us. There are two drawbacks with this problem:

1. It would be quite inconvenient for users to try our solution. For example, if user want just want to store everything in s3, they need to download flink, hadoop, and get familiar with flink SQL to create catalog/tables.
2. DDLs in two different systems is quite an error prone and misleading.

Describe the solution you'd like

Support creating iceberg catalog in risingwave as following:

CREATE CATALOG fs_catalog WITH (
  'type'='iceberg',
  'catalog-type'='hadoop',
  'warehouse' = 'file:///Users/renjieliu/Downloads/a'
);

USE fs_catalog;

CREATE DATABASE demo_db;

USE demo_db;

CREATE TABLE demo_table (
  v1 int,
  v2 bigint,
  v3 varchar
);

With this enabled, user can ingesting our solution to iceberg in one shot without any other dependencies.

Describe alternatives you've considered

No response

Additional context

No response

[StrikeW]

After DDL is supported, do you plan to support reading and writing to the demo_table (lakehouse tables) directly in RisingWave?

[neverchanje]

The reason why RisingWave supports Iceberg is to fit into users' existing tech stack where they have used Spark and Iceberg already. Iceberg is not a out-of-the-box system that you can use without any other dependencies. It's still required a separate process to run compaction, serve ad-hoc queries, and to manage catalogs. It will be non-trivial to build a management system for Iceberg. Therefore, from the product's perspective, I would view lakehouse as a completely distinct product line that RisingWave should only provide minimal functions to integrate, instead of combining them together.

Furthermore, you also need to consider what if the user's Iceberg is hosted by Tabular, Dremio, or any full-fledged lakehouse. It will then be unnecessary to create the lake in RisingWave.

[liurenjie1024]

After DDL is supported, do you plan to support reading and writing to the demo_table (lakehouse tables) directly in RisingWave?

I think reading without optimization will not require much effort and is worth doing. DML statement maybe complicated to implement, so I don't think we should do it.

[liurenjie1024]

The reason why RisingWave supports Iceberg is to fit into users' existing tech stack where they have used Spark and Iceberg already. Iceberg is not a out-of-the-box system that you can use without any other dependencies. It's still required a separate process to run compaction, serve ad-hoc queries, and to manage catalogs. It will be non-trivial to build a management system for Iceberg. Therefore, from the product's perspective, I would view lakehouse as a completely distinct product line that RisingWave should only provide minimal functions to integrate, instead of combining them together.

Agreed. However, our goal is not to build a fully managed solution for Iceberg. Rather, we want to make RisingWave easier to use and experiment with. Adding support for DDL statements will simplify integration tests, as it eliminates the need to download other systems like Spark, Flink, or Hadoop to create a catalog for us. This will also make it easier for beginners who want to try out RisingWave with Iceberg. This is similar to other DML statements in our system, which are not intended for production use, but rather make tests and experimentation easier. cc @neverchanje

[fuyufjh]

After DDL is supported, do you plan to support reading and writing to the demo_table (lakehouse tables) directly in RisingWave?

I think reading without optimization will not require much effort and is worth doing. DML statement maybe complicated to implement, so I don't think we should do it.

@StrikeW's comment is also my major concern. If we support these CREATE CATALOG ... commands as you mentioned, it looks like RisingWave has a full support for Iceberg including DDL & CRUD, but that's not the truth.

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.