Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#655] feat(jdbc): Initialize the JDBC module in Gravitino. #656

Merged
merged 7 commits into from
Nov 8, 2023

Conversation

Clearvive
Copy link
Contributor

@Clearvive Clearvive commented Nov 2, 2023

What changes were proposed in this pull request?

Regarding JDBC as a catalog, we should complete its design on how to use it in Gravitino.

Why are the changes needed?

Fix: #573

Does this PR introduce any user-facing change?

No

How was this patch tested?

No

Copy link

github-actions bot commented Nov 2, 2023

Code Coverage Report

Overall Project 67.24% 🟢

There is no coverage information present for the Files changed

@Clearvive Clearvive force-pushed the 655/jdbc-catalog-v1 branch from b51924b to 4fafa91 Compare November 2, 2023 03:56
@Clearvive Clearvive marked this pull request as ready for review November 2, 2023 03:56
@Clearvive Clearvive self-assigned this Nov 2, 2023
@Clearvive Clearvive added this to the Graviton 0.3.0 milestone Nov 2, 2023
@Clearvive Clearvive force-pushed the 655/jdbc-catalog-v1 branch from 4fafa91 to bcf3e40 Compare November 2, 2023 03:57
@jerryshao
Copy link
Contributor

@Clearvive is it ready for review?

@Clearvive
Copy link
Contributor Author

@Clearvive is it ready for review?

Yes, I have split the pull request for easier review.

@jerryshao
Copy link
Contributor

@FANNG1 would you please also take a review.

@FANNG1
Copy link
Contributor

FANNG1 commented Nov 3, 2023

@FANNG1 would you please also take a review.

ok

@jerryshao
Copy link
Contributor

@Clearvive can you please briefly describe how do you want to organize your code regard to jdbc catalog, say if we have common jdbc code and specific jdbc implementations (MySQL)?

I think we're not in the same page, it would be better if you can briefly describe it here.

@Clearvive Clearvive force-pushed the 655/jdbc-catalog-v1 branch from c14200c to eaa6225 Compare November 6, 2023 07:03
@jerryshao
Copy link
Contributor

@Clearvive can you please briefly describe how do you want to organize your code regard to jdbc catalog, say if we have common jdbc code and specific jdbc implementations (MySQL)?

I think we're not in the same page, it would be better if you can briefly describe it here.

@Clearvive please reply this.

@Clearvive
Copy link
Contributor Author

can you please briefly describe how do you want to organize your code regard to jdbc catalog, say if we have common jdbc code and specific jdbc implementations (MySQL)?

If we are dealing with MySQL, there will be a MysqlCatalog class that extends JdbcCatalog and implements the shortName method. Within the JdbcCatalogOperations class, there will be two specific operators for schema and table operations: JdbcDatabaseOperations and JdbcTableOperations. For special SQL operations, inheritors need to override the methods of these two operators.

Additionally, the corresponding directory structure for our modules is as follows:

  • catalog-jdbc-common --> com.datastrato.gravitino.catalog.jdbc
  • catalog-jdbc-mysql --> com.datastrato.gravitino.catalog.jdbc.mysql
  • catalog-jdbc-postgresql --> com.datastrato.gravitino.catalog.jdbc.postgresql

@jerryshao
Copy link
Contributor

can you please briefly describe how do you want to organize your code regard to jdbc catalog, say if we have common jdbc code and specific jdbc implementations (MySQL)?

If we are dealing with MySQL, there will be a MysqlCatalog class that extends JdbcCatalog and implements the shortName method. Within the JdbcCatalogOperations class, there will be two specific operators for schema and table operations: JdbcDatabaseOperations and JdbcTableOperations. For special SQL operations, inheritors need to override the methods of these two operators.

Additionally, the corresponding directory structure for our modules is as follows:

  • catalog-jdbc-common --> com.datastrato.gravitino.catalog.jdbc
  • catalog-jdbc-mysql --> com.datastrato.gravitino.catalog.jdbc.mysql
  • catalog-jdbc-postgresql --> com.datastrato.gravitino.catalog.jdbc.postgresql

I remembered that @mchades brought out a discussion about whether we should use different catalogs (shortName) to represent different jdbc sources, or we use a unified jdbc catalog with different types (MySQL, PostgreSQL...).

One thing I am concerned about is that if we have many jdbc databases supported, we may end up with many different implementations, and potentially many duplicated codes. So what is your thought?

@FANNG1
Copy link
Contributor

FANNG1 commented Nov 8, 2023

can you please briefly describe how do you want to organize your code regard to jdbc catalog, say if we have common jdbc code and specific jdbc implementations (MySQL)?

If we are dealing with MySQL, there will be a MysqlCatalog class that extends JdbcCatalog and implements the shortName method. Within the JdbcCatalogOperations class, there will be two specific operators for schema and table operations: JdbcDatabaseOperations and JdbcTableOperations. For special SQL operations, inheritors need to override the methods of these two operators.
Additionally, the corresponding directory structure for our modules is as follows:

  • catalog-jdbc-common --> com.datastrato.gravitino.catalog.jdbc
  • catalog-jdbc-mysql --> com.datastrato.gravitino.catalog.jdbc.mysql
  • catalog-jdbc-postgresql --> com.datastrato.gravitino.catalog.jdbc.postgresql

I remembered that @mchades brought out a discussion about whether we should use different catalogs (shortName) to represent different jdbc sources, or we use a unified jdbc catalog with different types (MySQL, PostgreSQL...).

One thing I am concerned about is that if we have many jdbc databases supported, we may end up with many different implementations, and potentially many duplicated codes. So what is your thought?

Offline talked to @mchades @Clearvive , it's hard to just change a type to reuse existing Jdbc-connectors like MySQL, because Jdbc is mainly for commucatition. Every engine had it's specify grammar, properties, type systems. see more in #651

@jerryshao
Copy link
Contributor

OK, I see.

@jerryshao
Copy link
Contributor

@FANNG1 can you please help to review again.

catalogs/catalog-jdbc-common/build.gradle.kts Outdated Show resolved Hide resolved
build.gradle.kts Outdated Show resolved Hide resolved
@FANNG1
Copy link
Contributor

FANNG1 commented Nov 8, 2023

LGTM

@jerryshao jerryshao merged commit b1e19c3 into apache:main Nov 8, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[EPIC] Add Jdbc catalog support in Graviton
5 participants