Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#3888] Metadata Support ClickHouse #5819

Open
wants to merge 19 commits into
base: main
Choose a base branch
from

Conversation

flaming-archer
Copy link

Title

  1. [[FEATURE] Support Clickhouse catalog #3888] feat(operator): support clickhouse"

What changes were proposed in this pull request?

Support clickhouse as a metadata.

Why are the changes needed?

No new api.
Because ck is a new metadata, there has been a change in page classification.

@flaming-archer flaming-archer changed the title Clickhouse 0.7 forpr [#3888] Metadata Support ClickHouse Dec 10, 2024
@flaming-archer
Copy link
Author

@shaofengshi @xunliu Pls help to review this pr.

Copy link
Contributor

@tengqm tengqm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we split this to a few smaller PRs?
One PR with 5000+ lines changes in 38 files is too big to review.


@Override
public String fromGravitino(Type type) {
if (type instanceof Types.ByteType) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd do typeName = type.getTypeName() and use a switch statement for the name.
The if ... else if is a headache to me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My gut feeling is that this conversion between type and type names can be generalized
and then put to JdbcTypeConverter.
We can then maintain the peculiarities for ClickHouse here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd do typeName = type.getTypeName() and use a switch statement for the name. The if ... else if is a headache to me.

MySql, Doris,OceanBase, psql ,it's all like this. Perhaps maintaining this would be better?Should we use one issue to make the changes uniformly in the future.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My gut feeling is that this conversion between type and type names can be generalized and then put to JdbcTypeConverter. We can then maintain the peculiarities for ClickHouse here.

I thinl, in fact, it is simply taking its own type from the gravitino type, processing mostly the types of its own engine, most of which are different. Unless the type of gravitino is separated and a callback is written to return its own type. But the code is also quite obscure.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My suggestion was to promote this to the JDBC level from ClickHouse level.
I was not suggesting to promote this to the Gravitino level.

@flaming-archer
Copy link
Author

Can we split this to a few smaller PRs? One PR with 5000+ lines changes in 38 files is too big to review.

Among the 5000 lines, most of them are ck code, and most of them are ck test code. As a new catlog, it may still be quite normal.

@mchades
Copy link
Contributor

mchades commented Dec 11, 2024

Hi @flaming-archer thanks a lot for your contribution. Since this is a big feature, would you please create a epic issue and break down things into small tasks, also it would be nice to have a design doc beforehand. You can check and follow what OceanBase did (#4848). Thanks a lot.

@jerryshao
Copy link
Contributor

Hi @flaming-archer thanks a lot for your contribution, can we just break down into small subtasks, and start with a design doc? @mchades from the community will help you to get everything merged. Thanks a lot.

@flaming-archer
Copy link
Author

Hi @flaming-archer thanks a lot for your contribution. Since this is a big feature, would you please create a epic issue and break down things into small tasks, also it would be nice to have a design doc beforehand. You can check and follow what OceanBase did (#4848). Thanks a lot.

OK

@flaming-archer
Copy link
Author

Hi @flaming-archer thanks a lot for your contribution, can we just break down into small subtasks, and start with a design doc? @mchades from the community will help you to get everything merged. Thanks a lot.

OK

Comment on lines 65 to 66
try (Statement statement = connection.createStatement()) {
try (ResultSet resultSet = statement.executeQuery("SHOW DATABASES")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what we gain from these three levels of try blocks ...

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what we gain from these three levels of try blocks ...

fixed

private static String quote(String name) {
return BACK_QUOTE + name + BACK_QUOTE;
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sir. This file is not trivial. Please consider move it out to a separate PR.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sir. This file is not trivial. Please consider move it out to a separate PR.

There are indeed many unique logic behind it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants