Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#1483] POC of spark-connector to query simple hive table #1521

Closed
wants to merge 1 commit into from

Conversation

FANNG1
Copy link
Contributor

@FANNG1 FANNG1 commented Jan 16, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Fix: #1483

Does this PR introduce any user-facing change?

How was this patch tested?

my local environment with HDFS and Hive

@FANNG1
Copy link
Contributor Author

FANNG1 commented Jan 16, 2024

A simple POC of read&write simple hive data, some details should be expended, if the overall architecture is ok, I'll continue to polish it and split small PRs to try to merge in 0.4:

  1. use scala related jars with 2.12 not 2.13, because spark hive connector only supports scala 2.12
  2. the basic catalog skeleton include GravitinoPlugin GravitinoCatalogManager GravitinoCatalog without detailed implement.
  3. hive table support, includes create table, load table, write table, read table

@jerryshao @qqqttt123 any suggestions to the current implement skeleton and next plan? please skip some code details.

@coolderli
Copy link
Contributor

Should we support a different spark version? In different versions, there may be some differences. For example, the timestampntz type is introduced on spark3.4 but is not supported by the older version.

@FANNG1
Copy link
Contributor Author

FANNG1 commented Jan 23, 2024

Should we support a different spark version? In different versions, there may be some differences. For example, the timestampntz type is introduced on spark3.4 but is not supported by the older version.

yes, plan to support multi spark version in #1574 after the basic function is setup.

@FANNG1 FANNG1 closed this Mar 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Subtask] POC of spark-connector to query simple hive table
2 participants