Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Subtask] [spark connector] support hive table location properties #2621

Closed
FANNG1 opened this issue Mar 21, 2024 · 11 comments · Fixed by #2805
Closed

[Subtask] [spark connector] support hive table location properties #2621

FANNG1 opened this issue Mar 21, 2024 · 11 comments · Fixed by #2805
Assignees
Labels
good first issue Good for newcomers subtask Subtasks of umbrella issue

Comments

@FANNG1
Copy link
Contributor

FANNG1 commented Mar 21, 2024

Describe the subtask

supports location properties, transform from Spark location to Gravitino location, please add UT to TestHivePropertiesConverter, add IT to SparkHiveCatalogIT.

CREATE TABLE xx LOCATION xxx

Parent issue

#1549

@FANNG1 FANNG1 added the subtask Subtasks of umbrella issue label Mar 21, 2024
@FANNG1
Copy link
Contributor Author

FANNG1 commented Mar 21, 2024

@Yangxuhao123 you could refer #2605

@FANNG1 FANNG1 added this to the Gravitino 0.5.0 milestone Mar 21, 2024
@FANNG1 FANNG1 assigned FANNG1 and unassigned FANNG1 Mar 21, 2024
@Yangxuhao123
Copy link
Contributor

Sorry, I've dealt with some things during this time. Have you completed this part of the work? @FANNG1

@FANNG1 FANNG1 added the good first issue Good for newcomers label Apr 1, 2024
@FANNG1
Copy link
Contributor Author

FANNG1 commented Apr 1, 2024

Sorry, I've dealt with some things during this time. Have you completed this part of the work? @FANNG1

No, I haven't finished it, do you have time to work on this?

@Yangxuhao123
Copy link
Contributor

Ok, I will do it.

@charliecheng630
Copy link
Contributor

May I take on this issue?

@FANNG1
Copy link
Contributor Author

FANNG1 commented Apr 3, 2024

May I take on this issue?

@Yangxuhao123 is working on this

@Yangxuhao123
Copy link
Contributor

May I take on this issue?

OK, you can do it, I can work on other issues in the future.

@charliecheng630
Copy link
Contributor

charliecheng630 commented Apr 4, 2024

@FANNG1 Should I support external property? CREATE EXTERNAL TABLE xxx LOCATION xxx

three testing scenario cases I thought:

  1. put a file into HDFS first
    then CREATE EXTERNAL TABLE xxx LOCATION xxx
    checkTableReadWrite
    dropTable
    checkDataFileExists

  2. CREATE EXTERNAL TABLE xxx LOCATION xxx
    checkTableReadWrite
    dropTable
    checkDataFileExists

  3. CREATE TABLE xxx LOCATION xxx
    checkTableReadWrite
    dropTable
    checkDataFileExists

What are your thoughts?

@FANNG1
Copy link
Contributor Author

FANNG1 commented Apr 4, 2024

For location property, the main target of the test is to check whether data is placed in the specified location, so I think we should test the following scenes for managed table and external table.

CREATE [EXTERNAL] TABLE xxx LOCATION xxx

  1. checkTableReadWrite
  2. checkTablelocation equals the specified location. please note that hive may change the location from /xxpath to hdfs://xx:xx/xxpath
  3. checkDataFileExists

@charliecheng630
Copy link
Contributor

@FANNG1 the key of location property in Gravitino is same as Spark, do we still need to modify the converter?

@FANNG1
Copy link
Contributor Author

FANNG1 commented Apr 4, 2024

@FANNG1 the key of location property in Gravitino is same as Spark, do we still need to modify the converter?

I think it's necessary to transform it explicitly, they are happens to the same.

FANNG1 pushed a commit that referenced this issue Apr 8, 2024
…2805)

### What changes were proposed in this pull request?

supports location properties, transform from Spark location to Gravitino
location
`CREATE TABLE xx LOCATION xxx`

### Why are the changes needed?

Fix: #2621 

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

IT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers subtask Subtasks of umbrella issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants