Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add in checks for Parquet LEGACY date/time rebase #435

Merged
merged 3 commits into from
Jul 27, 2020

Conversation

revans2
Copy link
Collaborator

@revans2 revans2 commented Jul 27, 2020

Add in some checks to better support spark.sql.legacy.parquet.datetimeRebaseModeInWrite and spark.sql.legacy.parquet.datetimeRebaseModeInRead.

Follow on work would be needed to fully support them, but this at least make it possible for us to corrupt the data. Either we fail on a read if we would corrupt the data, or we fall back to the CPU on writes.

This fixes #144

@revans2 revans2 added bug Something isn't working SQL part of the SQL/Dataframe plugin labels Jul 27, 2020
@revans2 revans2 added this to the Jul 20 - Jul 31 milestone Jul 27, 2020
@revans2 revans2 self-assigned this Jul 27, 2020
@revans2
Copy link
Collaborator Author

revans2 commented Jul 27, 2020

build

Signed-off-by: Robert (Bobby) Evans <[email protected]>
@revans2
Copy link
Collaborator Author

revans2 commented Jul 27, 2020

build

@revans2 revans2 merged commit eb7b329 into NVIDIA:branch-0.2 Jul 27, 2020
@revans2 revans2 deleted the date_time_rebase_start branch July 27, 2020 18:01
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
pxLi pushed a commit to pxLi/spark-rapids that referenced this pull request May 12, 2022
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this pull request Nov 30, 2023
Signed-off-by: spark-rapids automation <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working SQL part of the SQL/Dataframe plugin
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] spark.sql.legacy.parquet.datetimeRebaseModeInWrite is ignored
3 participants