Support rebase checking for nested dates and timestamps #9617
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ttnghia
added
the
task
Signed-off-by: Nghia Truong <[email protected]>
ttnghia
force-pushed
the
check_rebase_nested
branch
from
534e647
to
7343b17
Compare
|
build |
ttnghia
commented
Nov 3, 2023
|
build |
andygrove
reviewed
Nov 3, 2023
| s"${SparkShimImpl.int96ParquetRebaseReadKey} is EXCEPTION") | ||
| } | ||
| case "CORRECTED" => // Good | ||
| case "EXCEPTION" | "CORRECTED" => // Good |
There was a problem hiding this comment.
Why is EXCEPTION ok now? Do we have any tests that exercise this path?
There was a problem hiding this comment.
EXCEPTION is handled somewhere else, after loading data. This is the planning step, thus we are OK to run on the GPU with it.
There was a problem hiding this comment.
We have test for it too. That is one of the two that I modified.
There was a problem hiding this comment.
I was confused because I didn't see EXCEPTION explicitly in the tests, but I checked the docs, and it is the default setting for datetimeRebaseModeInRead, so it is covered.
andygrove
approved these changes
Nov 3, 2023
ttnghia
added a commit
to ttnghia/spark-rapids
that referenced
this pull request
Nov 6, 2023
…IA#9617)" This reverts commit 401d0d8. Signed-off-by: Nghia Truong <[email protected]>
ttnghia
added a commit
that referenced
this pull request
Nov 6, 2023
…" (#9641) This reverts commit 401d0d8. Signed-off-by: Nghia Truong <[email protected]>
ttnghia
added a commit
to ttnghia/spark-rapids
that referenced
this pull request
Nov 6, 2023
…IA#9617)" This reverts commit 401d0d8. Signed-off-by: Nghia Truong <[email protected]> # Conflicts: # sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
ttnghia
added a commit
that referenced
this pull request
Nov 17, 2023
…owed timestamp [databricks] (#9736) * Add check for nested types * Add check for nested types * Recursively check for rebasing * Extract common code * Allow nested type in rebase check * Enable nested timestamp in roundtrip test * Fix another test Signed-off-by: Nghia Truong <[email protected]> * Enable `LEGACY` rebase in read * Remove comment * Change function/class signatures * Complete modification * Misc Signed-off-by: Nghia Truong <[email protected]> * Add explicit type Signed-off-by: Nghia Truong <[email protected]> * Rename file and add some stuff in DateTimeRebaseHelpers.scala * Move file and rename class * Adopt new enum type Signed-off-by: Nghia Truong <[email protected]> * Add name for the enum classes * Change exception messages * Does not yet support legacy rebase in read Signed-off-by: Nghia Truong <[email protected]> * Change legacy to corrected mode Signed-off-by: Nghia Truong <[email protected]> * Extract common code Signed-off-by: Nghia Truong <[email protected]> * Rename functions Signed-off-by: Nghia Truong <[email protected]> * Reformat Signed-off-by: Nghia Truong <[email protected]> * Make classes serializable Signed-off-by: Nghia Truong <[email protected]> * Revert "Support rebase checking for nested dates and timestamps (#9617)" This reverts commit 401d0d8. Signed-off-by: Nghia Truong <[email protected]> # Conflicts: # sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala * Implement date time rebase * Optimize rebase op * Change comment Signed-off-by: Nghia Truong <[email protected]> * Move tests * Add test for datatime rebase Signed-off-by: Nghia Truong <[email protected]> * Various changes Signed-off-by: Nghia Truong <[email protected]> * Various changes Signed-off-by: Nghia Truong <[email protected]> # Conflicts: # sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala * Fix compile errors Signed-off-by: Nghia Truong <[email protected]> * Fix comments Signed-off-by: Nghia Truong <[email protected]> * Fix indentations Signed-off-by: Nghia Truong <[email protected]> * Change comments and indentations Signed-off-by: Nghia Truong <[email protected]> * Allow nested check for rebase * Write different timestamp types in test * Fix conversion if timestamp is not micros * Rename var * Dont have to down cast after up cast Signed-off-by: Nghia Truong <[email protected]> * Change comment Signed-off-by: Nghia Truong <[email protected]> * Still cast timestamp to the old type after rebasing Signed-off-by: Nghia Truong <[email protected]> * Rename test Signed-off-by: Nghia Truong <[email protected]> * Should not transform non-datetime types Signed-off-by: Nghia Truong <[email protected]> * Fix test * Update tests Signed-off-by: Nghia Truong <[email protected]> * Enable int96 rebase in write * Change tests Signed-off-by: Nghia Truong <[email protected]> * Complete tests Signed-off-by: Nghia Truong <[email protected]> * Revert unrelated changes Signed-off-by: Nghia Truong <[email protected]> * Change configs Signed-off-by: Nghia Truong <[email protected]> * Merge tests Signed-off-by: Nghia Truong <[email protected]> * Simplify test data Signed-off-by: Nghia Truong <[email protected]> * Add a new write test Signed-off-by: Nghia Truong <[email protected]> * Add a mixed rebase test Signed-off-by: Nghia Truong <[email protected]> * Change tests Signed-off-by: Nghia Truong <[email protected]> * Fix `seed` in tests Signed-off-by: Nghia Truong <[email protected]> * Rename tests Signed-off-by: Nghia Truong <[email protected]> * Remove seed override * Change TimestampGen Signed-off-by: Nghia Truong <[email protected]> * Remove default seed Signed-off-by: Nghia Truong <[email protected]> * Add default seed Signed-off-by: Nghia Truong <[email protected]> * Remove default seed Signed-off-by: Nghia Truong <[email protected]> --------- Signed-off-by: Nghia Truong <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements recursive checking to see if the input column contains nested dates/timestamps that need to be rebased.
Some directly related tests are fixed. Some others need to be cleanup/fixed after having
LEGACYrebase mode support in parquet reading.Partially contributes to #1126.