Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support spark.sql.parquet.datetimeRebaseModeInRead=LEGACY and spark.sql.parquet.int96RebaseModeInRead=LEGACY [databricks] #9649

Merged
merged 62 commits into from
Nov 15, 2023
Merged

Support spark.sql.parquet.datetimeRebaseModeInRead=LEGACY and spark.sql.parquet.int96RebaseModeInRead=LEGACY [databricks] #9649

merged 62 commits into from
Nov 15, 2023

Commits on Nov 2, 2023

  1. Add check for nested types

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    e368aa6 View commit details
    Browse the repository at this point in the history

  2. Recursively check for rebasing

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    7da416b View commit details
    Browse the repository at this point in the history

  3. Extract common code

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    df8f861 View commit details
    Browse the repository at this point in the history

  4. Allow nested type in rebase check

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    95d19ee View commit details
    Browse the repository at this point in the history

  5. Enable nested timestamp in roundtrip test

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    b426610 View commit details
    Browse the repository at this point in the history

  6. Fix another test 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    7343b17 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'check_rebase_nested' into rebase_datatime

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    0d48f57 View commit details
    Browse the repository at this point in the history

  8. Enable LEGACY rebase in read

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    024e6c9 View commit details
    Browse the repository at this point in the history

  9. Remove comment

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    9a39628 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2023

  1. Change function/class signatures

    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    e686bb0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-23.12' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/RebaseHelper.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    b49963e View commit details
    Browse the repository at this point in the history

  3. Complete modification

    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    2c232f8 View commit details
    Browse the repository at this point in the history

  4. Misc 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    ac0f3e4 View commit details
    Browse the repository at this point in the history

  5. Add explicit type 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    c773794 View commit details
    Browse the repository at this point in the history

  6. Rename file and add some stuff in DateTimeRebaseHelpers.scala

    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    29df7cd View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2023

  1. Move file and rename class

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    1b5112d View commit details
    Browse the repository at this point in the history

  2. Adopt new enum type 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    63342a9 View commit details
    Browse the repository at this point in the history

  3. Add name for the enum classes

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    6b2d795 View commit details
    Browse the repository at this point in the history

  4. Change exception messages

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    37aa40b View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-23.12' into refactor_parquet_scan

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    d4cdc1b View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2023

  1. Does not yet support legacy rebase in read 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    03f681e View commit details
    Browse the repository at this point in the history

  2. Change legacy to corrected mode 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    14f230f View commit details
    Browse the repository at this point in the history

  3. Extract common code 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    1b464ec View commit details
    Browse the repository at this point in the history

  4. Rename functions 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    0d26d97 View commit details
    Browse the repository at this point in the history

  5. Reformat 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    c2504fd View commit details
    Browse the repository at this point in the history

  6. Make classes serializable 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    edb6c81 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2023

  1. Revert "Support rebase checking for nested dates and timestamps (NVID… 

    …IA#9617)"

This reverts commit 401d0d8.

Signed-off-by: Nghia Truong <[email protected]>

# Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    ea86e8f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'refactor_parquet_scan' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/RebaseHelper.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    b14463f View commit details
    Browse the repository at this point in the history

  3. Implement date time rebase

    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    adc8ae2 View commit details
    Browse the repository at this point in the history

  4. Optimize rebase op

    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    791573c View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-23.12' into refactor_parquet_scan 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/RebaseHelper.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    54e959f View commit details
    Browse the repository at this point in the history

  6. Change comment 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    3f01690 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'refactor_parquet_scan' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    6d9c20b View commit details
    Browse the repository at this point in the history

  8. Move tests

    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    8c63273 View commit details
    Browse the repository at this point in the history

  9. Add test for datatime rebase 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    1b1fdc3 View commit details
    Browse the repository at this point in the history

  10. Various changes 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    e6559ce View commit details
    Browse the repository at this point in the history

  11. Various changes 

    Signed-off-by: Nghia Truong <[email protected]>

# Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    74fe84a View commit details
    Browse the repository at this point in the history

  12. Fix compile errors 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    a455a90 View commit details
    Browse the repository at this point in the history

  13. Fix comments 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    b87493c View commit details
    Browse the repository at this point in the history

  14. Fix indentations 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    321e516 View commit details
    Browse the repository at this point in the history

  15. Merge branch 'refactor_parquet_scan' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/datetimeRebaseUtils.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    4bc33be View commit details
    Browse the repository at this point in the history

  16. Change comments and indentations 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    4aab36b View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. Merge branch 'branch-23.12' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    c615925 View commit details
    Browse the repository at this point in the history

  2. Write different timestamp types in test

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    be92368 View commit details
    Browse the repository at this point in the history

  3. Fix conversion if timestamp is not micros

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    b09c61f View commit details
    Browse the repository at this point in the history

  4. Rename var

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    00d96e4 View commit details
    Browse the repository at this point in the history

  5. Dont have to down cast after up cast 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    7d81311 View commit details
    Browse the repository at this point in the history

  6. Change comment 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    116bf3e View commit details
    Browse the repository at this point in the history

  7. Still cast timestamp to the old type after rebasing 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    273b2c4 View commit details
    Browse the repository at this point in the history

  8. Rename test 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    996d9d4 View commit details
    Browse the repository at this point in the history

  9. Should not transform non-datetime types 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    5fd6ef5 View commit details
    Browse the repository at this point in the history

  10. Fix test

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    4144655 View commit details
    Browse the repository at this point in the history

  11. Update tests 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    5a8b44c View commit details
    Browse the repository at this point in the history

  12. Change tests 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    247f47f View commit details
    Browse the repository at this point in the history

  13. Merge branch 'branch-23.12' into rebase_datatime

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    fbe37d7 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2023

  1. Change configs 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    4a92d54 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. Merge branch 'branch-23.12' into rebase_datatime

    @ttnghia
    ttnghia committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    238d7dc View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2023

  1. Merge branch 'branch-23.12' into rebase_datatime

    @ttnghia
    ttnghia committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    29367d4 View commit details
    Browse the repository at this point in the history

  2. Add comment for TimestampNTZType 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    bd9ee93 View commit details
    Browse the repository at this point in the history

  3. Test with mixed rebase configs 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    8cf4820 View commit details
    Browse the repository at this point in the history

  4. Handle different rebase configs 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    74bef2e View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2023

  1. Merge branch 'branch-23.12' into rebase_datatime

    @ttnghia
    ttnghia committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    8930fbd View commit details
    Browse the repository at this point in the history