Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-3582] Support PageIndex #4634

Merged
merged 24 commits into from
Mar 20, 2024
Merged

[GLUTEN-3582] Support PageIndex #4634

merged 24 commits into from
Mar 20, 2024

Commits on Mar 19, 2024

  1. Fix typo 

    (cherry picked from commit c3fbf13)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    80fc28b View commit details
    Browse the repository at this point in the history

  2. 1. using FutureSetFromTuple instead of FutureSetFromStorage. FutureSe… 

    …tFromTuple can buildOrderedSetInplace automatocally, FutureSetFromStorage need set Sizelimits mannually

2. Support PageIndex,  set spark.gluten.sql.columnar.backend.ch.runtime_config.use_local_format to true again.

3. Remove skipped test
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    d5f98a7 View commit details
    Browse the repository at this point in the history

  3. refactor gtest

    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    842fc94 View commit details
    Browse the repository at this point in the history

  4. fix build due to apache#4664

    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    99367b4 View commit details
    Browse the repository at this point in the history

  5. v2 for finding performance issue

    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    9c169dd View commit details
    Browse the repository at this point in the history

  6. Refactor: 

    add ParquetFileReaderExtBase
add readColumnChunkPageBase
simpilefy build read
remove redundant codes
reemove current_row_group_
std::vector<int32_t> row_groups_ => std::deque<int32_t> row_groups_
std::vector<std::unique_ptr<RowRanges>> row_group_row_ranges_ => std::unordered_map<int32_t, std::unique_ptr<RowRanges>> row_group_row_ranges_
std::vector<std::unique_ptr<ColumnIndexStore>> row_group_column_index_stores_ => std::unordered_map<int32_t, std::unique_ptr<ColumnIndexStore>> row_group_column_index_stores_;
remove std::vector<std::unique_ptr<parquet::RowGroupMetaData>> row_group_metas_;
remove std::vector<std::shared_ptr<parquet::RowGroupPageIndexReader>> row_group_index_readers_
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    75ae5d5 View commit details
    Browse the repository at this point in the history

  7. new loop

    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    8b1841a View commit details
    Browse the repository at this point in the history

  8. Cleanup

    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    dbef527 View commit details
    Browse the repository at this point in the history

  9. Cleanup

    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    5e802e1 View commit details
    Browse the repository at this point in the history

  10. Revert: fix build due to apache#4664

    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    44d5cbf View commit details
    Browse the repository at this point in the history

  11. support case_insensitive_column_matching of parquet 

    (cherry picked from commit bce0c6668d7bb397127eefeac1943d4c02cf79dc)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    06fb154 View commit details
    Browse the repository at this point in the history

  12. fix case_insensitive_column_matching issue 

    fix a stupid bug!
add testDataPath
getTpcdsDataPath() => tpcdsDataPath
getClickHouseLibPath() => clickHouseLibPath
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    8637c20 View commit details
    Browse the repository at this point in the history

  13. add benchmark 

    (cherry picked from commit bb0267135243ff8ad980b0521d8302e150a2c4e4)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    1508abb View commit details
    Browse the repository at this point in the history

  14. lowercase first letter of function name 

    (cherry picked from commit 98dc9a79bf4f372ecabcac9b47aa06cd328f1aa4)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    9c0e3f1 View commit details
    Browse the repository at this point in the history

  15. add comments 

    (cherry picked from commit 2fb41831f4e338503ff620ce5eac9917bdb68f6a)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    0eb1036 View commit details
    Browse the repository at this point in the history

  16. Remove Camel case member variable 

    (cherry picked from commit 1ace73205a033e14ca1659f063eb1df65c3e9969)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    10194cf View commit details
    Browse the repository at this point in the history

  17. Use Int32 instead of int32_t 

    (cherry picked from commit e7d8fbe701fcd92fb6cb167686602561adc26ec4)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    f74e98b View commit details
    Browse the repository at this point in the history

  18. Camel case for function name 

    (cherry picked from commit 1ee0516e2eadf045b4aec63de67cf5cb97810217)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    c30eaf3 View commit details
    Browse the repository at this point in the history

  19. add ColumnIndexFilterPtr alias 

    (cherry picked from commit 1e9cdd3b08eb4e026a739ee558e9c2dd0c4c88fb)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    7f935d9 View commit details
    Browse the repository at this point in the history

  20. using RowRangesMap = absl::flat_hash_map<Int32, std::unique_ptr<RowRa… 

    …nges>>;

using ColumnIndexStoreMap = absl::flat_hash_map<Int32, std::unique_ptr<ColumnIndexStore>>;

(cherry picked from commit 610fcd038d24d54fa30bcc40ab0d4d39f60dd0c4)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    a9386ab View commit details
    Browse the repository at this point in the history

  21. fix style 

    (cherry picked from commit 8d85db48fe1c93dbc05404aa580b3f11de94c51d)
    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    9ba0b1a View commit details
    Browse the repository at this point in the history

  22. fix benchmark due to apache#4995

    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    89fc32c View commit details
    Browse the repository at this point in the history

  23. fix build due to ClickHouse/ClickHouse#61502

    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    d74bf7b View commit details
    Browse the repository at this point in the history

  24. fix assertion failed in Debug Build

    @baibaichen
    baibaichen committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    a4de18d View commit details
    Browse the repository at this point in the history