Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for struct type in ORC writer #9025

Merged
merged 82 commits into from
Sep 22, 2021
Merged

Add support for struct type in ORC writer #9025

merged 82 commits into from
Sep 22, 2021

Commits on Jul 28, 2021

  1. enable struct columns on the python layer; enable stream creation for… 

    … struct columns
    @vuule
    vuule committed Jul 28, 2021
    Configuration menu
    Copy the full SHA
    b28b2b9 View commit details
    Browse the repository at this point in the history

  2. move input metadata types to types.hpp

    @vuule
    vuule committed Jul 28, 2021
    Configuration menu
    Copy the full SHA
    77e07eb View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2021

  1. replace table_metadata and table_metadata_with_nullability; update tests

    @vuule
    vuule committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    d983db4 View commit details
    Browse the repository at this point in the history

  2. simplify table creation in orc tests

    @vuule
    vuule committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    524f7db View commit details
    Browse the repository at this point in the history

  3. metadata

    @vuule
    vuule committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    295312a View commit details
    Browse the repository at this point in the history

  4. chunked metadata test fix; nullable getter fix

    @vuule
    vuule committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    f6dba5b View commit details
    Browse the repository at this point in the history

  5. add list column to a metadata test

    @vuule
    vuule committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    6940680 View commit details
    Browse the repository at this point in the history

  6. Revert "add list column to a metadata test" 

    This reverts commit 6940680.
    @vuule
    vuule committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    08bf157 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2021

  1. refactor regular and chunked orc writer options

    @vuule
    vuule committed Jul 30, 2021
    Configuration menu
    Copy the full SHA
    3abf86d View commit details
    Browse the repository at this point in the history

  2. add the metadata + list test again

    @vuule
    vuule committed Jul 30, 2021
    Configuration menu
    Copy the full SHA
    c06fcbc View commit details
    Browse the repository at this point in the history

  3. generate default column names

    @vuule
    vuule committed Jul 30, 2021
    Configuration menu
    Copy the full SHA
    eec7941 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2021

  1. subtype footer

    @vuule
    vuule committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    7c74d95 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-writer-struct
    @vuule
    vuule committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    03bd98f View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2021

  1. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-writer-struct
    @vuule
    vuule committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    94d9892 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2021

  1. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-writer-struct
    @vuule
    vuule committed Aug 10, 2021
    Configuration menu
    Copy the full SHA
    6e83117 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2021

  1. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-writer-struct
    @vuule
    vuule committed Aug 12, 2021
    Configuration menu
    Copy the full SHA
    c90d04a View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2021

  1. don't use valid_buf as null_mask; remove a redundant syncthreads

    @vuule
    vuule committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    0fa6aa7 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2021

  1. typo fix

    @vuule
    vuule committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    f9cec92 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-write-struct
    @vuule
    vuule committed Aug 17, 2021
    Configuration menu
    Copy the full SHA
    8e333c9 View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2021

  1. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-write-struct
    @vuule
    vuule committed Aug 18, 2021
    Configuration menu
    Copy the full SHA
    4a65fad View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2021

  1. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-write-struct
    @vuule
    vuule committed Aug 20, 2021
    Configuration menu
    Copy the full SHA
    184c686 View commit details
    Browse the repository at this point in the history

  2. pushdown null masks pt 1

    @vuule
    vuule committed Aug 20, 2021
    Configuration menu
    Copy the full SHA
    444d4cf View commit details
    Browse the repository at this point in the history

  3. pushdown null masks pt2

    @vuule
    vuule committed Aug 20, 2021
    Configuration menu
    Copy the full SHA
    df2e916 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2021

  1. pushdown mask different approach

    @vuule
    vuule committed Aug 21, 2021
    Configuration menu
    Copy the full SHA
    6e76ccc View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2021

  1. pushdown for struct - host side copmplete

    @vuule
    vuule committed Aug 24, 2021
    Configuration menu
    Copy the full SHA
    7682899 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-write-struct
    @vuule
    vuule committed Aug 24, 2021
    Configuration menu
    Copy the full SHA
    11eef31 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2021

  1. null mask row group alignment - lists only

    @vuule
    vuule committed Aug 26, 2021
    Configuration menu
    Copy the full SHA
    9780279 View commit details
    Browse the repository at this point in the history

  2. move alignment to separate function

    @vuule
    vuule committed Aug 26, 2021
    Configuration menu
    Copy the full SHA
    4b7627f View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2021

  1. per rg valid count stub

    @vuule
    vuule committed Aug 27, 2021
    Configuration menu
    Copy the full SHA
    f965f3c View commit details
    Browse the repository at this point in the history

  2. set pd masks in d_column views

    @vuule
    vuule committed Aug 27, 2021
    Configuration menu
    Copy the full SHA
    b7c86a1 View commit details
    Browse the repository at this point in the history

  3. complete per rg valid counting

    @vuule
    vuule committed Aug 27, 2021
    Configuration menu
    Copy the full SHA
    fd35ba4 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2021

  1. basic logic bit borrowing

    @vuule
    vuule committed Aug 28, 2021
    Configuration menu
    Copy the full SHA
    1728e6c View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2021

  1. rowgroup alignment complete

    @vuule
    vuule committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    69563f6 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2021

  1. small prep clean up

    @vuule
    vuule committed Aug 31, 2021
    Configuration menu
    Copy the full SHA
    ba6b7c4 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2021

  1. further clean up, prep for null mask compacting

    @vuule
    vuule committed Sep 1, 2021
    Configuration menu
    Copy the full SHA
    5826eb9 View commit details
    Browse the repository at this point in the history

  2. encode_nested_null_mask placeholder

    @vuule
    vuule committed Sep 1, 2021
    Configuration menu
    Copy the full SHA
    4671003 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2021

  1. prep clean up in encode_nested_null_mask

    @vuule
    vuule committed Sep 2, 2021
    Configuration menu
    Copy the full SHA
    ef8d850 View commit details
    Browse the repository at this point in the history

  2. more clarity changes

    @vuule
    vuule committed Sep 2, 2021
    Configuration menu
    Copy the full SHA
    ee95438 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2021

  1. pd bits scan

    @vuule
    vuule committed Sep 4, 2021
    Configuration menu
    Copy the full SHA
    958bc01 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2021

  1. compact null masks - complete

    @vuule
    vuule committed Sep 7, 2021
    Configuration menu
    Copy the full SHA
    7807fe3 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2021

  1. fix a few logic errors; at least one more to go

    @vuule
    vuule committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    daca245 View commit details
    Browse the repository at this point in the history

  2. fix rg alignment

    @vuule
    vuule committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    24c7fa4 View commit details
    Browse the repository at this point in the history

  3. OOB fixes

    @vuule
    vuule committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    4112554 View commit details
    Browse the repository at this point in the history

  4. use pushdown masks in val encode

    @vuule
    vuule committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    94d90a3 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2021

  1. account for pushdown masks in decimal column size

    @vuule
    vuule committed Sep 9, 2021
    Configuration menu
    Copy the full SHA
    8465421 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-write-struct
    @vuule
    vuule committed Sep 9, 2021
    Configuration menu
    Copy the full SHA
    6419fc6 View commit details
    Browse the repository at this point in the history

  3. fix cython merge error; YOLO

    @vuule
    vuule committed Sep 9, 2021
    Configuration menu
    Copy the full SHA
    63c6e39 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2021

  1. add missing sync

    @vuule
    vuule committed Sep 10, 2021
    Configuration menu
    Copy the full SHA
    f23874f View commit details
    Browse the repository at this point in the history

  2. fix null access when column has a pushdown mask but no null mask

    @vuule
    vuule committed Sep 10, 2021
    Configuration menu
    Copy the full SHA
    240ae27 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2021

  1. fix value count kernel

    @vuule
    vuule committed Sep 11, 2021
    Configuration menu
    Copy the full SHA
    1f11e50 View commit details
    Browse the repository at this point in the history

  2. reset previously_borrowed when a rowgroup happens to be aligned

    @vuule
    vuule committed Sep 11, 2021
    Configuration menu
    Copy the full SHA
    77275f6 View commit details
    Browse the repository at this point in the history

  3. add py test

    @vuule
    vuule committed Sep 11, 2021
    Configuration menu
    Copy the full SHA
    38f6ab0 View commit details
    Browse the repository at this point in the history

  4. style fix

    @vuule
    vuule committed Sep 11, 2021
    Configuration menu
    Copy the full SHA
    f31172b View commit details
    Browse the repository at this point in the history

  5. rename test

    @vuule
    vuule committed Sep 11, 2021
    Configuration menu
    Copy the full SHA
    0c9b9ad View commit details
    Browse the repository at this point in the history

  6. pre-commit style fix

    @vuule
    vuule committed Sep 11, 2021
    Configuration menu
    Copy the full SHA
    b4bc467 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2021

  1. python style fix and such

    @vuule
    vuule committed Sep 12, 2021
    Configuration menu
    Copy the full SHA
    98d581d View commit details
    Browse the repository at this point in the history

  2. include column offset in null mask access

    @vuule
    vuule committed Sep 12, 2021
    Configuration menu
    Copy the full SHA
    71c5e82 View commit details
    Browse the repository at this point in the history

  3. init orc_column_view members >: (

    @vuule
    vuule committed Sep 12, 2021
    Configuration menu
    Copy the full SHA
    4a04bbd View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2021

  1. pushdown list null masks

    @vuule
    vuule committed Sep 13, 2021
    Configuration menu
    Copy the full SHA
    e3fdc76 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2021

  1. reduce_pushdown_masks docs

    @vuule
    vuule committed Sep 14, 2021
    Configuration menu
    Copy the full SHA
    db5fc6b View commit details
    Browse the repository at this point in the history

  2. move metadata comp util to test utils (un-duplicate code)

    @vuule
    vuule committed Sep 14, 2021
    Configuration menu
    Copy the full SHA
    06db73e View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-write-struct
    @vuule
    vuule committed Sep 14, 2021
    Configuration menu
    Copy the full SHA
    a7220e6 View commit details
    Browse the repository at this point in the history

  4. update yaml

    @vuule
    vuule committed Sep 14, 2021
    Configuration menu
    Copy the full SHA
    0d4dcd9 View commit details
    Browse the repository at this point in the history

  5. newline at EOF

    @vuule
    vuule committed Sep 14, 2021
    Configuration menu
    Copy the full SHA
    a433079 View commit details
    Browse the repository at this point in the history

  6. include structs in more cpp tests

    @vuule
    vuule committed Sep 14, 2021
    Configuration menu
    Copy the full SHA
    cd894cd View commit details
    Browse the repository at this point in the history

  7. kernel clean up

    @vuule
    vuule committed Sep 14, 2021
    Configuration menu
    Copy the full SHA
    c51cf4f View commit details
    Browse the repository at this point in the history

  8. writer clean up pt1

    @vuule
    vuule committed Sep 14, 2021
    Configuration menu
    Copy the full SHA
    a6be5b5 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2021

  1. writer clean up pt2

    @vuule
    vuule committed Sep 15, 2021
    Configuration menu
    Copy the full SHA
    1249913 View commit details
    Browse the repository at this point in the history

  2. Fix Java build after ORC struct write change

    @jlowe @vuule
    jlowe authored and vuule committed Sep 15, 2021
    Configuration menu
    Copy the full SHA
    01d3033 View commit details
    Browse the repository at this point in the history

  3. precision; null mask offset

    @vuule
    vuule committed Sep 15, 2021
    Configuration menu
    Copy the full SHA
    44d7538 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-write-struct
    @vuule
    vuule committed Sep 15, 2021
    Configuration menu
    Copy the full SHA
    db2a8cc View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-write-struct
    @vuule
    vuule committed Sep 15, 2021
    Configuration menu
    Copy the full SHA
    f05a503 View commit details
    Browse the repository at this point in the history

  6. rename bit_is_set_or

    @vuule
    vuule committed Sep 15, 2021
    Configuration menu
    Copy the full SHA
    b9ef40a View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2021

  1. have orc_column_device_view inherit from column_device_view

    @vuule
    vuule committed Sep 16, 2021
    Configuration menu
    Copy the full SHA
    cacef12 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2021

  1. sliced table support

    @vuule
    vuule committed Sep 17, 2021
    Configuration menu
    Copy the full SHA
    b9214c4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-write-struct
    @vuule
    vuule committed Sep 17, 2021
    Configuration menu
    Copy the full SHA
    3623391 View commit details
    Browse the repository at this point in the history

  3. delete submodules (?)

    @vuule
    vuule committed Sep 17, 2021
    Configuration menu
    Copy the full SHA
    be18ba4 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2021

  1. enable multi-rowgroup borrow even without pushdown masks

    @vuule
    vuule committed Sep 18, 2021
    Configuration menu
    Copy the full SHA
    3ddd197 View commit details
    Browse the repository at this point in the history

  2. review changes mostly

    @vuule
    vuule committed Sep 18, 2021
    Configuration menu
    Copy the full SHA
    181f897 View commit details
    Browse the repository at this point in the history

  3. experimental

    @vuule
    vuule committed Sep 18, 2021
    Configuration menu
    Copy the full SHA
    90bdbb5 View commit details
    Browse the repository at this point in the history

  4. list_struct_buff as module fixture

    @vuule
    vuule committed Sep 18, 2021
    Configuration menu
    Copy the full SHA
    f681fdb View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-21.10' of https://github.com/rapidsai/cudf into … 

    …fea-orc-write-struct
    @vuule
    vuule committed Sep 18, 2021
    Configuration menu
    Copy the full SHA
    8c20fb9 View commit details
    Browse the repository at this point in the history