Test ORC predicate pushdown (PPD) with timestamps decimals booleans #9068
Conversation
Signed-off-by: Haoyang Li <[email protected]>
|
This is a really scary failure. I did some debugging and it looks like we are not writing out the proper metrics for predicate push down to work. If I switch the write to the CPU it works. Could we extend the tests so that we do
|
|
I filed rapidsai/cudf#13899 to fix the timestamp issue. I think for now we either mark it as xfail or comment it out and point to this issue. I dug into it because I was scared that we could have data corruption, but this is just a performance issue, so we are okay. |
Signed-off-by: Haoyang Li <[email protected]>
|
Thanks @revans2 ! I verified that the tests only fail on files written by GPU and comment out failed cases. |
Signed-off-by: Haoyang Li <[email protected]>
|
build |
There was a problem hiding this comment.
Can we have a follow on issue to enable the tests? and also one to update all of the tests to do what I described in #9068 (comment)
Signed-off-by: Haoyang Li <[email protected]>
|
Filed follow on issue #9075 to enable the failed cases and updated other cases to also test CPU write GPU read and GPU write CPU read. |
Signed-off-by: Haoyang Li <[email protected]>
|
Decimal also has a similar issue, and decimal tests also fail on files written by the GPU. I filed rapidsai/cudf#13933 in cuDF and updated the decimal tests and issue #9075 . |
Signed-off-by: Haoyang Li <[email protected]>
|
build |
…VIDIA#9068) * Test predicate pushdown (PPD) with timestamps decimals booleans Signed-off-by: Haoyang Li <[email protected]> * extend the timestamp test Signed-off-by: Haoyang Li <[email protected]> * style fix Signed-off-by: Haoyang Li <[email protected]> * update other cases Signed-off-by: Haoyang Li <[email protected]> * style fix Signed-off-by: Haoyang Li <[email protected]> * comment out two decimal tests Signed-off-by: Haoyang Li <[email protected]> --------- Signed-off-by: Haoyang Li <[email protected]>
Closes #8823
Corresponding tests in Spark
Note: The test "Support for pushing down filters for timestamp types" will fail now. I think plugin is not support orc PPD with timestamps, don't know if it is a known issue. Will do some investigation on it.