Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

upgrade to arrow 3.0 #136

Closed
zhouyuan opened this issue Mar 1, 2021 · 0 comments · Fixed by #107
Closed

upgrade to arrow 3.0 #136

zhouyuan opened this issue Mar 1, 2021 · 0 comments · Fixed by #107
Labels
enhancement New feature or request performance

Comments

@zhouyuan
Copy link
Collaborator

zhouyuan commented Mar 1, 2021

arrow 0.17.0 is almost 2.5 years old now, we should upgrade to use arrow 3.0

@zhouyuan zhouyuan added the enhancement New feature or request label Mar 1, 2021
zhouyuan added a commit that referenced this issue Mar 4, 2021
rebase with arrow 3.0.0
https://github.com/oap-project/arrow/tree/arrow-3.0.0-oap
notable changes:

arrow compute function context was renamed
new layout on arrow compute kernels
return value changed on function calls(Result)
new API on vector compress/decompress
fixes: #136

Note since commit native sql engine will depends on arrow 3.0.0 instead of 0.17.0
===========
* initial try on rebase

Signed-off-by: Yuan Zhou <[email protected]>

* adding shuffle back

Signed-off-by: Yuan Zhou <[email protected]>

* rebase more kernels

Signed-off-by: Yuan Zhou <[email protected]>

* rebase compute related API

Signed-off-by: Yuan Zhou <[email protected]>

* rebase splitter.cc

* adding shuffle rebase

* remove parquet reader/writer

Signed-off-by: Yuan Zhou <[email protected]>

* fix unit tests

Signed-off-by: Yuan Zhou <[email protected]>

* rebase splitter.cc with latest native sql engine code

* enable shuffle tests

Signed-off-by: Yuan Zhou <[email protected]>

* rebase scala code

* try to rebase scala

Signed-off-by: Yuan Zhou <[email protected]>

* remove loadNextBatch method in ArrowCompressedStreamReader

* fix sort kernel

Signed-off-by: Yuan Zhou <[email protected]>

* Revert "remove loadNextBatch method in ArrowCompressedStreamReader"

This reverts commit 1f294a8.

* rebase the fastpfor codec and fix bug

* fix CI system

Signed-off-by: Yuan Zhou <[email protected]>

* fix

Signed-off-by: Yuan Zhou <[email protected]>

* Add the comment in the changes when calling the arrow API

* fix ut

Signed-off-by: Yuan Zhou <[email protected]>

* fix window sort

Signed-off-by: Yuan Zhou <[email protected]>

Co-authored-by: Ke Jia <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants