Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: replace arrow2 with official implementation 🎉 #753

Merged
merged 45 commits into from
Dec 15, 2022
Merged

feat: replace arrow2 with official implementation 🎉 #753

merged 45 commits into from
Dec 15, 2022

Conversation

v0y4g3r
Copy link
Contributor

@v0y4g3r v0y4g3r commented Dec 15, 2022 •
edited
Loading

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Finally! We are able to migrate arrow/parquet to offical implementation 🎉🥳🍾

  • Replace arrow2/parquet2 with official implementation and adapt datatypes and vectors.
  • Upgrade arrow-datafusion to 14.x

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

#555

waynexia and others added 30 commits November 21, 2022 14:19
@waynexia
chore: kick off. change datafusion/arrow/parquet to target version
571a84d 
Signed-off-by: Ruihang Xia <[email protected]>
@waynexia
chore: replace one last datafusion dep
e3201a4 
Signed-off-by: Ruihang Xia <[email protected]>
@evenyag
feat: arrow_array switch to arrow
d7626fd
@evenyag
chore: update dep of binary vector
61c4a36
@waynexia
Merge branch 'dev' into replace-arrow2
221f3e9
@waynexia
Merge branch 'replace-arrow2' of github.com:GreptimeTeam/greptimedb i…
d0686f9 
…nto replace-arrow2
@waynexia
Merge branch 'dev' into replace-arrow2
7151deb
@waynexia
chore: fix wrong merge commit
504059a 
Signed-off-by: Ruihang Xia <[email protected]>
@evenyag
feat: Switch to datatypes2
cc1ec26
@evenyag
feat: Make recordbatch compile
fe505fe
@evenyag
chore: sort Cargo.toml
99371fd
@evenyag
feat: Fix common::recordbatch compiler errors
8c66b7d
@evenyag
feat: Fix recordbatch test compiling issue
3c0adb0
@v0y4g3r
fix: api crate (#708)
b48ae21 
* fix: rename ConcreteDataType::timestamp_millis_type to ConcreteDataType::timestamp_millisecond_type. fix other warnings regarding timestamp

* fix: revert changes in datatypes2

* fix: helper
@v0y4g3r
chore: delete datatypes based on arrow2
0ccb8b4
@evenyag
feat: Fix some compiler errors in common::query (#710)
b32438e 
* feat: Fix some compiler errors in common::query

* feat: test_collect use vectors api
@waynexia
fix: common-query subcrate (#712)
829ff49 
* fix: record batch adapter

Signed-off-by: Ruihang Xia <[email protected]>

* fix error enum

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
@evenyag
fix: Fix common::query compiler errors (#713)
653906d 
* feat: Move conversion to ScalarValue to value.rs

* fix: Fix common::query compiler errors

This commit also make InnerError pub(crate)
@waynexia
Merge branch 'dev' into replace-arrow2
551cde2
@evenyag
feat: Implements diff accumulator using WrapperType (#715)
2ba9925 
* feat: Remove usage of opaque error from common::recordbatch

* feat: Remove opaque error from common::query

* feat: Fix diff compiler errors

Now common_function just use common_query's Error and Result. Adds
a LargestType associated type to LogicalPrimitiveType to get the largest
type a logical primitive type can cast to.

* feat: Remove LargestType from NativeType trait

* chore: Update comments

* feat: Restrict Scalar::RefType of WrapperType to itself

Add trait bound `for<'a> Scalar<RefType<'a> = Self>` to WrapperType

* chore: Address CR comments

* chore: Format codes
@waynexia
fix: fix compile error for mean/polyval/pow/interp ops
fb0b4eb 
Signed-off-by: Ruihang Xia <[email protected]>
@waynexia
Revert "fix: fix compile error for mean/polyval/pow/interp ops"
a562199 
This reverts commit fb0b4eb.
@evenyag
fix: Fix compiler errors in argmax/rate/median/norm_cdf (#716)
a898f84 
* fix: Fix compiler errors in argmax/rate/median/norm_cdf

* chore: Address CR comments
@waynexia
fix: fix compile error for mean/polyval/pow/interp ops (#717)
6f3baf9 
* fix: fix compile error for mean/polyval/pow/interp ops

Signed-off-by: Ruihang Xia <[email protected]>

* simplify type bounds

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
@evenyag
fix: fix argmin/percentile/clip/interp/scipy_stats_norm_pdf errors (#718
58c26de 
)

fix: fix argmin/percentile/clip/interp/scipy_stats_norm_pdf compiler errors
@waynexia
fix: fix other compile error in common-function (#719)
587bdc9 
* further fixing

Signed-off-by: Ruihang Xia <[email protected]>

* fix all compile errors in common function

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
@waynexia
fix: Fix tests and clippy for common-function subcrate (#726)
3687bc7 
* further fixing

Signed-off-by: Ruihang Xia <[email protected]>

* fix all compile errors in common function

Signed-off-by: Ruihang Xia <[email protected]>

* fix tests

Signed-off-by: Ruihang Xia <[email protected]>

* fix clippy

Signed-off-by: Ruihang Xia <[email protected]>

* revert test changes

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
@v0y4g3r
fix: row group pruning (#725)
1bde1ba 
* fix: row group pruning

* chore: use macro to simplify stats implemetation

* fxi: CR comments

* fix: row group metadata length mismatch

* fix: simplify code
@evenyag
fix: Fix common::grpc compiler errors (#722)
b936d8b 
* fix: Fix common::grpc compiler errors

This commit refactors RecordBatch and holds vectors in the RecordBatch
struct, so we don't need to cast the array to vector when doing
serialization or iterating the batch.

Now we use the vector API instead of the arrow API in grpc crate.

* chore: Address CR comments
@waynexia
fix common record batch
fff530c 
Signed-off-by: Ruihang Xia <[email protected]>
waynexia and others added 14 commits December 8, 2022 20:27
@waynexia @evenyag
fix: Fix compile error in server subcrate (#727)
d0892bf 
* fix: Fix compile error in server subcrate

Signed-off-by: Ruihang Xia <[email protected]>

* remove unused type alias

Signed-off-by: Ruihang Xia <[email protected]>

* explicitly panic

Signed-off-by: Ruihang Xia <[email protected]>

* Update src/storage/src/sst/parquet.rs

Co-authored-by: Yingwen <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
Co-authored-by: Yingwen <[email protected]>
@waynexia
fix: Fix common grpc expr (#730)
42fdc72 
* fix compile errors

Signed-off-by: Ruihang Xia <[email protected]>

* rename fn names

Signed-off-by: Ruihang Xia <[email protected]>

* fix styles

Signed-off-by: Ruihang Xia <[email protected]>

* fix wranings in common-time

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
@waynexia
fix: pre-cast to avoid tremendous match arms (#734)
95b2d86 
Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
@v0y4g3r
feat: upgrade storage crate to arrow and parquet offcial impl (#738)
4defde0 
* fix: compile erros

* fix: parquet reader and writer

* fix: parquet reader and writer

* fix: WriteBatch IPC encode/decode

* fix: clippy errors in storage subcrate

* chore: remove suspicious unwrap

* fix: some cr comments

* fix: CR comments

* fix: CR comments
@evenyag
fix: Fix compiler errors in catalog and mito crates (#742)
4b644aa 
* fix: Fix compiler errors in mito

* fix: Fix compiler errors in catalog crate

* style: Fix clippy

* chore: Fix use
@waynexia
Merge pull request #745
a712382 
* fix nyc-taxi and util

* Merge branch 'replace-arrow2' into fix-others

* fix substrait

* fix warnings and error in test
@evenyag
fix: Fix imports in optimizer.rs
36c929e
@v0y4g3r
fix: errors in optimzer
fa971c6
@v0y4g3r
fix: remove unwrap
652d59a
@evenyag @waynexia
fix: Fix compiler errors in query crate (#746)
dbb3034 
* fix: Fix compiler errors in state.rs

* fix: fix compiler errors in state

* feat: upgrade sqlparser to 0.26

* fix: fix datafusion engine compiler errors

* fix: Fix some tests in query crate

* fix: Fix all warnings in tests

* feat: Remove `Type` from timestamp's type name

* fix: fix query tests

Now datafusion already supports median, so this commit also remove the
median function

* style: Fix clippy

* feat: Remove RecordBatch::pretty_print

* chore: Address CR comments

* Update src/query/src/query_engine/state.rs

Co-authored-by: Ruihang Xia <[email protected]>
@v0y4g3r
fix: frontend compile errors (#747)
ce6d1cb 
fix: fix compile errors in frontend
@evenyag @waynexia
fix: Fix compiler errors in script crate (#749)
142dee4 
* fix: Fix compiler errors in state.rs

* fix: fix compiler errors in state

* feat: upgrade sqlparser to 0.26

* fix: fix datafusion engine compiler errors

* fix: Fix some tests in query crate

* fix: Fix all warnings in tests

* feat: Remove `Type` from timestamp's type name

* fix: fix query tests

Now datafusion already supports median, so this commit also remove the
median function

* style: Fix clippy

* feat: Remove RecordBatch::pretty_print

* chore: Address CR comments

* feat: Add column_by_name to RecordBatch

* feat: modify select_from_rb

* feat: Fix some compiler errors in vector.rs

* feat: Fix more compiler errors in vector.rs

* fix: fix table.rs

Signed-off-by: Ruihang Xia <[email protected]>

* fix: Fix compiler errors in coprocessor

* fix: Fix some compiler errors

* fix: Fix compiler errors in script

* chore: Remove unused imports and format code

* test: disable interval tests

* test: Fix test_compile_execute test

* style: Fix clippy

* feat: Support interval

* feat: Add RecordBatch::columns and fix clippy

Signed-off-by: Ruihang Xia <[email protected]>
Co-authored-by: Ruihang Xia <[email protected]>
@evenyag
Merge branch 'develop' into replace-arrow2
7c696da
@waynexia @v0y4g3r @evenyag
fix: Fix All The Tests! (#752)
0f3dcc1 
* fix: Fix several tests compile errors

Signed-off-by: Ruihang Xia <[email protected]>

* fix: some compile errors in tests

Signed-off-by: Ruihang Xia <[email protected]>

* fix: compile errors in frontend tests

* fix: compile errors in frontend tests

* test: Fix tests in api and common-query

* test: Fix test in sql crate

* fix: resolve substrait error

Signed-off-by: Ruihang Xia <[email protected]>

* chore: add more test

* test: Fix tests in servers

* fix instance_test

Signed-off-by: Ruihang Xia <[email protected]>

* test: Fix tests in tests-integration

Signed-off-by: Ruihang Xia <[email protected]>
Co-authored-by: Lei, HUANG <[email protected]>
Co-authored-by: evenyag <[email protected]>
@v0y4g3r v0y4g3r requested review from killme2008 and evenyag and removed request for killme2008 December 15, 2022 09:51
@v0y4g3r v0y4g3r changed the title Replace arrow2 feat: replace arrow2 with official implementation Dec 15, 2022
@v0y4g3r v0y4g3r changed the title feat: replace arrow2 with official implementation feat: replace arrow2 with official implementation 🎉 Dec 15, 2022
@codecov
Copy link

codecov bot commented Dec 15, 2022 •
edited
Loading

Codecov Report

Merging #753 (a8630cd) into develop (61d8bc2) will decrease coverage by 0.27%.
The diff coverage is 88.97%.

@@             Coverage Diff             @@
##           develop     #753      +/-   ##
===========================================
- Coverage    86.30%   86.03%   -0.28%     
===========================================
  Files          454      412      -42     
  Lines        59855    53761    -6094     
===========================================
- Hits         51659    46254    -5405     
+ Misses        8196     7507     -689
Flag Coverage Δ
rust 86.03% <88.97%> (-0.28%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/common/function/src/lib.rs 100.00% <ø> (ø)
src/common/function/src/scalars/aggregate.rs 97.29% <ø> (ø)
src/common/function/src/scalars/expression/ctx.rs 58.33% <ø> (ø)
...rc/common/function/src/scalars/expression/unary.rs 0.00% <0.00%> (ø)
src/common/function/src/scalars/function.rs 83.33% <ø> (ø)
src/common/function/src/scalars/math.rs 100.00% <ø> (ø)
src/common/function/src/scalars/math/pow.rs 91.48% <ø> (ø)
src/common/function/src/scalars/numpy.rs 100.00% <ø> (ø)
src/common/function/src/scalars/test.rs 87.50% <ø> (ø)
src/common/function/src/scalars/timestamp.rs 100.00% <ø> (ø)
... and 257 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

evenyag
evenyag approved these changes Dec 15, 2022
View reviewed changes
Copy link
Contributor

@evenyag evenyag left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel good to me 🥰

waynexia
waynexia approved these changes Dec 15, 2022
View reviewed changes
Copy link
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It gives me a good night 🥰

@waynexia waynexia merged commit 0653301 into develop Dec 15, 2022
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
@v0y4g3r @waynexia
@evenyag @paomian
feat: replace arrow2 with official implementation 🎉 (GreptimeTeam#753)
cfffa68 
* chore: kick off. change datafusion/arrow/parquet to target version

Signed-off-by: Ruihang Xia <[email protected]>

* chore: replace one last datafusion dep

Signed-off-by: Ruihang Xia <[email protected]>

* feat: arrow_array switch to arrow

* chore: update dep of binary vector

* chore: fix wrong merge commit

Signed-off-by: Ruihang Xia <[email protected]>

* feat: Switch to datatypes2

* feat: Make recordbatch compile

* chore: sort Cargo.toml

* feat: Fix common::recordbatch compiler errors

* feat: Fix recordbatch test compiling issue

* fix: api crate (GreptimeTeam#708)

* fix: rename ConcreteDataType::timestamp_millis_type to ConcreteDataType::timestamp_millisecond_type. fix other warnings regarding timestamp

* fix: revert changes in datatypes2

* fix: helper

* chore: delete datatypes based on arrow2

* feat: Fix some compiler errors in common::query (GreptimeTeam#710)

* feat: Fix some compiler errors in common::query

* feat: test_collect use vectors api

* fix: common-query subcrate (GreptimeTeam#712)

* fix: record batch adapter

Signed-off-by: Ruihang Xia <[email protected]>

* fix error enum

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>

* fix: Fix common::query compiler errors (GreptimeTeam#713)

* feat: Move conversion to ScalarValue to value.rs

* fix: Fix common::query compiler errors

This commit also make InnerError pub(crate)

* feat: Implements diff accumulator using WrapperType (GreptimeTeam#715)

* feat: Remove usage of opaque error from common::recordbatch

* feat: Remove opaque error from common::query

* feat: Fix diff compiler errors

Now common_function just use common_query's Error and Result. Adds
a LargestType associated type to LogicalPrimitiveType to get the largest
type a logical primitive type can cast to.

* feat: Remove LargestType from NativeType trait

* chore: Update comments

* feat: Restrict Scalar::RefType of WrapperType to itself

Add trait bound `for<'a> Scalar<RefType<'a> = Self>` to WrapperType

* chore: Address CR comments

* chore: Format codes

* fix: fix compile error for mean/polyval/pow/interp ops

Signed-off-by: Ruihang Xia <[email protected]>

* Revert "fix: fix compile error for mean/polyval/pow/interp ops"

This reverts commit fb0b4eb.

* fix: Fix compiler errors in argmax/rate/median/norm_cdf (GreptimeTeam#716)

* fix: Fix compiler errors in argmax/rate/median/norm_cdf

* chore: Address CR comments

* fix: fix compile error for mean/polyval/pow/interp ops (GreptimeTeam#717)

* fix: fix compile error for mean/polyval/pow/interp ops

Signed-off-by: Ruihang Xia <[email protected]>

* simplify type bounds

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>

* fix: fix argmin/percentile/clip/interp/scipy_stats_norm_pdf errors (GreptimeTeam#718)

fix: fix argmin/percentile/clip/interp/scipy_stats_norm_pdf compiler errors

* fix: fix other compile error in common-function (GreptimeTeam#719)

* further fixing

Signed-off-by: Ruihang Xia <[email protected]>

* fix all compile errors in common function

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>

* fix: Fix tests and clippy for common-function subcrate (GreptimeTeam#726)

* further fixing

Signed-off-by: Ruihang Xia <[email protected]>

* fix all compile errors in common function

Signed-off-by: Ruihang Xia <[email protected]>

* fix tests

Signed-off-by: Ruihang Xia <[email protected]>

* fix clippy

Signed-off-by: Ruihang Xia <[email protected]>

* revert test changes

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>

* fix: row group pruning (GreptimeTeam#725)

* fix: row group pruning

* chore: use macro to simplify stats implemetation

* fxi: CR comments

* fix: row group metadata length mismatch

* fix: simplify code

* fix: Fix common::grpc compiler errors (GreptimeTeam#722)

* fix: Fix common::grpc compiler errors

This commit refactors RecordBatch and holds vectors in the RecordBatch
struct, so we don't need to cast the array to vector when doing
serialization or iterating the batch.

Now we use the vector API instead of the arrow API in grpc crate.

* chore: Address CR comments

* fix common record batch

Signed-off-by: Ruihang Xia <[email protected]>

* fix: Fix compile error in server subcrate (GreptimeTeam#727)

* fix: Fix compile error in server subcrate

Signed-off-by: Ruihang Xia <[email protected]>

* remove unused type alias

Signed-off-by: Ruihang Xia <[email protected]>

* explicitly panic

Signed-off-by: Ruihang Xia <[email protected]>

* Update src/storage/src/sst/parquet.rs

Co-authored-by: Yingwen <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
Co-authored-by: Yingwen <[email protected]>

* fix: Fix common grpc expr (GreptimeTeam#730)

* fix compile errors

Signed-off-by: Ruihang Xia <[email protected]>

* rename fn names

Signed-off-by: Ruihang Xia <[email protected]>

* fix styles

Signed-off-by: Ruihang Xia <[email protected]>

* fix wranings in common-time

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>

* fix: pre-cast to avoid tremendous match arms (GreptimeTeam#734)

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>

* feat: upgrade storage crate to arrow and parquet offcial impl (GreptimeTeam#738)

* fix: compile erros

* fix: parquet reader and writer

* fix: parquet reader and writer

* fix: WriteBatch IPC encode/decode

* fix: clippy errors in storage subcrate

* chore: remove suspicious unwrap

* fix: some cr comments

* fix: CR comments

* fix: CR comments

* fix: Fix compiler errors in catalog and mito crates (GreptimeTeam#742)

* fix: Fix compiler errors in mito

* fix: Fix compiler errors in catalog crate

* style: Fix clippy

* chore: Fix use

* Merge pull request GreptimeTeam#745

* fix nyc-taxi and util

* Merge branch 'replace-arrow2' into fix-others

* fix substrait

* fix warnings and error in test

* fix: Fix imports in optimizer.rs

* fix: errors in optimzer

* fix: remove unwrap

* fix: Fix compiler errors in query crate (GreptimeTeam#746)

* fix: Fix compiler errors in state.rs

* fix: fix compiler errors in state

* feat: upgrade sqlparser to 0.26

* fix: fix datafusion engine compiler errors

* fix: Fix some tests in query crate

* fix: Fix all warnings in tests

* feat: Remove `Type` from timestamp's type name

* fix: fix query tests

Now datafusion already supports median, so this commit also remove the
median function

* style: Fix clippy

* feat: Remove RecordBatch::pretty_print

* chore: Address CR comments

* Update src/query/src/query_engine/state.rs

Co-authored-by: Ruihang Xia <[email protected]>

* fix: frontend compile errors (GreptimeTeam#747)

fix: fix compile errors in frontend

* fix: Fix compiler errors in script crate (GreptimeTeam#749)

* fix: Fix compiler errors in state.rs

* fix: fix compiler errors in state

* feat: upgrade sqlparser to 0.26

* fix: fix datafusion engine compiler errors

* fix: Fix some tests in query crate

* fix: Fix all warnings in tests

* feat: Remove `Type` from timestamp's type name

* fix: fix query tests

Now datafusion already supports median, so this commit also remove the
median function

* style: Fix clippy

* feat: Remove RecordBatch::pretty_print

* chore: Address CR comments

* feat: Add column_by_name to RecordBatch

* feat: modify select_from_rb

* feat: Fix some compiler errors in vector.rs

* feat: Fix more compiler errors in vector.rs

* fix: fix table.rs

Signed-off-by: Ruihang Xia <[email protected]>

* fix: Fix compiler errors in coprocessor

* fix: Fix some compiler errors

* fix: Fix compiler errors in script

* chore: Remove unused imports and format code

* test: disable interval tests

* test: Fix test_compile_execute test

* style: Fix clippy

* feat: Support interval

* feat: Add RecordBatch::columns and fix clippy

Signed-off-by: Ruihang Xia <[email protected]>
Co-authored-by: Ruihang Xia <[email protected]>

* fix: Fix All The Tests! (GreptimeTeam#752)

* fix: Fix several tests compile errors

Signed-off-by: Ruihang Xia <[email protected]>

* fix: some compile errors in tests

Signed-off-by: Ruihang Xia <[email protected]>

* fix: compile errors in frontend tests

* fix: compile errors in frontend tests

* test: Fix tests in api and common-query

* test: Fix test in sql crate

* fix: resolve substrait error

Signed-off-by: Ruihang Xia <[email protected]>

* chore: add more test

* test: Fix tests in servers

* fix instance_test

Signed-off-by: Ruihang Xia <[email protected]>

* test: Fix tests in tests-integration

Signed-off-by: Ruihang Xia <[email protected]>
Co-authored-by: Lei, HUANG <[email protected]>
Co-authored-by: evenyag <[email protected]>

* fix: clippy errors

Signed-off-by: Ruihang Xia <[email protected]>
Co-authored-by: Ruihang Xia <[email protected]>
Co-authored-by: evenyag <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@waynexia waynexia waynexia approved these changes

@evenyag evenyag evenyag approved these changes

@killme2008 killme2008 Awaiting requested review from killme2008

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@v0y4g3r @evenyag @waynexia