Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: replace arrow2 with official implementation 🎉 #753

Merged
merged 45 commits into from
Dec 15, 2022
Merged

feat: replace arrow2 with official implementation 🎉 #753

merged 45 commits into from
Dec 15, 2022

Commits on Nov 21, 2022

  1. chore: kick off. change datafusion/arrow/parquet to target version 

    Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    571a84d View commit details
    Browse the repository at this point in the history

  2. chore: replace one last datafusion dep 

    Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    e3201a4 View commit details
    Browse the repository at this point in the history

  3. feat: arrow_array switch to arrow

    @evenyag
    evenyag committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    d7626fd View commit details
    Browse the repository at this point in the history

  4. chore: update dep of binary vector

    @evenyag
    evenyag committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    61c4a36 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'dev' into replace-arrow2

    @waynexia
    waynexia committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    221f3e9 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'replace-arrow2' of github.com:GreptimeTeam/greptimedb i… 

    …nto replace-arrow2
    @waynexia
    waynexia committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    d0686f9 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2022

  1. Merge branch 'dev' into replace-arrow2

    @waynexia
    waynexia committed Dec 5, 2022
    Configuration menu
    Copy the full SHA
    7151deb View commit details
    Browse the repository at this point in the history

  2. chore: fix wrong merge commit 

    Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia committed Dec 5, 2022
    Configuration menu
    Copy the full SHA
    504059a View commit details
    Browse the repository at this point in the history

  3. feat: Switch to datatypes2

    @evenyag
    evenyag committed Dec 5, 2022
    Configuration menu
    Copy the full SHA
    cc1ec26 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2022

  1. feat: Make recordbatch compile

    @evenyag
    evenyag committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    fe505fe View commit details
    Browse the repository at this point in the history

  2. chore: sort Cargo.toml

    @evenyag
    evenyag committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    99371fd View commit details
    Browse the repository at this point in the history

  3. feat: Fix common::recordbatch compiler errors

    @evenyag
    evenyag committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    8c66b7d View commit details
    Browse the repository at this point in the history

  4. feat: Fix recordbatch test compiling issue

    @evenyag
    evenyag committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    3c0adb0 View commit details
    Browse the repository at this point in the history

  5. fix: api crate (#708) 

    * fix: rename ConcreteDataType::timestamp_millis_type to ConcreteDataType::timestamp_millisecond_type. fix other warnings regarding timestamp

* fix: revert changes in datatypes2

* fix: helper
    @v0y4g3r
    v0y4g3r authored Dec 6, 2022
    Configuration menu
    Copy the full SHA
    b48ae21 View commit details
    Browse the repository at this point in the history

  6. chore: delete datatypes based on arrow2

    @v0y4g3r
    v0y4g3r committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    0ccb8b4 View commit details
    Browse the repository at this point in the history

  7. feat: Fix some compiler errors in common::query (#710) 

    * feat: Fix some compiler errors in common::query

* feat: test_collect use vectors api
    @evenyag
    evenyag authored Dec 6, 2022
    Configuration menu
    Copy the full SHA
    b32438e View commit details
    Browse the repository at this point in the history

  8. fix: common-query subcrate (#712) 

    * fix: record batch adapter

Signed-off-by: Ruihang Xia <[email protected]>

* fix error enum

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia authored Dec 6, 2022
    Configuration menu
    Copy the full SHA
    829ff49 View commit details
    Browse the repository at this point in the history

  9. fix: Fix common::query compiler errors (#713) 

    * feat: Move conversion to ScalarValue to value.rs

* fix: Fix common::query compiler errors

This commit also make InnerError pub(crate)
    @evenyag
    evenyag authored Dec 6, 2022
    Configuration menu
    Copy the full SHA
    653906d View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2022

  1. Merge branch 'dev' into replace-arrow2

    @waynexia
    waynexia committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    551cde2 View commit details
    Browse the repository at this point in the history

  2. feat: Implements diff accumulator using WrapperType (#715) 

    * feat: Remove usage of opaque error from common::recordbatch

* feat: Remove opaque error from common::query

* feat: Fix diff compiler errors

Now common_function just use common_query's Error and Result. Adds
a LargestType associated type to LogicalPrimitiveType to get the largest
type a logical primitive type can cast to.

* feat: Remove LargestType from NativeType trait

* chore: Update comments

* feat: Restrict Scalar::RefType of WrapperType to itself

Add trait bound `for<'a> Scalar<RefType<'a> = Self>` to WrapperType

* chore: Address CR comments

* chore: Format codes
    @evenyag
    evenyag authored Dec 7, 2022
    Configuration menu
    Copy the full SHA
    2ba9925 View commit details
    Browse the repository at this point in the history

  3. fix: fix compile error for mean/polyval/pow/interp ops 

    Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    fb0b4eb View commit details
    Browse the repository at this point in the history

  4. Revert "fix: fix compile error for mean/polyval/pow/interp ops" 

    This reverts commit fb0b4eb.
    @waynexia
    waynexia committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    a562199 View commit details
    Browse the repository at this point in the history

  5. fix: Fix compiler errors in argmax/rate/median/norm_cdf (#716) 

    * fix: Fix compiler errors in argmax/rate/median/norm_cdf

* chore: Address CR comments
    @evenyag
    evenyag authored Dec 7, 2022
    Configuration menu
    Copy the full SHA
    a898f84 View commit details
    Browse the repository at this point in the history

  6. fix: fix compile error for mean/polyval/pow/interp ops (#717) 

    * fix: fix compile error for mean/polyval/pow/interp ops

Signed-off-by: Ruihang Xia <[email protected]>

* simplify type bounds

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia authored Dec 7, 2022
    Configuration menu
    Copy the full SHA
    6f3baf9 View commit details
    Browse the repository at this point in the history

  7. fix: fix argmin/percentile/clip/interp/scipy_stats_norm_pdf errors (#718 

    )

fix: fix argmin/percentile/clip/interp/scipy_stats_norm_pdf compiler errors
    @evenyag
    evenyag authored Dec 7, 2022
    Configuration menu
    Copy the full SHA
    58c26de View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2022

  1. fix: fix other compile error in common-function (#719) 

    * further fixing

Signed-off-by: Ruihang Xia <[email protected]>

* fix all compile errors in common function

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia authored Dec 8, 2022
    Configuration menu
    Copy the full SHA
    587bdc9 View commit details
    Browse the repository at this point in the history

  2. fix: Fix tests and clippy for common-function subcrate (#726) 

    * further fixing

Signed-off-by: Ruihang Xia <[email protected]>

* fix all compile errors in common function

Signed-off-by: Ruihang Xia <[email protected]>

* fix tests

Signed-off-by: Ruihang Xia <[email protected]>

* fix clippy

Signed-off-by: Ruihang Xia <[email protected]>

* revert test changes

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia authored Dec 8, 2022
    Configuration menu
    Copy the full SHA
    3687bc7 View commit details
    Browse the repository at this point in the history

  3. fix: row group pruning (#725) 

    * fix: row group pruning

* chore: use macro to simplify stats implemetation

* fxi: CR comments

* fix: row group metadata length mismatch

* fix: simplify code
    @v0y4g3r
    v0y4g3r authored Dec 8, 2022
    Configuration menu
    Copy the full SHA
    1bde1ba View commit details
    Browse the repository at this point in the history

  4. fix: Fix common::grpc compiler errors (#722) 

    * fix: Fix common::grpc compiler errors

This commit refactors RecordBatch and holds vectors in the RecordBatch
struct, so we don't need to cast the array to vector when doing
serialization or iterating the batch.

Now we use the vector API instead of the arrow API in grpc crate.

* chore: Address CR comments
    @evenyag
    evenyag authored Dec 8, 2022
    Configuration menu
    Copy the full SHA
    b936d8b View commit details
    Browse the repository at this point in the history

  5. fix common record batch 

    Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia committed Dec 8, 2022
    Configuration menu
    Copy the full SHA
    fff530c View commit details
    Browse the repository at this point in the history

  6. fix: Fix compile error in server subcrate (#727) 

    * fix: Fix compile error in server subcrate

Signed-off-by: Ruihang Xia <[email protected]>

* remove unused type alias

Signed-off-by: Ruihang Xia <[email protected]>

* explicitly panic

Signed-off-by: Ruihang Xia <[email protected]>

* Update src/storage/src/sst/parquet.rs

Co-authored-by: Yingwen <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
Co-authored-by: Yingwen <[email protected]>
    @waynexia @evenyag
    waynexia and evenyag authored Dec 8, 2022
    Configuration menu
    Copy the full SHA
    d0892bf View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2022

  1. fix: Fix common grpc expr (#730) 

    * fix compile errors

Signed-off-by: Ruihang Xia <[email protected]>

* rename fn names

Signed-off-by: Ruihang Xia <[email protected]>

* fix styles

Signed-off-by: Ruihang Xia <[email protected]>

* fix wranings in common-time

Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia authored Dec 9, 2022
    Configuration menu
    Copy the full SHA
    42fdc72 View commit details
    Browse the repository at this point in the history

  2. fix: pre-cast to avoid tremendous match arms (#734) 

    Signed-off-by: Ruihang Xia <[email protected]>

Signed-off-by: Ruihang Xia <[email protected]>
    @waynexia
    waynexia authored Dec 9, 2022
    Configuration menu
    Copy the full SHA
    95b2d86 View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2022

  1. feat: upgrade storage crate to arrow and parquet offcial impl (#738) 

    * fix: compile erros

* fix: parquet reader and writer

* fix: parquet reader and writer

* fix: WriteBatch IPC encode/decode

* fix: clippy errors in storage subcrate

* chore: remove suspicious unwrap

* fix: some cr comments

* fix: CR comments

* fix: CR comments
    @v0y4g3r
    v0y4g3r authored Dec 13, 2022
    Configuration menu
    Copy the full SHA
    4defde0 View commit details
    Browse the repository at this point in the history

  2. fix: Fix compiler errors in catalog and mito crates (#742) 

    * fix: Fix compiler errors in mito

* fix: Fix compiler errors in catalog crate

* style: Fix clippy

* chore: Fix use
    @evenyag
    evenyag authored Dec 13, 2022
    Configuration menu
    Copy the full SHA
    4b644aa View commit details
    Browse the repository at this point in the history

  3. Merge pull request #745 

    * fix nyc-taxi and util

* Merge branch 'replace-arrow2' into fix-others

* fix substrait

* fix warnings and error in test
    @waynexia
    waynexia authored Dec 13, 2022
    Configuration menu
    Copy the full SHA
    a712382 View commit details
    Browse the repository at this point in the history

  4. fix: Fix imports in optimizer.rs

    @evenyag
    evenyag committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    36c929e View commit details
    Browse the repository at this point in the history

  5. fix: errors in optimzer

    @v0y4g3r
    v0y4g3r committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    fa971c6 View commit details
    Browse the repository at this point in the history

  6. fix: remove unwrap

    @v0y4g3r
    v0y4g3r committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    652d59a View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2022

  1. fix: Fix compiler errors in query crate (#746) 

    * fix: Fix compiler errors in state.rs

* fix: fix compiler errors in state

* feat: upgrade sqlparser to 0.26

* fix: fix datafusion engine compiler errors

* fix: Fix some tests in query crate

* fix: Fix all warnings in tests

* feat: Remove `Type` from timestamp's type name

* fix: fix query tests

Now datafusion already supports median, so this commit also remove the
median function

* style: Fix clippy

* feat: Remove RecordBatch::pretty_print

* chore: Address CR comments

* Update src/query/src/query_engine/state.rs

Co-authored-by: Ruihang Xia <[email protected]>
    @evenyag @waynexia
    evenyag and waynexia authored Dec 14, 2022
    Configuration menu
    Copy the full SHA
    dbb3034 View commit details
    Browse the repository at this point in the history

  2. fix: frontend compile errors (#747) 

    fix: fix compile errors in frontend
    @v0y4g3r
    v0y4g3r authored Dec 14, 2022
    Configuration menu
    Copy the full SHA
    ce6d1cb View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2022

  1. fix: Fix compiler errors in script crate (#749) 

    * fix: Fix compiler errors in state.rs

* fix: fix compiler errors in state

* feat: upgrade sqlparser to 0.26

* fix: fix datafusion engine compiler errors

* fix: Fix some tests in query crate

* fix: Fix all warnings in tests

* feat: Remove `Type` from timestamp's type name

* fix: fix query tests

Now datafusion already supports median, so this commit also remove the
median function

* style: Fix clippy

* feat: Remove RecordBatch::pretty_print

* chore: Address CR comments

* feat: Add column_by_name to RecordBatch

* feat: modify select_from_rb

* feat: Fix some compiler errors in vector.rs

* feat: Fix more compiler errors in vector.rs

* fix: fix table.rs

Signed-off-by: Ruihang Xia <[email protected]>

* fix: Fix compiler errors in coprocessor

* fix: Fix some compiler errors

* fix: Fix compiler errors in script

* chore: Remove unused imports and format code

* test: disable interval tests

* test: Fix test_compile_execute test

* style: Fix clippy

* feat: Support interval

* feat: Add RecordBatch::columns and fix clippy

Signed-off-by: Ruihang Xia <[email protected]>
Co-authored-by: Ruihang Xia <[email protected]>
    @evenyag @waynexia
    evenyag and waynexia authored Dec 15, 2022
    Configuration menu
    Copy the full SHA
    142dee4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'develop' into replace-arrow2

    @evenyag
    evenyag committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    7c696da View commit details
    Browse the repository at this point in the history

  3. fix: Fix All The Tests! (#752) 

    * fix: Fix several tests compile errors

Signed-off-by: Ruihang Xia <[email protected]>

* fix: some compile errors in tests

Signed-off-by: Ruihang Xia <[email protected]>

* fix: compile errors in frontend tests

* fix: compile errors in frontend tests

* test: Fix tests in api and common-query

* test: Fix test in sql crate

* fix: resolve substrait error

Signed-off-by: Ruihang Xia <[email protected]>

* chore: add more test

* test: Fix tests in servers

* fix instance_test

Signed-off-by: Ruihang Xia <[email protected]>

* test: Fix tests in tests-integration

Signed-off-by: Ruihang Xia <[email protected]>
Co-authored-by: Lei, HUANG <[email protected]>
Co-authored-by: evenyag <[email protected]>
    @waynexia @v0y4g3r @evenyag
    3 people authored Dec 15, 2022
    Configuration menu
    Copy the full SHA
    0f3dcc1 View commit details
    Browse the repository at this point in the history

  4. fix: clippy errors

    @v0y4g3r
    v0y4g3r committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    a8630cd View commit details
    Browse the repository at this point in the history