Support fixed_size_list for make_array #6759

jayzhan211 · 2023-06-24T12:26:17Z

Which issue does this PR close?

Ref #6560

Rationale for this change

Add FixedSizeList support for array methods

What changes are included in this PR?

Only MakeArray is done for this PR.

Add casting from fixedSizeList to list in analyzer (type_coercion).
Support ScalarValue::FixedSizeList, not all feature is implemented
Able to parse List for sqllogictest.

Are these changes tested?

sqllogictest
array.slt
unit test in type_coercion.rs

Are there any user-facing changes?

Task

MakeArray
Other array methods

jayzhan211 · 2023-06-25T00:10:09Z

datafusion/core/tests/data/fixed_size_list_array.parquet

I create fixedsizelist parquet from my own script.
Is there any existing way for us to easily create/update this kind of test case?

Did you look at the data sets available in https://github.com/apache/parquet-testing (which is a submodule of this repo)

e.g. https://github.com/apache/parquet-testing/tree/a11fc8f148f8a7a89d9281cc0da3eb9d56095fbf/data

alamb

Thank you for this contribution @jayzhan211 -- I think other than the patch to crates.io this PR is ready to go.

I left some suggestions, which I think would improve the PR

alamb · 2023-06-26T21:09:14Z

Cargo.toml

@@ -53,6 +53,14 @@ arrow-array = { version = "42.0.0", default-features = false, features = ["chron
 parquet = { version = "42.0.0", features = ["arrow", "async", "object_store"] }
 sqlparser = { version = "0.35", features = ["visitor"] }

+[patch.crates-io]


I think we will have to wait for this code to be released to crates.io (in about 10 days) before merging this PR

alamb · 2023-06-26T21:09:47Z

datafusion/common/src/scalar.rs

@@ -3646,29 +3675,9 @@ impl fmt::Display for ScalarValue {
            ScalarValue::TimestampNanosecond(e, _) => format_option!(f, e)?,
            ScalarValue::Utf8(e) => format_option!(f, e)?,
            ScalarValue::LargeUtf8(e) => format_option!(f, e)?,
-            ScalarValue::Binary(e) => match e {


alamb · 2023-06-26T21:11:42Z

datafusion/core/tests/data/fixed_size_list_array.parquet

Did you look at the data sets available in https://github.com/apache/parquet-testing (which is a submodule of this repo)

e.g. https://github.com/apache/parquet-testing/tree/a11fc8f148f8a7a89d9281cc0da3eb9d56095fbf/data

datafusion/core/tests/sqllogictests/test_files/array.slt

alamb · 2023-06-26T21:13:51Z

datafusion/sql/src/expr/arrow_cast.rs

@@ -486,6 +497,8 @@ impl<'a> Tokenizer<'a> {
            "Date32" => Token::SimpleType(DataType::Date32),
            "Date64" => Token::SimpleType(DataType::Date64),

+            "List" => Token::List,


alamb · 2023-06-26T21:15:20Z

FYI @izveigor

alamb · 2023-06-28T18:15:51Z

Marking as a draft as it looks like this PR is not waiting on review -- it is waiting on an upstream release (scheduled early next week) as well as to address some other comments

Signed-off-by: jayzhan211 <[email protected]>

* feat: supports NULL in arrays * feat: supports NULL in array functions * fix: array_fill error * fix: merge * fix: cargo fmt --------- Co-authored-by: Andrew Lamb <[email protected]>

Signed-off-by: jayzhan211 <[email protected]>

jayzhan211 · 2023-07-05T01:08:25Z

datafusion/physical-expr/src/array_expressions.rs

@@ -125,7 +125,7 @@ fn array_array(args: &[ArrayRef], data_type: DataType) -> Result<ArrayRef> {
            }

            let list_data_type =
-                DataType::List(Arc::new(Field::new("item", data_type, false)));
+                DataType::List(Arc::new(Field::new("item", data_type, true)));


Revert the change from #6662 to avoid failure on the schema mismatch

Signed-off-by: jayzhan211 <[email protected]>

alamb

Thank you @jayzhan211

alamb · 2023-07-05T17:35:16Z

cc @izveigor

* support make_array for fixed_size_list Signed-off-by: jayzhan211 <[email protected]> * add arrow-typeof in test Signed-off-by: jayzhan211 <[email protected]> * fix schema mismatch Signed-off-by: jayzhan211 <[email protected]> * cleanup code Signed-off-by: jayzhan211 <[email protected]> * create array data with correct len Signed-off-by: jayzhan211 <[email protected]> --------- Signed-off-by: jayzhan211 <[email protected]>

github-actions bot added core Core DataFusion crate optimizer Optimizer rules sql SQL Planner sqllogictest SQL Logic Tests (.slt) labels Jun 24, 2023

jayzhan211 commented Jun 25, 2023

View reviewed changes

alamb approved these changes Jun 26, 2023

View reviewed changes

alamb added the waiting-on-upstream PR is waiting on an upstream dependency to be updated label Jun 26, 2023

tustvold force-pushed the main branch from 9dd3c2b to 70d633a Compare June 27, 2023 09:44

tustvold changed the base branch from main to backup-main June 27, 2023 11:12

tustvold changed the base branch from backup-main to main June 27, 2023 11:12

alamb marked this pull request as draft June 28, 2023 18:15

jayzhan211 added 2 commits July 4, 2023 21:03

support make_array for fixed_size_list

b9ec2a1

Signed-off-by: jayzhan211 <[email protected]>

add arrow-typeof in test

9a8b1d2

Signed-off-by: jayzhan211 <[email protected]>

jayzhan211 referenced this pull request Jul 5, 2023

feat: support NULL in array functions (#6662)

9edfcdc

* feat: supports NULL in arrays * feat: supports NULL in array functions * fix: array_fill error * fix: merge * fix: cargo fmt --------- Co-authored-by: Andrew Lamb <[email protected]>

jayzhan211 added 2 commits July 5, 2023 08:58

fix schema mismatch

6d80691

Signed-off-by: jayzhan211 <[email protected]>

cleanup code

c6ba92a

Signed-off-by: jayzhan211 <[email protected]>

jayzhan211 force-pushed the fixedsizelist_autocast branch from ecc93dd to c6ba92a Compare July 5, 2023 01:04

github-actions bot added the physical-expr Physical Expressions label Jul 5, 2023

jayzhan211 commented Jul 5, 2023

View reviewed changes

jayzhan211 marked this pull request as ready for review July 5, 2023 02:24

jayzhan211 marked this pull request as draft July 5, 2023 07:27

create array data with correct len

6bd40a0

Signed-off-by: jayzhan211 <[email protected]>

jayzhan211 marked this pull request as ready for review July 5, 2023 08:41

alamb approved these changes Jul 5, 2023

View reviewed changes

alamb merged commit 658206a into apache:main Jul 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support fixed_size_list for make_array #6759

Support fixed_size_list for make_array #6759

jayzhan211 commented Jun 24, 2023

jayzhan211 Jun 25, 2023

alamb Jun 26, 2023

alamb left a comment •

edited

Loading

alamb Jun 26, 2023

alamb Jun 26, 2023

alamb Jun 26, 2023

alamb Jun 26, 2023

alamb commented Jun 26, 2023

alamb commented Jun 28, 2023

jayzhan211 Jul 5, 2023 •

edited

Loading

alamb left a comment

alamb commented Jul 5, 2023

Support fixed_size_list for make_array #6759

Support fixed_size_list for make_array #6759

Conversation

jayzhan211 commented Jun 24, 2023

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Task

jayzhan211 Jun 25, 2023

Choose a reason for hiding this comment

alamb Jun 26, 2023

Choose a reason for hiding this comment

alamb left a comment • edited Loading

Choose a reason for hiding this comment

alamb Jun 26, 2023

Choose a reason for hiding this comment

alamb Jun 26, 2023

Choose a reason for hiding this comment

alamb Jun 26, 2023

Choose a reason for hiding this comment

alamb Jun 26, 2023

Choose a reason for hiding this comment

alamb commented Jun 26, 2023

alamb commented Jun 28, 2023

jayzhan211 Jul 5, 2023 • edited Loading

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

alamb commented Jul 5, 2023

alamb left a comment •

edited

Loading

jayzhan211 Jul 5, 2023 •

edited

Loading