Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

array functions don't work with FixedSizeList #8084

Open
universalmind303 opened this issue Nov 7, 2023 · 4 comments
Open

array functions don't work with FixedSizeList #8084

universalmind303 opened this issue Nov 7, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@universalmind303
Copy link
Contributor

Describe the bug

none of the scalar array_ functions work on FixedSizeList dtype.

To Reproduce

given a table of

+-----------------+--------+
| fixed_size_list | list   |
+-----------------+--------+
| [0, 1]          | [0, 1] |
| [0, 2]          | [0, 2] |
+-----------------+--------+

I should be able to use array_* functions on both list and fixed_size_list fields

SELECT array_length(list) FROM my_table

+-----------------------------+
| array_length(my_table.list) |
+-----------------------------+
| 2                           |
| 2                           |
+-----------------------------+

Full reproducible example here: https://gist.github.com/universalmind303/e2e59cd6d84656f5f20e78bd6e9c6c74

Expected behavior

SELECT array_length(fixed_size_list) FROM my_table

+----------------------------------------+
| array_length(my_table.fixed_size_list) |
+----------------------------------------+
| 2                                      |
| 2                                      |
+----------------------------------------+

Additional context

I don't think all of them should work. Such as ones that mutate the array (repeat, remove, ...). But I'd expect many of the immutable functions to work such as has, length, element, and so on.

@universalmind303 universalmind303 added the bug Something isn't working label Nov 7, 2023
@alamb
Copy link
Contributor

alamb commented Nov 7, 2023

The same thing probably goes for LargeList as well (it isn't supported)

@Weijun-H
Copy link
Member

Weijun-H commented Nov 8, 2023

The same thing probably goes for LargeList as well (it isn't supported)

Yes, I believe the array function now only accepts List. I am happy to take this ticket.

@alamb
Copy link
Contributor

alamb commented Nov 8, 2023

Yes, I believe the array function now only accepts List. I am happy to take this ticket.

Thanks @Weijun-H

In order to make this project tractable, I suggest picking one array function (ideally a simple one) and figuring out a good pattern for:

  1. Testing
  2. Implementing

In particular, I think this may require writing some generic code (over OffsetSize) trait

@jayzhan211
Copy link
Contributor

For LargeList, you can take #7897 for reference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants