Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

First-Class Array Abstractions #3880

Closed
tustvold opened this issue Mar 17, 2023 · 3 comments · Fixed by #4061
Closed

First-Class Array Abstractions #3880

tustvold opened this issue Mar 17, 2023 · 3 comments · Fixed by #4061
Assignees
@tustvold
Labels
arrow Changes to the arrow crate arrow-flight Changes to the arrow-flight crate enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently Array are wrappers around an ArrayData, storing it as a member of the array. This is redundant, confusing and results in API friction.

Describe the solution you'd like

I would like the Array implementations to just store their constituent parts, in the same vein as the typed ArrayData abstractions experimented with under #1799 did. This would involve the following

  • Add Array::nulls(&self) -> Option<&NullBuffer>
  • Add Array::to_data(&self) -> ArrayData
  • Implement Array::slice, Array::data_type, Array::len, Array::get_buffer_memory_size, etc... for each Array
  • Deprecate Array::data, Array::data_ref and Array::offset

Describe alternatives you've considered

Additional context

#3879
@tustvold tustvold added the enhancement Any new improvement worthy of a entry in the changelog label Mar 17, 2023
@tustvold tustvold self-assigned this Mar 17, 2023
@tustvold tustvold mentioned this issue Mar 17, 2023
Closed
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 17, 2023
@tustvold
Add Array::to_data and Array::nulls (apache#3880)
03e83c8
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 17, 2023
@tustvold
Add Array::to_data and Array::nulls (apache#3880)
0c1b2d1
@tustvold tustvold mentioned this issue Mar 17, 2023
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 17, 2023
@tustvold
Flesh out NullBuffer abstraction (apache#3880)
66ff479
@tustvold tustvold mentioned this issue Mar 17, 2023
Merged
tustvold added a commit that referenced this issue Mar 17, 2023
@tustvold
Add Array::to_data and Array::nulls (#3880) (#3881)
4e4a7f7 
* Add Array::to_data and Array::nulls (#3880)

* Review feedback

* Format
@tustvold tustvold mentioned this issue Mar 17, 2023
Closed
tustvold added a commit that referenced this issue Mar 21, 2023
@tustvold
Flesh out NullBuffer abstraction (#3880) (#3885)
67fe807 
* Flesh out NullBuffer abstraction (#3880)

* Review feedback
@tustvold tustvold mentioned this issue Mar 21, 2023
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
@tustvold
Array equality for &dyn Array (apache#3880)
8e60645
@tustvold tustvold mentioned this issue Mar 21, 2023
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
@tustvold
Array equality for &dyn Array (apache#3880)
9d618fa
@tustvold tustvold mentioned this issue Mar 22, 2023
Merged
tustvold added a commit that referenced this issue Mar 22, 2023
@tustvold
Array equality for &dyn Array (#3880) (#3899)
b1cfe84
@tustvold tustvold mentioned this issue Mar 22, 2023
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 23, 2023
@tustvold
Cleanup uses of Array::data_ref (apache#3880)
d58996f
@tustvold tustvold mentioned this issue Mar 23, 2023
Merged
tustvold added a commit that referenced this issue Mar 23, 2023
@tustvold
Cleanup uses of Array::data_ref (#3880) (#3918)
de9a90b 
* Cleanup uses of Array::data_ref (#3880)

* Further cleanup and fixes
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 23, 2023
@tustvold
Deprecate Array::data_ref (apache#3880)
d090c70
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 23, 2023
@tustvold
Deprecate Array::data_ref (apache#3880)
2801ce8
@tustvold tustvold mentioned this issue Mar 23, 2023
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 23, 2023
@tustvold
Deprecate Array::data_ref (apache#3880)
12ee185
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 23, 2023
@tustvold
Deprecate Array::data_ref (apache#3880)
71be2e7
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 24, 2023
@tustvold
Add PrimitiveArray slice (apache#3880)
e12f5b6
This was referenced Mar 24, 2023
Closed
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 24, 2023
@tustvold
Add PrimitiveArray slice (apache#3880)
5794c0c
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 24, 2023
@tustvold
Add typed buffers to UnionArray (apache#3880)
1cbf082
@tustvold tustvold mentioned this issue Mar 24, 2023
Merged
spebern pushed a commit to spebern/arrow-rs that referenced this issue Mar 25, 2023
@tustvold @spebern
Add Array::to_data and Array::nulls (apache#3880) (apache#3881)
33f8881 
* Add Array::to_data and Array::nulls (apache#3880)

* Review feedback

* Format
spebern pushed a commit to spebern/arrow-rs that referenced this issue Mar 25, 2023
@tustvold @spebern
Flesh out NullBuffer abstraction (apache#3880) (apache#3885)
f59c889 
* Flesh out NullBuffer abstraction (apache#3880)

* Review feedback
tustvold added a commit that referenced this issue Mar 28, 2023
@tustvold
Add Strongly Typed Array Slice (#3929) (#3930)
a667af8 
* Add PrimitiveArray slice (#3880)

* Add ByteArray::slice (#3929)

* Add strongly typed Array::slice (#3929)
tustvold added a commit that referenced this issue Mar 28, 2023
@tustvold
Deprecate Array::data_ref (#3880) (#3923)
151ce6f
@tustvold tustvold mentioned this issue Mar 28, 2023
Merged
tustvold added a commit that referenced this issue Mar 30, 2023
@tustvold @viirya
Add typed buffers to UnionArray (#3880) (#3933)
8d2f5dc 
* Add typed buffers to UnionArray (#3880)

* Clippy

* Update arrow-array/src/array/union_array.rs

Co-authored-by: Liang-Chi Hsieh <[email protected]>

---------

Co-authored-by: Liang-Chi Hsieh <[email protected]>
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 2, 2023
@tustvold
Cleanup more uses of Array::data (apache#3880)
2f38e6b
This was referenced Apr 2, 2023
Merged
Merged
Merged
tustvold added a commit that referenced this issue Apr 3, 2023
@tustvold
Cleanup more uses of Array::data (#3880) (#4002)
d7bba0a 
* Cleanup more uses of Array::data (#3880)

* Fix failing test

* Fix parquet map array

* Further cleanup

* Further cleanup

* More cleanup

* Fix test

* Clippy
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 4, 2023
@tustvold
Deprecate Array::data (apache#3880)
c4ea6ea
@tustvold tustvold mentioned this issue Apr 4, 2023
Merged
tustvold added a commit that referenced this issue Apr 5, 2023
@tustvold
Deprecate Array::data (#3880) (#4019)
39a48e1 
* Deprecate Array::data (#3880)

* Review feedback
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 12, 2023
@tustvold
Remove ArrayData from Array (apache#3880)
c2c61cf
@tustvold tustvold mentioned this issue Apr 12, 2023
Merged
@alamb alamb mentioned this issue Apr 12, 2023
Open
tustvold added a commit that referenced this issue Apr 12, 2023
@tustvold
Remove ArrayData from Array (#3880) (#4061)
efd8b92 
* Remove ArrayData from Array (#3880)

* Fix doc

* Fix pyarrow-integration-testing

* Review feedback
@tustvold tustvold added the arrow Changes to the arrow crate label Apr 21, 2023
@tustvold
Copy link
Contributor Author

label_issue.py automatically added labels {'arrow'} from #3877

@tustvold tustvold added the parquet Changes to the parquet crate label Apr 21, 2023
@tustvold
Copy link
Contributor Author

label_issue.py automatically added labels {'parquet'} from #3930

@tustvold
Copy link
Contributor Author

label_issue.py automatically added labels {'arrow-flight'} from #3965

@tustvold tustvold added the arrow-flight Changes to the arrow-flight crate label Apr 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tustvold tustvold

Labels
arrow Changes to the arrow crate arrow-flight Changes to the arrow-flight crate enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove ArrayData from Array (#3880) tustvold/arrow-rs
1 participant
@tustvold