[MATLAB] Add support for creating arrow.array.ListArrays from MATLAB cell arrays supplied to the arrow.array gateway construction function
#38353
Comments
This was referenced Oct 19, 2023
kevingurney
added a commit
that referenced
this issue
Oct 23, 2023
### Rationale for this change
Now that many of the commonly-used "primitive" array types have been added to the MATLAB interface, we can implement an `arrow.array.ListArray` class.
This pull request adds a new `arrow.array.ListArray` class which can be converted to a MATLAB `cell` array by calling the static `toMATLAB` method.
### What changes are included in this PR?
1. Added a new `arrow.array.ListArray` MATLAB class.
*Methods*
`cellArray = arrow.array.ListArray.toMATLAB()`
`listArray = arrow.array.ListArray.fromArrays(offsets, values)`
*Properties*
`Offsets` - `Int32Array` list offsets (uses zero-based indexing)
`Values` - Array of values in the list (supports nesting)
2. Added a new `arrow.type.traits.ListTraits` MATLAB class.
**Example**
```matlab
>> offsets = arrow.array(int32([0, 2, 3, 7]))
offsets =
[
0,
2,
3,
7
]
>> values = arrow.array(["A", "B", "C", "D", "E", "F", "G"])
values =
[
"A",
"B",
"C",
"D",
"E",
"F",
"G"
]
>> arrowArray = arrow.array.ListArray.fromArrays(offsets, values)
arrowArray =
[
[
"A",
"B"
],
[
"C"
],
[
"D",
"E",
"F",
"G"
]
]
>> matlabArray = arrowArray.toMATLAB()
matlabArray =
3x1 cell array
{2x1 string}
{["C" ]}
{4x1 string}
>> matlabArray{:}
ans =
2x1 string array
"A"
"B"
ans =
"C"
ans =
4x1 string array
"D"
"E"
"F"
"G"
```
### Are these changes tested?
Yes.
1. Added a new `tListArray.m` test class.
2. Added a new `tListTraits.m` test class.
3. Updated `arrow.internal.test.tabular.createAllSupportedArrayTypes` to include `ListArray`.
### Are there any user-facing changes?
Yes.
1. Users can now create an `arrow.array.ListArray` from an `offsets` and `values` array by calling the static `arrow.array.ListArray.fromArrays(offsets, values)` method. `ListArray`s can be converted into MATLAB `cell` arrays by calling the static `arrow.array.ListArray.toMATLAB` method.
### Notes
1. We chose to use the "missing-class" `missing` value as the `NullSubstitutionValue` for the time being for `ListArray`. However, we eventually want to add `arrow.array.NullArray`, and will most likely want to use the "missing-class" `missing` value to represent `NullArray` values in MATLAB. So, this could cause some ambiguity in the future. We have been thinking about whether we should consider introducing some sort of special "sentinel value" to represent null values when converting to MATLAB `cell` arrays. Perhaps, something like `arrow.Null`, or something to that effect, in order to avoid this ambiguity. If we think it makes sense to do that, we may want to retroactively change the `NullSubstitutionValue` to be `arrow.Null` and break compatibility. Since we are still in pre-`0.1`, we don't think the impact of such a behavior change would be very large.
2. Implementing `ListArray` is fairly involved. So, in the spirit of incremental delivery, we chose not to include an implementation of `arrow.array.ListArray.fromMATLAB` in this initial pull request. We plan on following up with some more changes to `arrow.array.ListArray`. See #38353, #38354, and #38361.
3. Thank you @ sgilmore10 for your help with this pull request!
### Future Directions
1. #38353
2. #38354
3. #38361
4. Consider adding a null sentinel value like `arrow.Null` for conversion to MATLAB `cell` arrays.
* Closes: #37815
Lead-authored-by: Kevin Gurney <[email protected]>
Co-authored-by: Sarah Gilmore <[email protected]>
Signed-off-by: Kevin Gurney <[email protected]>
JerAguilon
pushed a commit
to JerAguilon/arrow
that referenced
this issue
Oct 25, 2023
…ache#38357) ### Rationale for this change Now that many of the commonly-used "primitive" array types have been added to the MATLAB interface, we can implement an `arrow.array.ListArray` class. This pull request adds a new `arrow.array.ListArray` class which can be converted to a MATLAB `cell` array by calling the static `toMATLAB` method. ### What changes are included in this PR? 1. Added a new `arrow.array.ListArray` MATLAB class. *Methods* `cellArray = arrow.array.ListArray.toMATLAB()` `listArray = arrow.array.ListArray.fromArrays(offsets, values)` *Properties* `Offsets` - `Int32Array` list offsets (uses zero-based indexing) `Values` - Array of values in the list (supports nesting) 2. Added a new `arrow.type.traits.ListTraits` MATLAB class. **Example** ```matlab >> offsets = arrow.array(int32([0, 2, 3, 7])) offsets = [ 0, 2, 3, 7 ] >> values = arrow.array(["A", "B", "C", "D", "E", "F", "G"]) values = [ "A", "B", "C", "D", "E", "F", "G" ] >> arrowArray = arrow.array.ListArray.fromArrays(offsets, values) arrowArray = [ [ "A", "B" ], [ "C" ], [ "D", "E", "F", "G" ] ] >> matlabArray = arrowArray.toMATLAB() matlabArray = 3x1 cell array {2x1 string} {["C" ]} {4x1 string} >> matlabArray{:} ans = 2x1 string array "A" "B" ans = "C" ans = 4x1 string array "D" "E" "F" "G" ``` ### Are these changes tested? Yes. 1. Added a new `tListArray.m` test class. 2. Added a new `tListTraits.m` test class. 3. Updated `arrow.internal.test.tabular.createAllSupportedArrayTypes` to include `ListArray`. ### Are there any user-facing changes? Yes. 1. Users can now create an `arrow.array.ListArray` from an `offsets` and `values` array by calling the static `arrow.array.ListArray.fromArrays(offsets, values)` method. `ListArray`s can be converted into MATLAB `cell` arrays by calling the static `arrow.array.ListArray.toMATLAB` method. ### Notes 1. We chose to use the "missing-class" `missing` value as the `NullSubstitutionValue` for the time being for `ListArray`. However, we eventually want to add `arrow.array.NullArray`, and will most likely want to use the "missing-class" `missing` value to represent `NullArray` values in MATLAB. So, this could cause some ambiguity in the future. We have been thinking about whether we should consider introducing some sort of special "sentinel value" to represent null values when converting to MATLAB `cell` arrays. Perhaps, something like `arrow.Null`, or something to that effect, in order to avoid this ambiguity. If we think it makes sense to do that, we may want to retroactively change the `NullSubstitutionValue` to be `arrow.Null` and break compatibility. Since we are still in pre-`0.1`, we don't think the impact of such a behavior change would be very large. 2. Implementing `ListArray` is fairly involved. So, in the spirit of incremental delivery, we chose not to include an implementation of `arrow.array.ListArray.fromMATLAB` in this initial pull request. We plan on following up with some more changes to `arrow.array.ListArray`. See apache#38353, apache#38354, and apache#38361. 3. Thank you @ sgilmore10 for your help with this pull request! ### Future Directions 1. apache#38353 2. apache#38354 3. apache#38361 4. Consider adding a null sentinel value like `arrow.Null` for conversion to MATLAB `cell` arrays. * Closes: apache#37815 Lead-authored-by: Kevin Gurney <[email protected]> Co-authored-by: Sarah Gilmore <[email protected]> Signed-off-by: Kevin Gurney <[email protected]>
loicalleyne
pushed a commit
to loicalleyne/arrow
that referenced
this issue
Nov 13, 2023
…ache#38357) ### Rationale for this change Now that many of the commonly-used "primitive" array types have been added to the MATLAB interface, we can implement an `arrow.array.ListArray` class. This pull request adds a new `arrow.array.ListArray` class which can be converted to a MATLAB `cell` array by calling the static `toMATLAB` method. ### What changes are included in this PR? 1. Added a new `arrow.array.ListArray` MATLAB class. *Methods* `cellArray = arrow.array.ListArray.toMATLAB()` `listArray = arrow.array.ListArray.fromArrays(offsets, values)` *Properties* `Offsets` - `Int32Array` list offsets (uses zero-based indexing) `Values` - Array of values in the list (supports nesting) 2. Added a new `arrow.type.traits.ListTraits` MATLAB class. **Example** ```matlab >> offsets = arrow.array(int32([0, 2, 3, 7])) offsets = [ 0, 2, 3, 7 ] >> values = arrow.array(["A", "B", "C", "D", "E", "F", "G"]) values = [ "A", "B", "C", "D", "E", "F", "G" ] >> arrowArray = arrow.array.ListArray.fromArrays(offsets, values) arrowArray = [ [ "A", "B" ], [ "C" ], [ "D", "E", "F", "G" ] ] >> matlabArray = arrowArray.toMATLAB() matlabArray = 3x1 cell array {2x1 string} {["C" ]} {4x1 string} >> matlabArray{:} ans = 2x1 string array "A" "B" ans = "C" ans = 4x1 string array "D" "E" "F" "G" ``` ### Are these changes tested? Yes. 1. Added a new `tListArray.m` test class. 2. Added a new `tListTraits.m` test class. 3. Updated `arrow.internal.test.tabular.createAllSupportedArrayTypes` to include `ListArray`. ### Are there any user-facing changes? Yes. 1. Users can now create an `arrow.array.ListArray` from an `offsets` and `values` array by calling the static `arrow.array.ListArray.fromArrays(offsets, values)` method. `ListArray`s can be converted into MATLAB `cell` arrays by calling the static `arrow.array.ListArray.toMATLAB` method. ### Notes 1. We chose to use the "missing-class" `missing` value as the `NullSubstitutionValue` for the time being for `ListArray`. However, we eventually want to add `arrow.array.NullArray`, and will most likely want to use the "missing-class" `missing` value to represent `NullArray` values in MATLAB. So, this could cause some ambiguity in the future. We have been thinking about whether we should consider introducing some sort of special "sentinel value" to represent null values when converting to MATLAB `cell` arrays. Perhaps, something like `arrow.Null`, or something to that effect, in order to avoid this ambiguity. If we think it makes sense to do that, we may want to retroactively change the `NullSubstitutionValue` to be `arrow.Null` and break compatibility. Since we are still in pre-`0.1`, we don't think the impact of such a behavior change would be very large. 2. Implementing `ListArray` is fairly involved. So, in the spirit of incremental delivery, we chose not to include an implementation of `arrow.array.ListArray.fromMATLAB` in this initial pull request. We plan on following up with some more changes to `arrow.array.ListArray`. See apache#38353, apache#38354, and apache#38361. 3. Thank you @ sgilmore10 for your help with this pull request! ### Future Directions 1. apache#38353 2. apache#38354 3. apache#38361 4. Consider adding a null sentinel value like `arrow.Null` for conversion to MATLAB `cell` arrays. * Closes: apache#37815 Lead-authored-by: Kevin Gurney <[email protected]> Co-authored-by: Sarah Gilmore <[email protected]> Signed-off-by: Kevin Gurney <[email protected]>
dgreiss
pushed a commit
to dgreiss/arrow
that referenced
this issue
Feb 19, 2024
…ache#38357) ### Rationale for this change Now that many of the commonly-used "primitive" array types have been added to the MATLAB interface, we can implement an `arrow.array.ListArray` class. This pull request adds a new `arrow.array.ListArray` class which can be converted to a MATLAB `cell` array by calling the static `toMATLAB` method. ### What changes are included in this PR? 1. Added a new `arrow.array.ListArray` MATLAB class. *Methods* `cellArray = arrow.array.ListArray.toMATLAB()` `listArray = arrow.array.ListArray.fromArrays(offsets, values)` *Properties* `Offsets` - `Int32Array` list offsets (uses zero-based indexing) `Values` - Array of values in the list (supports nesting) 2. Added a new `arrow.type.traits.ListTraits` MATLAB class. **Example** ```matlab >> offsets = arrow.array(int32([0, 2, 3, 7])) offsets = [ 0, 2, 3, 7 ] >> values = arrow.array(["A", "B", "C", "D", "E", "F", "G"]) values = [ "A", "B", "C", "D", "E", "F", "G" ] >> arrowArray = arrow.array.ListArray.fromArrays(offsets, values) arrowArray = [ [ "A", "B" ], [ "C" ], [ "D", "E", "F", "G" ] ] >> matlabArray = arrowArray.toMATLAB() matlabArray = 3x1 cell array {2x1 string} {["C" ]} {4x1 string} >> matlabArray{:} ans = 2x1 string array "A" "B" ans = "C" ans = 4x1 string array "D" "E" "F" "G" ``` ### Are these changes tested? Yes. 1. Added a new `tListArray.m` test class. 2. Added a new `tListTraits.m` test class. 3. Updated `arrow.internal.test.tabular.createAllSupportedArrayTypes` to include `ListArray`. ### Are there any user-facing changes? Yes. 1. Users can now create an `arrow.array.ListArray` from an `offsets` and `values` array by calling the static `arrow.array.ListArray.fromArrays(offsets, values)` method. `ListArray`s can be converted into MATLAB `cell` arrays by calling the static `arrow.array.ListArray.toMATLAB` method. ### Notes 1. We chose to use the "missing-class" `missing` value as the `NullSubstitutionValue` for the time being for `ListArray`. However, we eventually want to add `arrow.array.NullArray`, and will most likely want to use the "missing-class" `missing` value to represent `NullArray` values in MATLAB. So, this could cause some ambiguity in the future. We have been thinking about whether we should consider introducing some sort of special "sentinel value" to represent null values when converting to MATLAB `cell` arrays. Perhaps, something like `arrow.Null`, or something to that effect, in order to avoid this ambiguity. If we think it makes sense to do that, we may want to retroactively change the `NullSubstitutionValue` to be `arrow.Null` and break compatibility. Since we are still in pre-`0.1`, we don't think the impact of such a behavior change would be very large. 2. Implementing `ListArray` is fairly involved. So, in the spirit of incremental delivery, we chose not to include an implementation of `arrow.array.ListArray.fromMATLAB` in this initial pull request. We plan on following up with some more changes to `arrow.array.ListArray`. See apache#38353, apache#38354, and apache#38361. 3. Thank you @ sgilmore10 for your help with this pull request! ### Future Directions 1. apache#38353 2. apache#38354 3. apache#38361 4. Consider adding a null sentinel value like `arrow.Null` for conversion to MATLAB `cell` arrays. * Closes: apache#37815 Lead-authored-by: Kevin Gurney <[email protected]> Co-authored-by: Sarah Gilmore <[email protected]> Signed-off-by: Kevin Gurney <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the enhancement requested
In order to make it easy to create
arrow.array.ListArrays, we should add support for creating them from MATLABcellarrays passed to thearrow.arraygateway construction function.Component(s)
MATLAB
The text was updated successfully, but these errors were encountered: