-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[C++][Python] Conversion of RecordBatch and Table to 2D array / Tensor #40058
Comments
This was referenced Feb 13, 2024
jorisvandenbossche
pushed a commit
that referenced
this issue
Mar 5, 2024
…or (#40064) ### Rationale for this change There is no method currently in Arrow C++ to convert `Table` or `RecordBatch` to a `Tensor`. In #40058 we are proposing to add the conversion and this PR starts with the basic implementation for `RecordBatch`. ### What changes are included in this PR? Basic conversion `RecordBatch` → `Tensor` is added together with Python bindings. The implementation details are: - One data type (all columns having for example an `int32` data type) support. - No missing values support (only `NaN`). - Column-major layout of the resulting `Tensor`. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: #40059 Lead-authored-by: AlenkaF <[email protected]> Co-authored-by: Alenka Frim <[email protected]> Co-authored-by: Benjamin Kietzman <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>
mapleFU
pushed a commit
to mapleFU/arrow
that referenced
this issue
Mar 7, 2024
…w Tensor (apache#40064) ### Rationale for this change There is no method currently in Arrow C++ to convert `Table` or `RecordBatch` to a `Tensor`. In apache#40058 we are proposing to add the conversion and this PR starts with the basic implementation for `RecordBatch`. ### What changes are included in this PR? Basic conversion `RecordBatch` → `Tensor` is added together with Python bindings. The implementation details are: - One data type (all columns having for example an `int32` data type) support. - No missing values support (only `NaN`). - Column-major layout of the resulting `Tensor`. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#40059 Lead-authored-by: AlenkaF <[email protected]> Co-authored-by: Alenka Frim <[email protected]> Co-authored-by: Benjamin Kietzman <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>
thisisnic
pushed a commit
to thisisnic/arrow
that referenced
this issue
Mar 8, 2024
…w Tensor (apache#40064) ### Rationale for this change There is no method currently in Arrow C++ to convert `Table` or `RecordBatch` to a `Tensor`. In apache#40058 we are proposing to add the conversion and this PR starts with the basic implementation for `RecordBatch`. ### What changes are included in this PR? Basic conversion `RecordBatch` → `Tensor` is added together with Python bindings. The implementation details are: - One data type (all columns having for example an `int32` data type) support. - No missing values support (only `NaN`). - Column-major layout of the resulting `Tensor`. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#40059 Lead-authored-by: AlenkaF <[email protected]> Co-authored-by: Alenka Frim <[email protected]> Co-authored-by: Benjamin Kietzman <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the enhancement requested
Background
There is currently one way to convert tabular PyArrow data structure into an array with contiguous memory layout (
_Tabular
to numpy ndarray) and that is through the array protocol using the__array__
method.In the PyArrow implementation of the array protocol we use NumPy to convert each of the table columns into a numpy array and then to stack the columns together into a 2-dimensional array. The method produces a row-major layout as we stack the columns by axis 1.
There is no method in Arrow C++ to convert Table/RecordBatch to a Tensor.
There exists another conversion producing a similar result. This conversion is used to convert PyArrow
_Tabluar
object to a pandas DataFrame. Pandas DataFrame has a 2D array layout structure defined for each data type present. This 2D data structure is called a block. Therefore a dataframe with 3int8
columns and 2float32
columns will have a structure of 2 blocks, first being anint8
block with size(3, n)
and second being a float32 block of size(2, n)
.If all columns in the PyArrow table were of the same data type we would get one 2D block that would have a contiguous memory layout.
The resulting tensor memory layout is in this case column-major. For row-major memory layout the design will need a bit more research and discussion.
Code example
Feature proposal
Add a new feature in Arrow C++ to convert Arrow
Table
andRecordBatch
to aTensor
. The conversion should:Tensor
(notFixedShapeTensorArray
!),uint
(8/16/32/64),int
(8/16/32/64) andfloat
(16/32/64)NaNs
,NULL
toNaN
(validity bitmaps not supported in the resultingTensor
),We also plan to implement the DLPack protocol for the Tensor class separately. With these features the ML libraries will have the option to consume matrices (2D tensors) directly or via the DLPack protocol without the need to do their own gymnastics.
Umbrella issue
This issue is an umbrella issue for smaller tasks created for easier reviews:
Documentation issues:
Component(s)
C++, Python
The text was updated successfully, but these errors were encountered: