Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++][Python] Conversion of RecordBatch and Table to 2D array / Tensor #40058

Open
7 of 12 tasks
AlenkaF opened this issue Feb 13, 2024 · 0 comments
Open
7 of 12 tasks

[C++][Python] Conversion of RecordBatch and Table to 2D array / Tensor #40058

AlenkaF opened this issue Feb 13, 2024 · 0 comments

Comments

@AlenkaF
Copy link
Member

AlenkaF commented Feb 13, 2024

Describe the enhancement requested

Background

There is currently one way to convert tabular PyArrow data structure into an array with contiguous memory layout (_Tabular to numpy ndarray) and that is through the array protocol using the __array__ method.

In the PyArrow implementation of the array protocol we use NumPy to convert each of the table columns into a numpy array and then to stack the columns together into a 2-dimensional array. The method produces a row-major layout as we stack the columns by axis 1.

There is no method in Arrow C++ to convert Table/RecordBatch to a Tensor.

There exists another conversion producing a similar result. This conversion is used to convert PyArrow _Tabluar object to a pandas DataFrame. Pandas DataFrame has a 2D array layout structure defined for each data type present. This 2D data structure is called a block. Therefore a dataframe with 3 int8 columns and 2 float32 columns will have a structure of 2 blocks, first being an int8 block with size (3, n) and second being a float32 block of size (2, n).

If all columns in the PyArrow table were of the same data type we would get one 2D block that would have a contiguous memory layout.

The resulting tensor memory layout is in this case column-major. For row-major memory layout the design will need a bit more research and discussion.

Code example
In [1]: import pyarrow as pa
   ...: import numpy as np
   ...: 
   ...: # Construct a pyarrow.Table
   ...: arr_1 = pa.array([2, 4, 5, 100], type=pa.int8())
   ...: arr_2 = pa.array([1, 2, 3, 4], type=pa.int16())
   ...: names = ["arr_1", "arr_2"]
   ...: t = pa.table([arr_1, arr_2], names=names)
   ...: # Define common data type and cast all the columns
   ...: common_dtype = pa.int16()
   ...: column_arrays = [
   ...:     t.column(i).cast(target_type=common_dtype) for i in range(t.num_columns)
   ...: ]
   ...: 
   ...: options = dict(
   ...:     pool=None,
   ...:     strings_to_categorical=False,
   ...:     zero_copy_only=False,
   ...:     integer_object_nulls=False,
   ...:     date_as_object=True,
   ...:     timestamp_as_object=False,
   ...:     use_threads=True,
   ...:     deduplicate_objects=True,
   ...:     safe=True,
   ...:     split_blocks=False,
   ...:     self_destruct=False,
   ...:     maps_as_pydicts=None,
   ...:     coerce_temporal_nanoseconds=False
   ...: )
   ...: 
   ...: # CONVERT TABLE TO 2 DIMENSIONAL NDARRAY
   ...:
   ...: # Using pyarrow -> pandas conversion
   ...: r1 = pa.lib.table_to_blocks(options, pa.table(column_arrays, names=names), None, None)[0]["block"]
   ...: # Using array protocol
   ...: r2 = t.__array__()

In [2]: t
Out[2]: 
pyarrow.Table
arr_1: int8
arr_2: int16
----
arr_1: [[2,4,5,100]]
arr_2: [[1,2,3,4]]

In [3]: r1
Out[3]: 
array([[  2,   4,   5, 100],
       [  1,   2,   3,   4]], dtype=int16)

In [4]: r2
Out[4]: 
array([[  2,   1],
       [  4,   2],
       [  5,   3],
       [100,   4]], dtype=int16)

In [5]: r1.flags
Out[5]: 
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False

In [6]: r2.flags
Out[6]: 
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False

Feature proposal

Add a new feature in Arrow C++ to convert Arrow Table and RecordBatch to a Tensor. The conversion should:

  • return an Arrow Tensor (not FixedShapeTensorArray!),
  • support following data types: uint (8/16/32/64), int (8/16/32/64) and float (16/32/64)
  • support NaNs,
  • add an option to convert NULL to NaN (validity bitmaps not supported in the resulting Tensor),
  • support column and row-major layout.

We also plan to implement the DLPack protocol for the Tensor class separately. With these features the ML libraries will have the option to consume matrices (2D tensors) directly or via the DLPack protocol without the need to do their own gymnastics.

Umbrella issue

This issue is an umbrella issue for smaller tasks created for easier reviews:

Documentation issues:

Component(s)

C++, Python

@AlenkaF AlenkaF self-assigned this Feb 13, 2024
jorisvandenbossche pushed a commit that referenced this issue Mar 5, 2024
…or (#40064)

### Rationale for this change

There is no method currently in Arrow C++ to convert `Table` or `RecordBatch` to a `Tensor`. In #40058 we are proposing to add the conversion and this PR starts with the basic implementation for `RecordBatch`.

### What changes are included in this PR?

Basic conversion `RecordBatch` → `Tensor` is added together with Python bindings. The implementation details are:

- One data type (all columns having for example an `int32` data type) support.
- No missing values support (only `NaN`).
- Column-major layout of the resulting `Tensor`.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: #40059

Lead-authored-by: AlenkaF <[email protected]>
Co-authored-by: Alenka Frim <[email protected]>
Co-authored-by: Benjamin Kietzman <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
mapleFU pushed a commit to mapleFU/arrow that referenced this issue Mar 7, 2024
…w Tensor (apache#40064)

### Rationale for this change

There is no method currently in Arrow C++ to convert `Table` or `RecordBatch` to a `Tensor`. In apache#40058 we are proposing to add the conversion and this PR starts with the basic implementation for `RecordBatch`.

### What changes are included in this PR?

Basic conversion `RecordBatch` → `Tensor` is added together with Python bindings. The implementation details are:

- One data type (all columns having for example an `int32` data type) support.
- No missing values support (only `NaN`).
- Column-major layout of the resulting `Tensor`.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: apache#40059

Lead-authored-by: AlenkaF <[email protected]>
Co-authored-by: Alenka Frim <[email protected]>
Co-authored-by: Benjamin Kietzman <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
thisisnic pushed a commit to thisisnic/arrow that referenced this issue Mar 8, 2024
…w Tensor (apache#40064)

### Rationale for this change

There is no method currently in Arrow C++ to convert `Table` or `RecordBatch` to a `Tensor`. In apache#40058 we are proposing to add the conversion and this PR starts with the basic implementation for `RecordBatch`.

### What changes are included in this PR?

Basic conversion `RecordBatch` → `Tensor` is added together with Python bindings. The implementation details are:

- One data type (all columns having for example an `int32` data type) support.
- No missing values support (only `NaN`).
- Column-major layout of the resulting `Tensor`.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* Closes: apache#40059

Lead-authored-by: AlenkaF <[email protected]>
Co-authored-by: Alenka Frim <[email protected]>
Co-authored-by: Benjamin Kietzman <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant