GH-40061: [C++][Python] Basic conversion of RecordBatch to Arrow Tensor - add option to cast NULL to NaN

[AlenkaF]

Rationale for this change

The conversion from RecordBatch to Tensor class exists but it doesn't support record batches with validity bitmaps. This PR adds support for an option to convert null values to NaN.

What changes are included in this PR?

This PR adds a nul_to_nan option in RecordBatch::ToTensor so that null values are converted to NaN in the resulting Tensor. This for example works:

>>> import pyarrow as pa
>>> batch = pa.record_batch(
...     [
...         pa.array([1, 2, 3, 4, None], type=pa.int32()),
...         pa.array([10, 20, 30, 40, None], type=pa.float32()),
...     ], names = ["a", "b"]
... )

>>> batch
pyarrow.RecordBatch
a: int32
b: float
----
a: [1,2,3,4,null]
b: [10,20,30,40,null]

>>> batch.to_tensor(null_to_nan=True)
<pyarrow.Tensor>
type: double
shape: (5, 2)
strides: (8, 40)

>>> batch.to_tensor(null_to_nan=True).to_numpy()
array([[ 1., 10.],
       [ 2., 20.],
       [ 3., 30.],
       [ 4., 40.],
       [nan, nan]])

but default would raise:

>>> batch.to_tensor()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/table.pxi", line 3421, in pyarrow.lib.RecordBatch.to_tensor
    a: int32
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
    return check_status(status)
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
    raise convert_status(status)
pyarrow.lib.ArrowTypeError: Can only convert a RecordBatch with no nulls. Set null_to_nan to true to convert nulls to nan

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

• GitHub Issue: [C++][Python] Basic conversion of RecordBatch to Arrow Tensor - add option to cast NULL to NaN #40061

[github-actions]

⚠️ GitHub issue #40061 has been automatically assigned in GitHub to PR creator.

[AlenkaF]

@bkietz this is a smaller chunk of code as a follow-up of the work done in #40060. Would be happy to get your 👀 on it. Thank you!

cc @jorisvandenbossche

[jorisvandenbossche]

Looks good!

[jorisvandenbossche]

Thanks @AlenkaF !

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 96f686b.

There were 2 benchmark results with an error:

• Commit Run on ursa-i9-9960x at 2024-03-29 07:44:49Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-14, scale_factor=10
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-21, scale_factor=10

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.