[C++] Change Scalar::CastTo to do safe cast by default and allow to specify cast options ?

[jorisvandenbossche]

Describe the enhancement requested

See #34901 for a longer discussion, but summarizing: the pyarrow.Scalar object has a cast() method, but in contrast with other cast methods in pyarrow it does an unsafe cast by default. We should probably change this to do a safe cast by default, and at the same time also allow to specify CastOptions (so a user can still choose to do an unsafe cast).

Example:

# scalar behaviour
>>> pa.scalar(1.5)
<pyarrow.DoubleScalar: 1.5>
>>> pa.scalar(1.5).cast(pa.int64())
<pyarrow.Int64Scalar: 1>

# vs array behaviour
>>> pa.array([1.5]).cast(pa.int64())
...
ArrowInvalid: Float value 1.5 was truncated converting to int64

The python cast() method calls the C++ Scalar::ToCast:

arrow/cpp/src/arrow/scalar.h

Lines 99 to 100 in e488942

	// TODO(bkietz) add compute::CastOptions
	Result<std::shared_ptr<Scalar>> CastTo(std::shared_ptr<DataType> to) const;

which currently indeed doesn't have the option to pass CastOptions.

In addition, it seems that for casting Scalars, we do have a somewhat custom implementation, and this doesn't use the generic Cast implementation (from the compute kernels), but has a custom CastImpl in scalar.cc. Not fully sure about the reason for this, but maybe historically we wanted to have scalar casting without relying on the optional compute module? (cfr #25025)

Component(s)

C++

[westonpace]

I think this is a good idea. The result will probably be a slight bit slower but working with scalars is already slightly slower anyways and it would be nice to have only a single implementation.

[westonpace]

Also, we've since extracted the cast kernels from the rest of the kernels and declared them as "essential kernels" that will always be present (even if ARROW_COMPUTE is OFF) so I think it is safe to rely on them.

[danepitkin]

FYI I tried swapping the Scalar.cast() implementation to the compute kernel and current (but small) test suite passed while also fixing #35370.

...
    def cast(self, object target_type=None, safe=None, options=None, memory_pool=None):
        return _pc().cast(self, target_type, safe=safe,
                          options=options, memory_pool=memory_pool)
...

[AlenkaF]

Casting of scalar now uses the compute kernel in pyarrow: #35395

Do we want to keep this issue open to also make changes to the Scalar::CastTo in the C++? Or maybe linking the PR for pyarrow with a different issue? @jorisvandenbossche

[danepitkin]

I created #35560 to track the issue!

[AlenkaF]

Great Dane, thanks!