[C++][Python] utf8_slice_codeunits produces invalid unicode sequence

[wirable23]

Describe the bug, including details regarding any error messages, version, and platform.

pa.compute.utf8_slice_codeunits(f"AB{chr(127917)}C{chr(127917)}ㇱD", start=2, stop=None, step=4).as_py()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow\scalar.pxi", line 632, in pyarrow.lib.StringScalar.as_py
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 4: invalid start byte
>>>

The result of utf8_slice_codeunits produced an invalid unicode sequence.

Component(s)

Python

[pitrou]

It gets even better in debug mode:

>>> import pyarrow.compute as pc
>>> import pyarrow as pa
>>> 
>>> pa.compute.utf8_slice_codeunits(f"AB{chr(127917)}C{chr(127917)}ㇱD", start=2, stop=None, step=4)
/home/antoine/arrow/dev/cpp/src/arrow/compute/kernels/scalar_string_internal.h:109:  Check failed: (output_ncodeunits) <= (max_output_ncodeunits) 

@rok Would you like to take a look?

[pitrou]

Even a trivial identity slice fails:

>>> s = f"AB{chr(127917)}C{chr(127917)}ㇱD"
>>> a = pa.array([s])
>>> pc.utf8_slice_codeunits(a, start=0, stop=None)
Traceback (most recent call last):
  Cell In[11], line 1
    pc.utf8_slice_codeunits(a, start=0, stop=None)
  File ~/arrow/dev/python/pyarrow/compute.py:261 in wrapper
    return func.call(args, options, memory_pool)
  File pyarrow/_compute.pyx:367 in pyarrow._compute.Function.call
    result = GetResultValue(
  File pyarrow/error.pxi:144 in pyarrow.lib.pyarrow_internal_check_status
    return check_status(status)
  File pyarrow/error.pxi:100 in pyarrow.lib.check_status
    raise ArrowInvalid(message)
ArrowInvalid: Negative buffer resize: -4
/home/antoine/arrow/dev/cpp/src/arrow/memory_pool.cc:931  buffer->Resize(size)
/home/antoine/arrow/dev/cpp/src/arrow/compute/kernels/scalar_string_internal.h:88  ctx->Allocate(max_output_ncodeunits)
/home/antoine/arrow/dev/cpp/src/arrow/compute/exec.cc:920  kernel_->exec(kernel_ctx_, input, &output)
/home/antoine/arrow/dev/cpp/src/arrow/compute/function.cc:276  executor->Execute(input, &listener)

Note that the input is valid:

>>> a.validate(full=True)
>>>

[pitrou]

@felipecrv @benibus Perhaps one of you is interested as well.

[benibus]

I can try taking a look ...might be nice to do some unicode debugging again

[rok]

I can take a look but need to complete something else first. Will ping if I start work.

[raulcd]

@pitrou should this be a blocker?

[pitrou]

I don't think it needs to.

[benibus]

Fix for this is in progress, but it seems the default value for SliceOptions::stop (which is INT64_MAX) isn't handled in a few calculations, resulting in overflows.

Seems kinda crazy that this hasn't come up before actually... since the specific uinicode sequence doesn't appear to be relevant.