[Python] Implement __arrow_c_stream__ on ChunkedArray

[kylebarron]

Describe the enhancement requested

I'm really excited about the new PyCapsule API.

It doesn't look like any dunder methods have been implemented on the ChunkedArray class. It would seem natural to implement __arrow_c_stream__ on ChunkedArray. Otherwise, other bindings have to go through a pyarrow-specific API to page through each array.

Component(s)

Python

[jorisvandenbossche]

Indeed, it seems we missed that in the initial PR. I think it makes a lot of sense to add the stream protocol for ChunkedArray.

[wjones127]

IIRC, Arrow C++ doesn't support exporting arrays as a C Arrow Stream, so I think that needs to be implemented first.

[jorisvandenbossche]

Ah, yes, now I remember. For the stream support, we currently generally assume you have a stream of batches, i.e. some form of RecordBatchReader

[paleolimbot]

Yes, I think this needs C++-level support (but I think it's worth it!). I will try to take a stab at an implementation before 15.0.0...I would like to use it in the R bindings as well.

To summarise where this came up in GeoArrow-land...we'd like to base our Python ecosystem on the C data/stream interfaces (e.g., functions use the dunder methods when consuming input, and return an object that implements a dunder); however, right now we can't consume or return ChunkedArray. We could force a concatenation (maybe expensive and maybe resulting in overflow since one common extension type is based on binary), or we could constantly special-case the ChunkedArray and loop over chunks at the Python level (verbose, and performance degrades when chunks get small/numerous). The ChunkedArray is the most common Array representation in pyarrow (e.g., column in a Table!), so this comes up quite a bit.