ARROW-9761: [C/C++] Add experimental C stream inferface #8052

pitrou · 2020-08-25T17:30:39Z

The goal is to have a standardized ABI to communicate streams of homogeneous arrays or record batches (for example for database result sets).

The trickiest part is error reporting. This proposal tries to strike a compromise between simplicity (an integer error code mapping to errno values) and expressivity (an optional description string for application-specific and context-specific details).

pitrou · 2020-08-25T17:30:48Z

cc @wesm

github-actions · 2020-08-25T17:32:53Z

https://issues.apache.org/jira/browse/ARROW-9761

emkornfield · 2020-08-26T04:12:16Z

cpp/src/arrow/c/abi.h

+  int (*get_schema)(struct ArrowArrayStream*, struct ArrowSchema* out);
+  // Callback to get the next array
+  // (if no error and the array is released, the stream has ended)
+  // Return value: 0 if successful, an `errno`-compatible error code otherwise.


silly question is errno-compatible a well defined unix/windows term? someplace where I can read more about it?

I don't know how to phrase it: it returns value that are errno error codes (in case of error). A number of values are standard in C++: https://en.cppreference.com/w/cpp/error/errno_macros

emkornfield · 2020-08-26T04:13:19Z

cpp/src/arrow/c/abi.h

+  // Release callback: release the stream's own resources.
+  // Note that arrays returned by `get_next` must be individually released.
+  void (*release)(struct ArrowArrayStream*);
+  // Opaque producer-specific data


nit ntew line here or remove the new line above?

emkornfield · 2020-08-26T04:14:56Z

cpp/src/arrow/c/abi.h

+  const char* (*get_last_error)(struct ArrowArrayStream*);
+
+  // Release callback: release the stream's own resources.
+  // Note that arrays returned by `get_next` must be individually released.


are arrays produced by this stream tied to the lifecycle of this stream (must they be released first?)

Hmm... I'd say no.

emkornfield · 2020-08-26T04:15:48Z

cpp/src/arrow/c/abi.h

+  // Return value: 0 if successful, an `errno`-compatible error code otherwise.
+  int (*get_schema)(struct ArrowArrayStream*, struct ArrowSchema* out);
+  // Callback to get the next array
+  // (if no error and the array is released, the stream has ended)


released is something defined in the ArrowArray-spec. Is it stronger or weaker guarantee then returning a nullptr here?

A nullptr cannot be returned. The callback returns an int. However, we could say that returning -1 means end of stream.

hannes · 2020-08-26T05:55:27Z

I have one suggestion: The proposal indicates that the stream is finished when the array resulting from get_next is released. This seems a bit odd, how about just setting its length to 0? Or is it possible in the stream API that individual chunks are of length 0 but subsequent chunks are not?

pitrou · 2020-08-26T09:16:33Z

It's possible to have some chunks with 0 length in a stream, yes. I don't see any reason to forbid it in this API (and such corner cases are often annoying to deal with).

pitrou · 2020-08-26T13:23:32Z

However, as said above, perhaps returning -1 (rather than 0 for success or a positive errno-like value for errors) could mean an end of stream. Thoughts?

wesm

This looks good to me. The next steps would be to integrate this in pyarrow and do the plumbing to get something working end-to-end with a third party project (e.g. DuckDB)

pitrou · 2020-09-01T15:02:08Z

I added some docs, please take a look.

pitrou · 2020-09-09T13:49:25Z

Rebased. Does someone want to review the doc at https://pitrou.net/arrowdevdoc/format/CStreamInterface.html ?

emkornfield · 2020-09-10T05:35:50Z

Rebased. Does someone want to review the doc at https://pitrou.net/arrowdevdoc/format/CStreamInterface.html ?

I took a quick look and seemed reasonable (nothing jumped out at me that needed changes).

pitrou · 2020-09-21T15:18:27Z

Rebased again. We can probably merge soon, if CI agrees.

hannes · 2020-09-21T18:01:15Z

One thing we noticed: There is no way to rewind such a stream, but we would really like to be able to do so. For example, if someone defines an Arrow-backed SQL VIEW, it can be scanned multiple times. Once it has been read once, it is currently invalidated and cannot be used again. Would it be possible to add a callback for that? If not supported, it could be set to NULL.

pitrou · 2020-09-21T18:02:01Z

Would rewind go back to the start of stream always?

hannes · 2020-09-21T18:03:49Z

Would rewind go back to the start of stream always?

I would think so, no?

pitrou · 2020-09-21T18:12:46Z

Ok, I just wanted to make sure you weren't looking to go back a number of rows given as argument :-)

emkornfield · 2020-09-21T18:20:41Z

I think JDBC might in some cases allow for "scrolling" type of rewind vs restart (not saying this should support it but it is worth mentioning.

pitrou · 2020-09-21T18:22:24Z

Each environment (Python, C++, JDBC, etc.) has its own feature set for streams / iterators, which doesn't make our task easy :-/

emkornfield · 2020-09-21T18:24:29Z

Starting simple sounds good to me. At least for JDBC, I don't think it is required for driver implementations to support scrolling.

bkietz · 2020-09-21T18:50:55Z

Rewinding doesn't strike me as something which needs to be part of the C stream protocol. APIs can still provide rewind and other semantics while using a simple-as-possible stream as a building block.

In the case of a SQL view which can be viewed multiple times for example, let the SQL view need not be a stream. It could instead be a function returning streams (each beginning at the start of the view):

-ArrowArrayStream* MakeSqlView(...);
+SqlView* MakeSqlView(...);
+ArrowArrayStream* GetStream(SqlView*);

Rewind, scrolling, offsets, reverse iteration, etc can all be accommodated in this fashion so IMHO they don't belong in the protocol.

wesm · 2020-09-21T20:31:09Z

Another thing that occurred to me is whether we want to enable batch-level metadata (which would be implementation-defined). This is supported in Flight for example

https://github.com/apache/arrow/blob/master/format/Flight.proto#L316

The goal is to have a standardized ABI to communicate streams of homogeneous arrays or record batches (for example for database result sets). The trickiest part is error reporting. This proposal tries to strike a compromise between simplicity (an integer error code mapping to errno values) and expressivity (an optional description string for application-specific and context-specific details).

pitrou · 2020-10-01T09:42:15Z

I don't think it makes sense to continue hesitating on the API. I think I'm going to merge the PR as-is. The interface is marked experimental so can still evolve (or even be removed).

The goal is to have a standardized ABI to communicate streams of homogeneous arrays or record batches (for example for database result sets). The trickiest part is error reporting. This proposal tries to strike a compromise between simplicity (an integer error code mapping to errno values) and expressivity (an optional description string for application-specific and context-specific details). Closes apache#8052 from pitrou/ARROW-9761-c-array-stream Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

pitrou mentioned this pull request Aug 25, 2020

Support Arrow for table scans and result sets duckdb/duckdb#151

Closed

emkornfield reviewed Aug 26, 2020

View reviewed changes

hannes mentioned this pull request Aug 26, 2020

Initial Arrow support duckdb/duckdb#866

Merged

wesm reviewed Aug 26, 2020

View reviewed changes

pitrou force-pushed the ARROW-9761-c-array-stream branch 6 times, most recently from ec7c5aa to 1064c70 Compare September 1, 2020 14:35

pitrou marked this pull request as ready for review September 1, 2020 14:35

pitrou force-pushed the ARROW-9761-c-array-stream branch from 1064c70 to 04c25d8 Compare September 1, 2020 14:45

pitrou changed the title ~~ARROW-9761: [C/C++] Add experimental C ArrowArrayStream ABI~~ ARROW-9761: [C/C++] Add experimental C stream inferface Sep 1, 2020

pitrou force-pushed the ARROW-9761-c-array-stream branch from 04c25d8 to b71d965 Compare September 9, 2020 13:47

pitrou force-pushed the ARROW-9761-c-array-stream branch from b71d965 to 61e2c5d Compare September 21, 2020 15:18

pitrou added 6 commits September 30, 2020 20:12

Add C++ stream import API

f25feb2

Add Python wrapper and tests

b79e88f

Add C++ tests

565889e

Add docs

645406f

Add link to spec from C++ API doc

3badb80

pitrou force-pushed the ARROW-9761-c-array-stream branch from 61e2c5d to 9ade9a9 Compare September 30, 2020 19:39

Allow creating and exporting a RecordBatchReader from a Python iterable

87e8f73

pitrou force-pushed the ARROW-9761-c-array-stream branch from 9ade9a9 to 87e8f73 Compare September 30, 2020 19:44

pitrou closed this in 97879eb Oct 1, 2020

pitrou deleted the ARROW-9761-c-array-stream branch October 1, 2020 10:00

jorgecarleitao mentioned this pull request Oct 1, 2020

ARROW-10149: [Rust] Improved support for externally owned memory regions #8316

Closed

wjones127 mentioned this pull request Feb 20, 2022

FFI for Arrow C Stream Interface apache/arrow-rs#1348

Closed

jorgecarleitao mentioned this pull request Feb 21, 2022

Added support to the Arrow C stream interface (read and write) jorgecarleitao/arrow2#857

Merged

asfimport mentioned this pull request Oct 1, 2020

[C++] Add experimental pull-based iterator structures to C interface implementation #25812

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-9761: [C/C++] Add experimental C stream inferface #8052

ARROW-9761: [C/C++] Add experimental C stream inferface #8052

pitrou commented Aug 25, 2020 •

edited

Loading

pitrou commented Aug 25, 2020

github-actions bot commented Aug 25, 2020

emkornfield Aug 26, 2020

pitrou Aug 26, 2020 •

edited

Loading

emkornfield Aug 26, 2020

emkornfield Aug 26, 2020

pitrou Aug 26, 2020

emkornfield Aug 26, 2020

pitrou Aug 26, 2020

hannes commented Aug 26, 2020

pitrou commented Aug 26, 2020

pitrou commented Aug 26, 2020

wesm left a comment

pitrou commented Sep 1, 2020

pitrou commented Sep 9, 2020

emkornfield commented Sep 10, 2020

pitrou commented Sep 21, 2020

hannes commented Sep 21, 2020

pitrou commented Sep 21, 2020 •

edited

Loading

hannes commented Sep 21, 2020

pitrou commented Sep 21, 2020

emkornfield commented Sep 21, 2020

pitrou commented Sep 21, 2020

emkornfield commented Sep 21, 2020

bkietz commented Sep 21, 2020

wesm commented Sep 21, 2020

pitrou commented Oct 1, 2020

ARROW-9761: [C/C++] Add experimental C stream inferface #8052

ARROW-9761: [C/C++] Add experimental C stream inferface #8052

Conversation

pitrou commented Aug 25, 2020 • edited Loading

pitrou commented Aug 25, 2020

github-actions bot commented Aug 25, 2020

emkornfield Aug 26, 2020

Choose a reason for hiding this comment

pitrou Aug 26, 2020 • edited Loading

Choose a reason for hiding this comment

emkornfield Aug 26, 2020

Choose a reason for hiding this comment

emkornfield Aug 26, 2020

Choose a reason for hiding this comment

pitrou Aug 26, 2020

Choose a reason for hiding this comment

emkornfield Aug 26, 2020

Choose a reason for hiding this comment

pitrou Aug 26, 2020

Choose a reason for hiding this comment

hannes commented Aug 26, 2020

pitrou commented Aug 26, 2020

pitrou commented Aug 26, 2020

wesm left a comment

Choose a reason for hiding this comment

pitrou commented Sep 1, 2020

pitrou commented Sep 9, 2020

emkornfield commented Sep 10, 2020

pitrou commented Sep 21, 2020

hannes commented Sep 21, 2020

pitrou commented Sep 21, 2020 • edited Loading

hannes commented Sep 21, 2020

pitrou commented Sep 21, 2020

emkornfield commented Sep 21, 2020

pitrou commented Sep 21, 2020

emkornfield commented Sep 21, 2020

bkietz commented Sep 21, 2020

wesm commented Sep 21, 2020

pitrou commented Oct 1, 2020

pitrou commented Aug 25, 2020 •

edited

Loading

pitrou Aug 26, 2020 •

edited

Loading

pitrou commented Sep 21, 2020 •

edited

Loading