-
Notifications
You must be signed in to change notification settings - Fork 784
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FFI listarray lead to undefined behavior. #20
Comments
I have spent my weekend on this problem 😭 I now understand what we are doing wrong, but the solution requires a major design change of the FFI. This gist is that we currently treat children as independent from the parent and The C data interface requires (MUST) that the parent is responsible for all child deallocations. When we destruct a list array imported from C++, the following currently happens:
We should remove step 3 from this sequence, and let step 4 do its job. However, for this, we need to figure out a way to share a ref counted owned parent to all child arrays, so that when the last array is deallocated, we can release the parent and all associated child arrays on the same |
Oof, I feel you. Working on this depressed me as well.
Just thinking aloud here. Does it make sense to have some kind of marker (boolean) to indicate that an array is a child, and then make |
I solved this in arrow2 jorgecarleitao/arrow2#67 by using a different struct for an |
Ported from https://github.com/jorgecarleitao/arrow2 Fix apache#20 Fix apache#251 Signed-off-by: roee88 <[email protected]>
Ported from https://github.com/jorgecarleitao/arrow2 Fix apache#20 Fix apache#251 Signed-off-by: roee88 <[email protected]>
Ported from https://github.com/jorgecarleitao/arrow2 Fix apache#20 Fix apache#251 Signed-off-by: roee88 <[email protected]>
* fix: support nested types in FFI Ported from https://github.com/jorgecarleitao/arrow2 Fix #20 Fix #251 Signed-off-by: roee88 <[email protected]> * Removed Clone from FFI_ArrowArray Signed-off-by: roee88 <[email protected]> * Add nesting to FFI struct test Signed-off-by: roee88 <[email protected]>
* feat: Implement multi-part upload Co-authored-by: Raphael Taylor-Davies <[email protected]> * chore: simplify local file implementation * chore: Remove pin-project * feat: make cleanup_upload() top-level * docs: Add some docs for upload * chore: fix linting issue * fix: rename to put_multipart * feat: Implement multi-part upload for GCP * fix: Get GCS test to pass * chore: remove more upload language * fix: Add guard to test so we don't run with fake gcs server * chore: small tweaks * fix: apply suggestions from code review Co-authored-by: Raphael Taylor-Davies <[email protected]> * feat: switch to quick-xml * feat: remove throttle implementation of multipart * fix: rename from cleanup to abort * feat: enforce upload not readable until shutdown * fix: ensure we close files before moving them * chore: fix lint issue Co-authored-by: Raphael Taylor-Davies <[email protected]> Co-authored-by: Raphael Taylor-Davies <[email protected]>
* feat: Implement multi-part upload Co-authored-by: Raphael Taylor-Davies <[email protected]> * chore: simplify local file implementation * chore: Remove pin-project * feat: make cleanup_upload() top-level * docs: Add some docs for upload * chore: fix linting issue * fix: rename to put_multipart * feat: Implement multi-part upload for GCP * fix: Get GCS test to pass * chore: remove more upload language * fix: Add guard to test so we don't run with fake gcs server * chore: small tweaks * fix: apply suggestions from code review Co-authored-by: Raphael Taylor-Davies <[email protected]> * feat: switch to quick-xml * feat: remove throttle implementation of multipart * fix: rename from cleanup to abort * feat: enforce upload not readable until shutdown * fix: ensure we close files before moving them * chore: fix lint issue Co-authored-by: Raphael Taylor-Davies <[email protected]> Co-authored-by: Raphael Taylor-Davies <[email protected]>
* feat: Add stream upload (multi-part upload) (#20) * feat: Implement multi-part upload Co-authored-by: Raphael Taylor-Davies <[email protected]> * chore: simplify local file implementation * chore: Remove pin-project * feat: make cleanup_upload() top-level * docs: Add some docs for upload * chore: fix linting issue * fix: rename to put_multipart * feat: Implement multi-part upload for GCP * fix: Get GCS test to pass * chore: remove more upload language * fix: Add guard to test so we don't run with fake gcs server * chore: small tweaks * fix: apply suggestions from code review Co-authored-by: Raphael Taylor-Davies <[email protected]> * feat: switch to quick-xml * feat: remove throttle implementation of multipart * fix: rename from cleanup to abort * feat: enforce upload not readable until shutdown * fix: ensure we close files before moving them * chore: fix lint issue Co-authored-by: Raphael Taylor-Davies <[email protected]> Co-authored-by: Raphael Taylor-Davies <[email protected]> * fmt * RAT multipart * Fix build * fix: merge issue Co-authored-by: Will Jones <[email protected]> Co-authored-by: Raphael Taylor-Davies <[email protected]> Co-authored-by: Raphael Taylor-Davies <[email protected]>
Describe the bug
When sending an array array with child data over FFI (e.g. via pyarrow for instance) we encounter undefined behavior.
I have found SIGILL and SEGFAULTS
To Reproduce
The tests in this
arrow-pyarrow-integration
andarrow/src/ffi.rs
I ran the the tests in MIRI (great tool!), but I have to admit, I am stuck, and seem to go in circles.
The text was updated successfully, but these errors were encountered: