-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-7624: [Rust] Soundness issues via Buffer
methods
#6397
Conversation
Would it be helpful for me to review this? At first glance, I noticed another issue. (The I do have one question before reviewing in more detail. I'm confused about the ownership semantics of |
Absolutely, that would be great!
It is an owned block of data and I would say that |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@paddyhoran Yes, you are right. LGTM. |
Reading over the code, I found a few more issues in
I have a work-in-progress set of changes on the |
@jturner314 A few of those issues are also fixed in #6395. Started going through buffer a bit over the weekend. |
Thanks @jturner314 and @Marwes. I need to take a look at the issues around allocation (the first 3 points above).
Actually, I raised this issue when this was merged. This is expected, the higher level array types allow for the
So, we pad to 64 bytes in Rust. This is mentioned in the spec. This allows us to use SIMD easily, basically we allow SIMD to operate on the padded region so that's why the padded region is included in Anyone have any issues with merging this and opening follow up JIRA's for the other items? |
I just noticed another issue -- the Since you're depending on padding outside of the length for use in SIMD (which means the padding needs to be initialized), perhaps the simplest solution would be to always get initialized memory (replace calls to
Thanks for pointing out that PR. It looks like it fixes most of the issues related to null pointers in the buffer types. The remaining thing I see is that the
Okay, that's reasonable.
Padding to 64 bytes is fine. There are a couple of things to be careful about, though. The first is that if code accesses the padding, e.g. for SIMD, it needs to be initialized. The second is that when the memory is freed, it needs to be freed with the same layout it was allocated with (the "layout" is the size and alignment). The
Sure, that's okay with me.The remaining issues can be fixed in other PRs. |
@jturner314 does #4331 address the de-allocation issue? I opened this JIRA for the initialization issue. |
@paddyhoran Yes, #4331 fixes the issue I was describing regarding |
No description provided.