Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Add support for the buffer protocol #227

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

mloebel
Copy link

@mloebel mloebel commented May 18, 2020

This adds initial support for implementing the buffer protocol via py_class! {}.

This works by defining a magic method:

def __buffer__(&self) -> PyResult<MyBuffer> { ... }

where MyBuffer has to implement the following trait.

pub unsafe trait BufferHandle: 'static + Send {
    /// Returns the data of `Self` as a continous array of bytes.
    ///
    /// # Safety
    ///
    /// This needs to return an address that remains valid
    /// and stable if `self` gets moved or transformed to or from a void pointer.
    fn as_bytes(&self) -> &[u8];

    /// Convert `self` into an owned void pointer.
    ///
    /// # Safety
    ///
    /// The returned pointer has to be a valid pointer that
    /// can be converted back to `Self`.
    fn to_owned_void_pointer(self) -> *mut libc::c_void;

    /// Convert an owned void pointer back to `Self`. This takes owenrship of the pointer.
    ///
    /// # Safety
    ///
    /// The passed `ptr` has been created by this trait, and is only
    /// used at most once to convert back to `Self`.
    unsafe fn from_owned_void_pointer(ptr: *mut libc::c_void) -> Self;
}

Remaining issues

  • Does not work for Python2.7 for reasons I don't understand yet. The support for the "new" python 3 buffer interface exists in 2.7 with all the right type definitions, but the tests fails at runtime with "cannot make memory view because object does not have the buffer interface", so I've disabled it there for now. Help would be appreciated with this, as I'm not actively using 2.7 (which is eol, besides 😅).
  • Only supports exporting a read-only byte array view for now.
  • The BufferHandle API does not allow rejecting a buffer request.
  • The BufferHandle API does not allow keeping ownership of the memory inside the class, as I was not sure if that can be safely done.
  • Need to add docs for the magic method.

This partially fixes #190

@mloebel
Copy link
Author

mloebel commented May 25, 2020

Update: Turns out that some of my use-cases for the buffer API use PyArgs weird definition of what a "read-only bytes-like" object is: Something without a destructor for created PyBuffers.

In practice this means that the buffer memory exported by the object has to be manged by the exporting object itself, and needs to be immutable for the entire lifetime of the object.

I tried to model this by adding an alternative magic method that expects this signature:

for<'a> fn __direct_buffer__(&'a Self, Python<'a>) -> PyResult<&'a [u8]>;

...with the idea that you can only implement this by returning references to data members or static memory. As far as I understood, data members are allocated in a stable memory location as part of the PyObject created by py_class!, so retrieving and accessing the &'a [u8] should be valid at any point as long as the object is alive. And by requiring it to be a direct reference to self, we prevent mutation through internal mutability wrapper.

…s.rs specific data structures into the same module.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

create PyBuffer from rust code
1 participant