Memoryview serialisation

[martindurant]

Fixes #3640 (for discussion)

Does not create an intermediate bytes object for serialization, passes through the memoryview. That, of course, still gets sent as bytes on the wire, so for deserialization, we just wrap it.

[mrocklin]

+1

[TomAugspurger]

Is it worth special casing length-1 frames, since then you can just return frames[0] and avoid the copy? Maybe?

[martindurant]

Good point

[TomAugspurger]

Looks good other than the one question.

[jakirkham]

Thanks Martin!

So one thing I'm curious about is there's a fair bit of logic in NumPy serialization, which we seem to be skipping here. For example it is possible to have Python objects much like one can in NumPy arrays. So it might make sense to pickle in that case. Similarly we may want logic to flatten N-D data, cast larger types, and split large frames. Though we don't need to worry about datetime or timedelta objects as they are not supported types in memoryviews. Given this, what do you think about repurposing this logic for memoryviews and allowing NumPy serialization to just piggyback on memoryview serialization?

[martindurant]

t is possible to have Python objects

I am surprised to find that memoryview.format == "O" is indeed allowed. Does this actually ever happen? I suppose this just failed previously, or it never ever happened, but I am not sure.

I think the same goes for complex types and multiple dimensions - those should be already using numpy. I daresay that in every case here, we have a bytes-like thing (from arrow, whatever) that we don't want to accidentally copy.

In short, I am not keen to change the numpy side of things, even if it just amounts to moving code around a little, to support memoryviews that I'd be surprised if they got used. Prepard to be proven wrong...

[jakirkham]

I think this goes back to the fact that the Python Buffer Protocol allows things like arrays of pointers. This was to make PIL developers happy with the PEP IIRC. So memoryview.format == "O" is one such case as one would have PyObject* in the array. In practice, this might come up when serializing a column of strings? 🤷‍♂️

Was mostly thinking we could get more out of the same code. Though you may be right this seldom comes up in practice.

Yeah I guess we wait and see if there's an issue. 🙂

[martindurant]

In practice, this might come up when serializing a column of strings?

Since pandas and numpy already have their own serialisation, someone would really have to be trying to break Dask :)