Use flatbuffers to serialize dtypes #126
Conversation
|
We should discuss the aim of this PR. Currently it uses Flatbuffers for DType serde, which kind of defeats the point of flatbuffers? (Not that it should block, just that we should figure out how we want to design our code to make more effective use of flatbuffers) |
|
I think the point of this pr is to standardise the serialisation, ie use a supported framework instead of custom binary serialisation. Hopefully this code can be easily adapted to fit into overall serialisation process. |
vortex-schema/src/serde.rs
Outdated
| fn serialize(&self) -> Vec<u8> { | ||
| let mut fbb = FlatBufferBuilder::new(); | ||
| let wip_dtype = self.write_to_builder(&mut fbb); | ||
| fbb.finish(wip_dtype, None); |
There was a problem hiding this comment.
You can use finish_minimal instead of passing None, does the same thing internally
vortex-schema/src/serde.rs
Outdated
| let mut fbb = FlatBufferBuilder::new(); | ||
| let wip_dtype = self.write_to_builder(&mut fbb); | ||
| fbb.finish(wip_dtype, None); | ||
| fbb.finished_data().to_vec() |
There was a problem hiding this comment.
This trait design seems to force this copy, instead of just writing the finished slice into whatever we're serialising to.
There was a problem hiding this comment.
Yes, I’ve done the dumb thing but agree we should avoid it. I think we can pass a writer here
No description provided.