Support reading and writing custom IPC metadata

[nameexhaustion]

Description

The IpcReader should have a function that returns custom metadata stored in the file, and the IpcWriter should have an option to set custom metadata to be written.

[PrettyWood]

@nameexhaustion @alexander-beedie @ritchie46 Following up on #18527 (comment) we would really need to keep those metadata
And we also would love to be able to use the latest polars release 😅
@lukapeschke or myself are ready to work on this if needed. We just need to have your insights on the desired API

[nameexhaustion]

Hi @PrettyWood

Thanks for the offer to help! We haven't had the time to look at this yet due to how busy we've been.

For some background, it was removed from the arrow schema struct because we internally didn't want to carry custom metadata around on it. This is still the case - but we'd be happy for there to be functions on the IpcReader and IpcWriter structs themselves that allow for reading / writing custom metadata.

For the implementation, I would start with making something similar to this -

struct IpcReader {
    fn custom_metadata(&self) -> &Arc<PlHashMap<PlSmallStr, PlSmallStr>> {}
}

struct IpcWriter {
    fn set_custom_metadata(&mut self, custom_metdata: Arc<PlHashMap<PlSmallStr, PlSmallStr>>) ->  {}
}

It could also be helpful to look at some of the deleted code in https://github.com/pola-rs/polars/pull/18527/files.

I believe IPC supports message-level metadata, but for the initial PR, we should restrict to only support writing file-level custom metadata

[lukapeschke]

Hi @nameexhaustion thanks for the hints! I've started working on this, and I noticed that the async IPC stream sink was gone as well since #19223 .

We were using it, and would like to keep using an async writer. Is this something you'd be willing to accept if we worked on it ?

[lukapeschke]

@nameexhaustion would something like #20066 work for you ? If yes, I'll add a few tests

[nameexhaustion]

Hi @nameexhaustion thanks for the hints! I've started working on this, and I noticed that the async IPC stream sink was gone as well since #19223 .

We were using it, and would like to keep using an async writer. Is this something you'd be willing to accept if we worked on it ?

Hello,

I won't be able to give you an answer for this one. From what I can tell, it was removed due to not being worth the maintenance burden as we were not using it. I think it would be better to consult with Ritchie for this matter.