Does arrow2 support codecs when reading IPC files?

[ghuls]

Does arrow2 support codecs when reading IPC files (Feather v2)?

Feather v2 files created by pandas will use lz4 compression by default. arrow-rs crashes on this kind of files (for now).
Does arrow2 support reading IPC files with a compression codes (lz4, zstd).

apache/arrow-rs#286

[jorgecarleitao]

Not yet. I will take a look and check how difficult it is to implement.

[ghuls]

At least checking if IPC files are using a compression codec would already be a good start even if actually decompression is not supported yet.

[jorgecarleitao]

@ghuls , there is now a PR for it, #168 :)

[ghuls]

Wow, that was quite fast. Any chance that is also will end up in arrow-rs (apache/arrow-rs#286)? Depending on how fast polars switches to arrow2, it might be less of an issue (although now arrow-rs will not see that the IPC stream is compressed and will try to read at the wrong locations, resulting in weird error messages, so at least detection of compression would be good to have there too.