Write to parquet

[jsignell]

Closes #72

I'm pretty sure this is not what you had in mind, but wanted to put this up as at least an option. Even if we scrap the implementation, the test should stay the same.

I didn't include remote writing since it seemed like that would be something that we'd want to implement for all the file outputs not just for one.

[TomNicholas]

What does "raw" mean in this specification?

[jsignell]

I think it means "inlined" in the kerchunk library tests this ref: "a/6": b"data", results in a non-null "raw"

https://github.com/fsspec/kerchunk/blob/a5eae4d45f7f601346d0fdc58c0864911330c5d6/kerchunk/tests/test_df.py#L62

[TomNicholas]

Thank you @jsignell ! That was quick 😁

I'm pretty sure this is not what you had in mind, but wanted to put this up as at least an option. Even if we scrap the implementation, the test should stay the same.

Agree that's useful already! I think this implementation is useful to start with, and tests ensure we can change it later without breaking behaviour.

I was wondering if there is a more performant way to go from ManifestArray to parquet, but that implementation will depend on how the references are stored in-memory, so should wait until #107 is merged.

I didn't include remote writing since it seemed like that would be something that we'd want to implement for all the file outputs not just for one.

👍

[TomNicholas]

@jsignell to merge this we should have some docs (on the usage page).

It also might be worth checking that round-tripping works - see the in-progress PR #42.

[jsignell]

I just stole the easiest test off that PR for roundtripping. See what you think

[TomNicholas]

This looks good to me! Let's get it in and iterate if necessary!

Thanks @jsignell !