Output formats

[brainstorm]

As raised earlier, we should decide on a desirable (subset of?) output format(s). The premises/deciders for them are:

1. Cloud native: Popular enough so that commercial cloud vendors have adopted it.
2. Mature: Open source libraries have adopted it and enjoys good community support.
3. Performant: Should be within the top N in filesize and operations performance.

Here are a few candidates:

Apache Arrow and/or Parquet

Might be the strongest candidate due to its (Rust) library support and having prior art being used in other sequencing tech such as nanopore. It is also a cloud native format, no expensive ETLs are needed to consume it.

Apache Avro

One of the formats supported by default by AWS Athena.

Amazon Ion

Relatively new format that has been opensourced by Amazon. Its dual text and binary representation capabilities are interesting as well as its Rust bindings and compatibility with the AWS ecosystem (i.e Athena can consume it).

Apache ORC

Still cloud native and has some interesting compression and runtime improvements when compared with Parquet. On the other hand, Rust library support might not be as mature as with other formats.

GenomicSQLite

Has a fair amount of work done and can be interesting to benchmark and interface with. Interestingly, it even has a Rust binding.

D4

Highly specific around BED/BigWig so input-to-ouput format mapping might not be even be feasible for a number of input formats.

insert and discuss further suggestions here

[brainstorm]

@GabrielSimonetto What would be your take(s) on this discussion? Which output format would you choose for now?

[GabrielSimonetto]

I am happy with starting with Apache/Feather/Arrow, mostly because I have zero experience with any of the other options. I have mixed feelings with the parquet crate (as used in the bed_to_parquet_conversion demo), because the API is really not approachable to customization and tweaking, you can only work with a few high level functions.

The only two ways I can see this being a problem in the future, is if the crate cannot generalize well enough for all of the desired formats, or if we are unable to enhance performance because of the high level functions, both seem unlikely. That being said, using a familiar technology should boost our development speed, which is nice.

[brainstorm]

Thanks for the feedback Gabriel! I was a bit partial to Parquet as it's supported by default on AWS's Athena. But you raised a fair point about the high level functions and their flexibility.

Could you estimate how hard/easy would it be to modify the upstream crate (parquet/parquet_derive) to suit our needs better? It's not the first time that a GSoC project needed such dependency contributions, they are valuable as they help an even broader set of users (i.e htsget-rs did require modifications on the upstream noodles-* crates).

I've added Avro on the above list, by the way.

[multimeric]

I think you should be able to decently customize it if you use the low level APIs: https://docs.rs/parquet/15.0.0/parquet/column/index.html#example-of-writing-and-reading-data. The main parameters you will need to tweak will be the column types you use, and the compression, maybe some other minor things.

[GabrielSimonetto]

@brainstorm it's good to know that tweaking the crate itself is not out of the question, that could help a lot at some point. As we develop the first POC I will keep an eye on it

@multimeric that's a good point, I somehow missed the column API and was thinking about some test cases I tried to make on the bed demo. Do you think using column would be a problem in the sense that it wouldn't be powered by arrow? I need to understand better how/why arrow works, but I was under the impression that it packs a lot of optimizations for big data; also, it is even possible that you can build columns by yourself and then pass it to arrow.

[multimeric]

fwiw Apache Feather is the name of the Arrow IPC format.

I'm also wondering if we should benchmark some other formats that are maybe not obvious HDF5, SQLite (e.g. https://github.com/mlin/GenomicSQLite).

Finally, I wonder if we should rank these from most important to least, so we can easily extend the project to more formats if we have time.

[brainstorm]

fwiw Apache Feather is the name of the Arrow IPC format.

Ah, so you mean we should also perhaps consider Feather instead of Protobuf? Sorry if I misunderstood.

(...) rank these from most important to least

Indeed, I ordered them already by my own perception of priority, but please @multimeric @GabrielSimonetto, let me know if you want it re-ordered/changed (or even edit the first post directly).

I'm also wondering if we should benchmark some other formats that are maybe not obvious HDF5, SQLite (e.g. https://github.com/mlin/GenomicSQLite).

I'm a bit skeptical about HDF5 and SQLite, to be honest, I'm not sure they'll perform nor integrate as well as the other contenders... if I had to put them on the list, that'd be below ORC perhaps? I'm saying this based on personal experience with both, which might not be the best metric/decider.

[multimeric]

Ah, so you mean we should also perhaps consider Feather instead of Protobuf? Sorry if I misunderstood.

No, I just meant that Apache Arrow isn't a format, but Feather is.

The order looks fine, assuming D4 is viable.

I'm happy to put those two last (I'd put the SQLite one slightly higher because the github repo I linked means it's been setup and tested already), but I think there is merit in trying quite diverse formats. Perhaps I'm thinking of this too much like a scientific paper, but I wouldn't be satisfied if a saw a benchmark that only included <old format known to be bad> vs <new format we proposed> without <intermediate format that someone proposed last year>.

[brainstorm]

Copy that for GenomicsSQL, it'd be very interesting to compare notes. Also, since we are in GSoC so-called "community bonding" mode, that's a work made by @mlin, who's also part of GA4GH, hello Mike 👋🏻. @GabrielSimonetto might ping you at some time in the future about it ;)

/cc @ohofmann

[ohofmann]

D4 is viable - and it's a good fit, I was discussing with @arq5x how to better support it in the context of GA4GH a while ago.

[brainstorm]

Correct me if I'm wrong folks, but I think that when we said "viable" we were referring to having the output format realistically model/map all input formats (*AM, VCF) with some reasonable performance guarantees... I was under the impression that D4 is designed/tailored for variant data only whereas the other formats (i.e Arrow) are more generalistic?

[multimeric]

Yeah it's a tricky dilemma, because there isn't necessarily one globally optimal format. But for the scope of this project I think we have to make the simplification that we want 1 output format for all input formats, and that we only consider tabular input formats. So if D4 is invalidated by those rules then we should cut it.

[brainstorm]

Good read about some of the output formats we are targetting: https://blog.openml.org/openml/data/2020/03/23/Finding-a-standard-dataset-format-for-machine-learning.html