Investigate using bgzipped and indexed fasta as alternative to 2bit, and standardize on a standard reference format for spark tools

[akiezun]

the 2bit fasta isa bit of a pain to deal with. #1580 shows that size-wise bgzip fasta is comparable.

This ticket is to:

• establish whether we can use a bgzipped+index fasta as a reference (htjdk support and speed)
• if it works for walkers, evaluate performance of distributing the reference+index to spark executors and using it as a replacement for 2 bit

[SHuang-Broad]

@akiezun, when you say "reference+index" do you mean .fai only or other index files like the ones required/generated by bwa like .bwt index files?

[droazen]

@akiezun Since 2bit format has a more compact in-memory representation than fasta (~4x smaller), you'd need to document the effects on Spark memory usage as well.

[akiezun]

it would be a whole bunch of work though. htsjdk does not know about bgzipped fastas nor gzi index files. I'm looking for a way to estimate the benefits without doing the work yet. Suggestions welcome

[akiezun]

Moving this past alpha2 - requires substantial development in htsjdk

[droazen]

Reassigning to @jamesemery

[droazen]

Untagging for alpha-3, since we're going to try #2074 first as a cheaper alternative. Keeping this as an alpha3-candidate

[tomwhite]

We no longer require 2bit for Spark: #5127