Better HDFS Support

[echeipesh]

This PR improves HDFS layer writing and random value reading support.

HadoopValuesReader now opens up MapFile.Readers for each map file in the layer. These readers cache the index of available keys so they are able to provide quick lookups. This replaces and improves previous method of using FileInputFormat to query for a single record.

HadoopRDDWriter has multiple improvements:

• Accomplishes it's task with a single shuffle step
  • This is possible because instead of estimating the blocks required by counting we simply roll over to a new file when the record being written is about to surpass the block boundary.
  • Instead of using a groupBy we sort our records on shuffle which uses IO index to partition the records
• Writes happen through mapping the partition iterator, this is possible because the partition comes pre-sorted and greatly reduces memory pressure as records can be garbage collected after they are written.

Current testing shows:

• HDFS ingest speeds exceed those seen in Accumulo
• tile fetching latency is visually acceptable
• that jobs using the GroupedCumulativeIterator approach are able to complete in settings where .groupBy/sort/write jobs are killed for memory violations.

Future improvements that on the mind:

• Save bloom filter index for layer MapFiles to reduce memory requirements for HadoopValueReader and to produce quicker lookup in most cases.
• In HadoopValueReader when fetching a record, cache all the tiles that share that index before filtering down to a single tile. These records are very likely to be asked for next and should be stored in LRU cache.
• Devise a method to compare KeyIndex instances such that if the RDD to be saved is already partitioned by compatible index (where all key boundaries are shared) to avoid the extra shuffle.

[pomadchin]

how it would effect query time? (just curious)

[echeipesh]

Layer query time would not be effected at all by this. When we're reading off ranges we're already seeking through the file, so not having as many points would have minimal impact if any. This is going to have more of an impact on random access through value reader. Spinning up a cluster to figure that part out. For reference the default interval value is 128

[lossyrob]

bit => big

[lossyrob]

+1 after comments addressed