Queries over long time ranges balloon index usage

[robskillington]

Reported by Yoram Hekma @yhekma

Queries over a long range that return a significant number of IDs (just a single time series seems to be fine) greatly balloon the memory usage of an M3DB usage.

Query (over 14 day range):

sum(rate(nginx_http_requests_total{host="$service"}[1m]))

Estimate of number of time series being returned:

3196

Setup:

2 namespaces, 1 unaggregated with 96h retention and blocksize of 4h, and one aggregated with 336h retention and blocksize of 24h

Memory usage ballooning:

Heap profile flamegraph shows 16gb consumed in alloc_space for QueryIDs:

Heap profile:
https://www.dropbox.com/s/viiy934bp67zix8/heap.prof?dl=0

[favoretti]

I'll be listening in on this, as @yhekma and myself are on the same team. Thanks for looking into this folks!

[richardartoul]

@favoretti @yhekma

We just tested an index performance improvement that resolves this issue and saw order of magnitude improvements, so we will have a fix for this soon.

We need to contribute code to one of our dependencies (you can follow along in this issue if you like) so it may take a week or two, but this will be resolved soon.

[yhekma]

Awesome! Thanks bunches!

[richardartoul]

@yhekma We still have a few more large index performance P.Rs coming in, but I just merged #1238 to master which should really help with the issue you were seeing. If you have a chance to test out master let us know how it looks!

[yhekma]

@richardartoul great! I'll test it out tomorrow and will let you know. Thanks again

[yhekma]

@richardartoul wow, that make a BIG difference! I think this means this issue can be closed

[richardartoul]

@yhekma Great! @robskillington Has some more very large index performance improvements coming soon so stay tuned.