Poor performance for tag completion endpoints

[arnikola]

Users have been reporting poor performance when hitting any of the tag completion endpoints in m3query and m3coordinator; these requests are slow and often OOM clusters.

The root cause of this is that the M3DB client does not expose an endpoint to list available tag name/pairs, and m3query must make do by fetching all IDs matching the request then deduping them. Essentially, this means that trying to get a list of available tags pulls every single series into memory within M3DB, returns it up to Query, and then processed to return the list of valid tags.

This issue affects Grafana templating, Graphite find endpoints, and any tag completion features.

This issue will track work to build and expose a tag completion endpoint in M3DB which will greatly increase the performance, and reduce the memory overhead required to perform these requests.

Most of these are being done independently but there's a few dependancies which will be called out:

• Hook up the new endpoints m3query side ([query] Plumb potential tag completion/aggregate through to m3query endpoints #1481)
• Add Thrift RPC types for the client ([dbnode] Add rpc/AggregateQuery types #1460)
• Add the endpoint on the client session side (dbnode Aggregate RPC implementation #1483)
• Push down a single Results object into the query and merge all individual block postings lists into that, rather than allocating a Results per block then merging them together (Use a single index Results when querying across blocks #1474)
• Build the logic on the index itself ([index] Aggregating results on storage side #1463)
  • (this will be refactored to look more similar to Use a single index Results when querying across blocks #1474 above)

[evanhuang996]

How is the progress?

[arnikola]

Hey, will add a checklist to the issue post for slightly better tracking. Most of these have PRs close to landing; there'll be a little refactoring work to ensure our interfaces match on all sides and some amount of smoke testing required, but it's looking close.

[arnikola]

Resolved in 0.8.0 👍