Investigate Latency Contributors

[darunrs]

One of the goals of QueryApi is to speed up historical backfill processes. In order to do this, we first need to fully understand the contributors to latency in the current system. This issue tracks the progress to describing the latency contributions various parts of the system contribute. Most of the analysis takes place in Runner as it is responsible for the speed of the backfill.

In addition, some investigation into scalability will be done and some recommendations for metrics to include will be included in a document to go over the discoveries.