Rules/Alert storage improve store performance and architecture

[jtlisi]

I opened this thread as a place to start a discussion about how we can improve our alertmanager and ruler storage backends. Currently those backends suffer from high latency at scale. This is caused by the following factors:

• Cortex uses list operations throughout the storage backend since we don't have a reliable index.
• Cortex is unable to cache rule group objects since we mutate objects in the object storage backend and don't propagate those changes as messages.

Possible Solutions:

• Index: An index file could help reduce the number of list operations performed by the ruler storage backend. However, the ruler API allows multiple concurrent writes and maintaining an index on object storage is difficult without a coordinating mechanism (locking, leader election, sharding). Below is a crude example of issues that could occur with concurrent writes to the same index file:

--------------TIME-------------------------->
OP1(Delete Key1) ----> Index 1 (Key2)
    OP2(Delete Key2) -----> Index 2 (Key1)

Possible coordinating mechanisms for managing an index:

• Ring (Leader-Election): The simplest form of coordination would be to instantiate a Ring with a replication factor of 1. Then we would hard code a token value and the API in the ring responsible for that token is the leader. All HTTP requests will then be forwarded to the leader to ensure a single write to object storage and avoid concurrent writes to the same key (specifically the index). For gossip this will require significant tuning to ensure we have a stable ring before allowing writes.

• Exposing CAS primitives for Object Storage: GCS and Azure support conditionals that would allow an index to be maintained and protected from concurrent writes. However, S3 still does not. If we support an additional Locker (DynamoDB, see Terraform for example) for S3 and use the native implementation for other solutions, we could implement this using the object storage backends alone.

• KV Leader Election: Instead of reusing the ring lifecycler code for leader election we could implement a proper leader election algorithm on top of the KV interface. This would allow us to be more efficient with the election process and ideally expose more relevant and sane configuration options for tuning.

• Log Device (Virtual Consensus): Each rules API could maintain a log of events persisted to their API and persist this log to S3. This requires a unique ID for each rules API instance. These logs are only mutable by the agent that created them. Replaying these logs will allow each instance to build an index in real time of the state of the rules store backend. This is a bit of a TLDR but this idea was taken from this paper

Existing Questions:

• What is our availability target for the Alertmanager/Rule Storage APIs? The type of solution that works best is different between 99% and 99.999% availability and have a good understanding of this will help us choose the best solution.

[pstibrany]

• Designate single ruler as responsible for updating index for a user (via ring, as usual) – this is how compactors coordinate updating of bucket index.

• Make configuration files immutable (by giving them unique filename, either based on timestamp or content), and cacheable. It doesn’t solve “listing” problem, but it would help to avoid reloading of config files that were not changed, and make loading of changed files faster by using shared cache mechanism. [Discussed in the past, was never implemented]

[jtlisi]

If I'm not mistaken you wrote a design document suggesting this at some point?

[pstibrany]

Yes, I did, just never got to implementing it :(