Implement leader election for the target allocator

[jaronoff97]

Right now, the target allocator's allocation strategy (least connection) means that you can only run a single TA pod at a time. This proves difficult if a consumer wants a high availability option for their target allocation. In order to make this possible, we could use the built in go leader election package for the target allocator.

The rough process that a collector does looks like this right now:

1. collector starts up
2. prom receiver loads configuration
3. for each job
4. http_sd_config queries target allocator
5. list of targets and metadata is returned from TA
6. prom receiver runs relabel_configs on targets
7. prom receiver scrapes targets remaining
8. prom receiver applies metric_relabel_configs
9. collector converts prom to otel
10. moves config to processor stage ...

If the target allocator is down at step 4, the job will fail (and most likely the entire the scrape config.) Adding in support for HA would improve the reliability of the statefulset collector.

If this is something the community would like, I would be happy to implement and test it.

[Aneurysm9]

Interesting suggestion. Improving the resiliency of the target allocation layer would definitely be a plus. Would the followers use the existing API to obtain state from the leader, or would you expect an active-passive setup with failover?

[secustor]

Yes, some kind of HA is definitely needed for TA.

Regarding leader election and state sharing I throw memberlist in to the ring. Grafana uses it in all its distributed products ( Mimir, Loki, Tempo, ... ), so it seems stable.

That way we can simply point the collector to TA using a service, as we do now and don't need to think about manual failovers.

[jaronoff97]

@Aneurysm9 I was expecting an active-passive setup with failover so as to not complicate any of the existing logic. @secustor using memberlist, would we instead do state sharing, or would we just use that to determine who the active is?

[jeromeinsf]

@secustor memberlist might need to be adapted to get to a propagation delay adapted for an HA expectation.