QueryFrontend: HTTP/gRPC client load balancing

[bwplotka]

It would be amazing to use spitting mechanism better and allow distributing requests to multiple querier replicas.

Most of us use Kubernetes with K8s Serice only which are TCP based, so they don't do round robin balancing - they just pick one. Some of users don't have good lb handy even. I think it should be application logic to make that happen.

I already wrote similar code e.g in Kedge which works on production even now (https://github.com/improbable-eng/kedge/blob/772f9b2d2092a0ada972096945bee8cd49513da6/pkg/kedge/http/lbtransport/transport.go#L104), HOWEVER, it might be a better idea to use grpc rather. This is more tricky though but gives us support for load balancing on Querier towards other APIs as well. So we have two choices (or implement both):

a) http

Pros:

• We have code for it ready
• Query endpoints are in HTTP already.

b) grpc

Pros:

• LB for query frontend against just Prometheus is not needed
• We need these features for Querier -> Store/Rule/TargerAPIs as well anyway.
• We can implement richer metadata passing that will allow to loadbalance based on saturation(!)

Cons:

• There is some sketch of gRPC client LB code in gRPC ecosystem, but this is changing every release, so work has to be done.
• We need to expose Query APIs with gRPC and switch to gRPC port.

In both cases acceptance criteria are the same:

AC:

• Query frontend replicas can route to multiple queries and have different load balancing strategies (Round Robin for a start)

I would vote for B (: More ambitious and more benefits. Thoughts? @pracucci @yeya24 @brancz @kakkoyun

[kakkoyun]

I think we should immediately go for option a just because it's a low hanging fruit. And then we can devise a plan to implement option b.
I'm with you the ideal solution would be the option b.

So let's just make it work and then we can iterate and optimize it.

[brancz]

I agree with @kakkoyun.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[hitanshu-mehta]

I think I can work on this issue with some guidance. Can you please assign me if nobody is working on this?

[hitanshu-mehta]

I have one question regarding first part of AC, i.e.

Query frontend replicas can route to multiple queries

What approach everyone suggests? Should it be similar to cortex? ( which is as far as i understood 😅 , add a flag in querier -querier.frontend-address to connect it with frontend and querier will pull request from the frontend queue. )

[roidelapluie]

if you go for http, would it be something reusable for prometheus/prometheus#8402 ?

[bwplotka]

@roidelapluie Yes 🤗

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[kakkoyun]

Still valid.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[hdost]

Still valid.