Scale: Enabling leader election or Multiple EG writing status back for same resource.

[Xunzhuo]

Description:

Describe the desired behavior, what scenario it enables and how it
would be used.

Envoy Gateway disabled leader election at default and didn`t expose it through envoy gateway config. When scaling cp replicas, Multiple EG will be writing status back for same resource.

We need to find out which is more expensive - enabling leader election and EG sending heartbeats to API server or multiple EG writing status back for same resource.

cc @envoyproxy/gateway-maintainers

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[zirain]

Envoy Gateway disabled leader election at default and didn`t expose it through envoy gateway config.

this suprise me, IMO, EG should enable leader election.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

imo we should enable leader election (by default) to make sure only 1 EG controller can

• write status back
• can create/write infra (envoy proxy fleet & rate limit service)

but we need to make sure all EGs can read/watch resources and generate resources so any Envoy proxy can connect to it to get xds to support data plane scale out

cc @Xunzhuo can you help with this since you raised #2123 :)

[alexwo]

Hi, I'm interested in contributing.
Could you please assign me to this issue?

[alexwo]

Greetings,

I would like to share a draft proposal that I've been considering for this enhancement. Below is an overview and a suggestion for a phased approach I propose we can take to integrate this feature smoothly, minimizing potential disruptions.

Phase 1: Foundation for Leader Election

• Leader Election Mechanism: We'll integrate a leader election mechanism aligned with Kubernetes standards. This will identify and designate a leading EG pod to manage the xDS service, status updates, ensuring a single source of truth for configuration updates.

• Flexible Configuration: To cater to diverse operational needs, we can introduce configurable parameters for managing the leader election process. These can include options for adjusting lease durations, renewal intervals, and retry intervals, with Kubernetes' default settings as our fallback.

• Readiness Probes Adjustment: To reflect their status in the leader election hierarchy, we'll modify the readiness probes of our EG pods. Non-leader pods will be marked as "not ready" to prevent them from serving xDS requests, thereby maintaining configuration consistency and preventing potential misconfigurations for envoy proxies connected.

Phase 2: Expanded xDS Service Capability

Once we've established a stable leader election process, our next step can be to enable all replicas to serve the xDS service effectively over gRPC. This phase can focus on:

• Ensuring Envoy Proxy Routing Consistency: It's crucial that each Envoy proxy receives updates from the same xDS instance to avoid configuration discrepancies. By maintaining a consistent connection to a designated Envoy xDS service endpoint, we can achieve this uniformity.

• Multi-Replica xDS Service: We can introduce a new configuration option to allow xDS service distribution across multiple replicas. In this mode all xDS replicas are synchronized and updated for each resource change, with only the master replica handling the k8s resource status updates.

Feedback:

I am keen to hear your thoughts, insights, and any concerns you might have regarding this proposal.

[arkodg]

thanks for picking this one up and detailing your plans !

approach LGTM, my suggestion would be to not spend any extra efforts on Readiness Probes Adjustment and disable the other replicas , and instead invest it directly on Multi-Replica xDS Service ( which is how its designed today where kube-proxy is helping us spread the load today to different xDS servers ).
Only after 1 & 2, can we recommend users to confidently scale EG replicas

[alexwo]

@arkodg That sounds like a plan. We can move forward with implementing a comprehensive solution for xDS right from the start.

Is it still be beneficial to have an option for a deployment where only a single xDS instance is active & supported by a standby replica ?

This could offer better consistency and reliability, especially in scenarios where synchronization issues might lead to a replica becoming outdated.

[arkodg]

@arkodg That sounds like a plan. We can move forward with implementing a comprehensive solution for xDS right from the start.

Is it still be beneficial to have an option for a deployment where only a single xDS instance is active & supported by a standby replica ? This could offer better consistency and reliability, especially in scenarios where synchronization issues might lead to a replica becoming outdated.

we could consider Active/Passive control plane replica as Phase 3 :) , as an opt in, would be good to create a sub issue and get community feedback

[alexwo]

@arkodg Yes, sounds like phase 3 :) 👍

[alexwo]

I have prepared a PR, it's ready for review.

This use cases were manually validated:

1. Blocking the GRPC Port of a Leader (with IP Tables Rule)
   Result: the Envoy proxy is able to discover the xDS rules from at least one of the remaining EGs.

2. Blocking the GRPC Port of the leader + all secondary pods except one
   Result: Despite the widespread blockage, the Envoy proxy manages to obtain xDS rules from the remaining unblocked EG, maintaining operational resilience.

3. Apply k8s quickstart ( confirm only leader update status / infra)
   Result: status updates and infrastructure are exclusively managed by the leader controller.

4. K8S control plane API is not available for the leader instance
   The leader restarts, however, other instances continue to serve xDS and take leadership when possible

Best Regards,
Alex

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

completed with #2694