Non uniform connection distribution to Envoy worker threads

[ahmedelbazsc]

Title: Enhancing workload distribution to worker threads

Description:
We have a gRPC client server setup which uses Envoy as a sidecar proxy for egress and ingress calls as follows:
gRPC client -> client Envoy -> Server Envoy -> gRPC server
With Envoy used on the egress path, we observed degraded (and inconsistent) client throughout which appears related to the connection distribution of gRPC client over Envoy worker threads.

Connection Pooling:
To increase throughput we attempted pooling connections from the client to the local Envoy sidecar which demonstrated throughput improvements, but results were inconsistent across different runs. We saw maximum throughout when all Envoy worker threads are handling incoming connections but it appears there are no guarantees on how incoming connections are distributed to workers.

To confirm this, we attempted to regularly refresh connections in the pool, as opposed to keeping long persistent connections, which shows consistent best results with all threads utilized.

SO_REUSEPORT:
It is undesirable however to continuously refresh gRPC connections as opposed to leverage request multiplexing over stable connections. So an alternative we attempted was to configure the listener socket options to use SO_REUSEPORT as follows.

{
   "name": "listener-name",
   "address": { . . . },
   "filter_chains": [ . . .],
   "socket_options": [{
      "description": "SO_REUSEPORT",
      "level": 1,
      "name": 15,
      "int_value": 1
   }]
}

With this option we expect better connection distribution over worker threads, but we did not see any measurable change in behavior.
Is SO_REUSEPORT option not set properly above, or not honored in Envoy ? Any recommendations to mitigate this issue ?

Environment details:
Envoy version: 1.9.1
Image: alpine:3.10.0
We use default Envoy concurrency configuration. Instance has 8 Core CPU

[mattklein123]

This is a duplicate of #4602. Shall we track there?

[ahmedelbazsc]

Thanks @mattklein123 . The connection accept distribution over worker threads seems covered by the other issue. For SO_REUSEPORT, is the setting provided above expected to be honored by the listener ?

[mattklein123]

That config should enable SO_REUSEPORT, but it's unclear to me how that would help connection distribution. The kernel will still decide how to allocate connections to workers in a potentially non-optimal way.

[ahmedelbazsc]

Some references suggest this option provides better connection distribution across all threads listening on the socket, so this was the motivation for trying this option. Thanks for confirming the config correctness

[davidkilliansc]

I think @ahmedelbazsc is referring to strategy (c) described here (though it likely involves more than just setting SO_REUSEPORT): https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/

We're running into this issue in several contexts. For example, I have thousands of incoming gRPC connections to an envoy sidecar from a downstream service. The load across these incoming connections is distributed evenly. I can run top -H and see that one of the Envoy threads consistently dominates CPU usage and cumulative CPU time. Here's an example where you can see the top 1-3 threads in Envoy dominate the rest (the rest of the 8 threads basically are always zero CPU).

On top of this, Envoy's behavior here unbalances that load before proxying upstream. There's 8 gRPC connections from Envoy to my local gRPC java process. That java process has eight IO worker threads, each processing one of the incoming envoy connections. Almost all the load coming from Envoy is concentrated on one or a few of the incoming Envoy connections, so all get processed on one or a few of our java IO processing threads. Our java process load balances new incoming connections across nonblocking worker threads, but it doesn't help here because Envoy puts all the traffic down one or a few of the incoming connections.

In this particular setup, I'm not sure how I'd work around it. In other variants of the problem, we have applications doing contortions to workaround this. For example, we have gRPC services sending upstream requests through an Envoy sidecar, but need to create a connection pool to the Sidecar. This wasn't enough because most the connections would just get assigned to one Envoy thread. They even needed to do things to recycle the connections and randomize the time the connections were created so they don't end up all assigned to the same Envoy thread and cripple throughput.

I really think we need a simple way to get round-robin or least-conns assignment from new connections to worker threads.

[mattklein123]

Closing as tracking this in #4602