Alternative communication protocol between server and agent?

[charleszheng44]

Currently, communication tunnel between agent and server is based on gRPC. Before using ANP, I used to set up the tunnel using TCP. I did a small benchmark to compare the performance of using gRPC (i.e., ANP) over TCP. As expect, using gRPC introduces some extra overheads.

Run kubectl exec test-po -- date 100 times thorugh gRPC tunnel

 pkts      bytes target  prot opt in     out     source               destination
2730   530312            tcp   --   *       *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8091
2758   491823            tcp   --   *       *       0.0.0.0/0            0.0.0.0/0            tcp spt:8091

Run kubectl exec test-po -- date 100 times through TCP tunnel

  pkts      bytes target  prot opt in     out     source               destination
 3608   397313            tcp   --   *       *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8091
 3729   420205            tcp   --   *       *       0.0.0.0/0            0.0.0.0/0            tcp spt:8091

There're cases where tunnel may need to go through the public network, and lowering the overhead can help the user to save some costs. Should we consider having an alternate to gRPC?

[charleszheng44]

Or, maybe we can add gRPC compressor to reduce the bandwidth?

[cheftako]

+1
There is already a mode flag to control the communication between client (KAS) and server. We always new we would also want one between agent and server. I had been imagining it would be more likely to enable something like VPN support but your welcome to look at adding an optimized for bandwidth alternative.

[charleszheng44]

@cheftako Thanks for the reply.

Maybe we can have something like the following?

         gRPC    +------------+   gRPC   +-----------+  TCP   +-------+
     +---------->+proxy|server+--------->+proxy|agent+------->+kubelet|
     |           +------------+          +-----------+        +-------+
     |
+----+----+
|apiserver|
+----+----+
     |
     |           +------------+          +-----------+        +-------+
     +---------->+proxy|server+--------->+proxy|agent+------->+kubelet|
         HTTP    +------------+   TCP    +-----------+  TCP   +-------+

Just curious, is there any particular reason to choose gRPC over TCP at the beginning?

[mvladev]

Fundamentally, for multiplexing , TCP is a bad choice due to head of line problems which can affect all streams in the single TCP connection.

I've played around with quic and the native support for streams in the protocol make the implementation of the server and agent quite trivial. See this repo - https://github.com/mvladev/quic-reverse-http-tunnel:

apiserver --- TCP ----> [proxy-server] ---- QUIC ----> [proxy-agent]---TCP--> [kubelet]

1. the proxy-server listens for tcp (no HTTP server running) and quic.
2. The proxy-agent talks to the server and opens a quic session.
3. It starts a HTTP tunnel server that listens on that session for new streams.
4. When the API server / curl talks to the proxy-server, it creates a new quic stream and sends the data to the proxy-agent.
5. The HTTP server in the proxy-agent that listens on new quic streams accepts the stream, opens tcp connection to the requested host (from the CONNECT) and pipes the data back.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.