AWS/OCP 4.5.13: Submariner + Globalnet: Pod with HostNetworking on GW Node to remoteService failing

[sridhargaddam]

What happened:
In a Submariner Globalnet deployment, when e2e tests are executed on AWS/OCP Clusters, one of them is consistently failing.
Basically when a Pod with HostNetworking is trying to connect to a remoteService its failing.

STEP: Creating a listener pod in cluster "cluster-f", which will wait for a handshake over TCP
STEP: Pointing a service ClusterIP to the listener pod in cluster "cluster-f"
Dec  3 15:27:05.661: INFO: Will send traffic to IP: 169.254.32.200
STEP: Creating a connector pod in cluster "cluster-e", which will attempt the specific UUID handshake over TCP
STEP: Waiting for the connector pod "tcp-check-podxfwlq" to exit, returning what connector sent
Dec  3 15:27:11.621: INFO: Pod "tcp-check-podxfwlq" output:
169.254.32.200 (169.254.32.200:1234) open
[dataplane] listener says 26d4329b-2467-4dd8-84f1-6c2102d6f7f1

STEP: Waiting for the listener pod "tcp-check-listener6k5bp" to exit, returning what listener sent
Dec  3 15:31:12.234: INFO: Pod "tcp-check-listener6k5bp" output:
listening on 0.0.0.0:1234 ...
connect to 10.129.0.52:1234 from 169.254.0.135:34739 (169.254.0.135:34739)

Interestingly, if you look at the output of the listener pod, we can see that the connector pod was indeed able to reach the listener, but it was unable to send the UUID string (that is sent as part of e2e tests).

What you expected to happen:
e2e tests should pass consistently.

Anything else we need to know?:

Environment:

• Submariner version (use subctl version): v0.8.0-rc0
• Kubernetes version (use kubectl version): v1.18.3+47c0e71
• Cloud provider or hardware configuration: AWS/OCP 4.5.13
• OS (e.g: cat /etc/os-release): Alpine Linux
• Kernel (e.g. uname -a): 4.18.0-193.23.1.el8_2.x86_64
• Install tools:
• Network plugin and version (if this is a network-related bug): OpenshiftSDN
• Others:

[sridhargaddam]

On investigating this issue, it appears like its related to MTU mismatch and when the remoteCluster is sending ICMP unreachable - need to fragment packet, it does not seem to be properly handled.

Following is the tcpdump on the Gateway node of cluster-west when e2e test-scenario is executed.

tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
09:15:01.283456  In 16:c1:b5:6a:a0:a3 ethertype IPv4 (0x0800), length 72: (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto ICMP (1), length 56)
    10.0.48.1 > 10.0.49.156: ICMP 3.15.213.107 unreachable - need to frag (mtu 1500), length 36
        (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 3316)
    10.0.49.156.4500 > 3.15.213.107.4500: [no cksum] [|isakmp]
        0x0000:  4500 0038 0000 4000 ff01 0628 0a00 3001  E..8..@....(..0.
        0x0010:  0a00 319c 0304 c717 0000 05dc 4500 0cf4  ..1.........E...
        0x0020:  0000 4000 4011 19e3 0a00 319c 030f d56b  ..@[email protected]
        0x0030:  1194 1194 0ce0 0000 

[sridhargaddam]

By setting a value of 1 (or 2) for /proc/sys/net/ipv4/tcp_mtu_probing on the Gateway Node, this problem is resolved.

 tcp_mtu_probing (integer; default: 0; since Linux 2.6.17)
              This parameter controls TCP Packetization-Layer Path MTU
              Discovery.  The following values may be assigned to the file:
              0  Disabled
              1  Disabled by default, enabled when an ICMP black hole
                 detected
              2  Always enabled, use initial MSS of tcp_base_mss.

[mangelajo]

Co

By setting a value of 1 (or 2) for /proc/sys/net/ipv4/tcp_mtu_probing on the Gateway Node, this problem is resolved.

 tcp_mtu_probing (integer; default: 0; since Linux 2.6.17)
              This parameter controls TCP Packetization-Layer Path MTU
              Discovery.  The following values may be assigned to the file:
              0  Disabled
              1  Disabled by default, enabled when an ICMP black hole
                 detected
              2  Always enabled, use initial MSS of tcp_base_mss.

Cool, that seems like a better solution than mss clamping.

Let me try it

[mangelajo]

/proc/sys/net/ipv4/tcp_mtu_probing

Ok, it doesn't work for this case.

I will continue with the mss clamping