All of my Kubernetes API connection are stuck in SYN_SENT

[wizpresso-steve-cy-fan]

I saw this on my company's workstation

wizpresso@Wizpresso-02:~$ sudo conntrack -L | grep SYN_SENT
tcp      6 59 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60708 dport=443 [UNREPLIED] src=172.30.0.13 dst=10.96.0.1 sport=6443 dport=60708 mark=0 use=1
tcp      6 103 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60600 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60600 mark=0 use=1
tcp      6 105 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60608 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60608 mark=0 use=1
tcp      6 74 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60734 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60734 mark=0 use=1
tcp      6 52 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60520 dport=443 [UNREPLIED] src=172.30.0.13 dst=10.96.0.1 sport=6443 dport=60520 mark=0 use=1
tcp      6 65 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60694 dport=443 [UNREPLIED] src=172.30.0.13 dst=10.96.0.1 sport=6443 dport=60694 mark=0 use=1
tcp      6 59 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60718 dport=443 [UNREPLIED] src=172.30.0.13 dst=10.96.0.1 sport=6443 dport=60718 mark=0 use=1
tcp      6 35 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60700 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60700 mark=0 use=2
tcp      6 74 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60720 dport=443 [UNREPLIED] src=172.30.0.13 dst=10.96.0.1 sport=6443 dport=60720 mark=0 use=1
tcp      6 86 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60676 dport=443 [UNREPLIED] src=172.30.0.13 dst=10.96.0.1 sport=6443 dport=60676 mark=0 use=1
tcp      6 86 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60668 dport=443 [UNREPLIED] src=172.30.0.13 dst=10.96.0.1 sport=6443 dport=60668 mark=0 use=1
tcp      6 59 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60714 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60714 mark=0 use=1
tcp      6 111 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60680 dport=443 [UNREPLIED] src=172.30.0.13 dst=10.96.0.1 sport=6443 dport=60680 mark=0 use=1
tcp      6 59 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60702 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60702 mark=0 use=1
tcp      6 65 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60686 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60686 mark=0 use=1
tcp      6 29 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60660 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60660 mark=0 use=1
tcp      6 86 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60652 dport=443 [UNREPLIED] src=172.30.0.13 dst=10.96.0.1 sport=6443 dport=60652 mark=0 use=1
tcp      6 109 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60754 dport=80 [UNREPLIED] src=10.96.0.1 dst=10.96.0.1 sport=80 dport=60754 mark=0 use=1
tcp      6 105 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60602 dport=443 [UNREPLIED] src=172.30.0.13 dst=10.96.0.1 sport=6443 dport=60602 mark=0 use=1
tcp      6 86 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60664 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60664 mark=0 use=1
tcp      6 34 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60688 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60688 mark=0 use=2
tcp      6 86 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60674 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60674 mark=0 use=1
tcp      6 72 SYN_SENT src=10.96.0.1 dst=10.96.0.1 sport=60710 dport=443 [UNREPLIED] src=172.30.252.4 dst=10.96.0.1 sport=6443 dport=60710 mark=0 use=1

There is a lot of them. I can directly connect to the host, that means curl https://172,30.252.4:6443 or curl https://172.30.0.13:6443 works but not through IPVS itself, that means no curl https://10.96.0.1.

This is a very weird problem as this does not allow me to initialize the CNI, and so the whole node is not functional.

[wizpresso-steve-cy-fan]

wizpresso@Wizpresso-02:~$ sudo tcpdump -i eth0 -v src 10.96.0.1
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
17:37:50.303415 IP (tos 0x0, ttl 63, id 17751, offset 0, flags [DF], proto TCP (6), length 60)
    10.96.0.1.60664 > 172.30.0.13.6443: Flags [S], cksum 0xb6ba (incorrect -> 0x02b6), seq 4106059775, win 65495, options [mss 65495,sackOK,TS val 1363658905 ecr 0,nop,wscale 7], length 0
17:37:50.303433 IP (tos 0x0, ttl 63, id 17443, offset 0, flags [DF], proto TCP (6), length 60)
    10.96.0.1.60660 > 172.30.252.4.6443: Flags [S], cksum 0xb2b2 (incorrect -> 0x1fc5), seq 2947263502, win 65495, options [mss 65495,sackOK,TS val 1363658905 ecr 0,nop,wscale 7], length 0
17:37:50.383453 IP (tos 0x0, ttl 63, id 20540, offset 0, flags [DF], proto TCP (6), length 60)
    10.96.0.1.60670 > 172.30.252.4.6443: Flags [S], cksum 0xb2b2 (incorrect -> 0xca25), seq 2778336101, win 65495, options [mss 65495,sackOK,TS val 1363658985 ecr 0,nop,wscale 7], length 0
17:37:50.463418 IP (tos 0x0, ttl 63, id 30479, offset 0, flags [DF], proto TCP (6), length 60)
    10.96.0.1.60680 > 172.30.252.4.6443: Flags [S], cksum 0xb2b2 (incorrect -> 0x7e6c), seq 970703491, win 65495, options [mss 65495,sackOK,TS val 1363659065 ecr 0,nop,wscale 7], length 0
17:37:50.783407 IP (tos 0x0, ttl 63, id 8010, offset 0, flags [DF], proto TCP (6), length 60)
    10.96.0.1.60684 > 172.30.0.13.6443: Flags [S], cksum 0xb6ba (incorrect -> 0x5db5), seq 699330075, win 65495, options [mss 65495,sackOK,TS val 1363659385 ecr 0,nop,wscale 7], length 0
17:37:51.335942 IP (tos 0x0, ttl 63, id 50398, offset 0, flags [DF], proto TCP (6), length 60)
    10.96.0.1.60560 > 172.30.252.4.6443: Flags [S], cksum 0xb2b2 (incorrect -> 0x4d0e), seq 32385247, win 65495, options [mss 65495,sackOK,TS val 1363659937 ecr 0,nop,wscale 7], length 0
17:37:52.383406 IP (tos 0x0, ttl 63, id 50399, offset 0, flags [DF], proto TCP (6), length 60)
    10.96.0.1.60560 > 172.30.252.4.6443: Flags [S], cksum 0xb2b2 (incorrect -> 0x48f6), seq 32385247, win 65495, options [mss 65495,sackOK,TS val 1363660985 ecr 0,nop,wscale 7], length 0
17:37:54.463455 IP (tos 0x0, ttl 63, id 50400, offset 0, flags [DF], proto TCP (6), length 60)
    10.96.0.1.60560 > 172.30.252.4.6443: Flags [S], cksum 0xb2b2 (incorrect -> 0x40d6), seq 32385247, win 65495, options [mss 65495,sackOK,TS val 1363663065 ecr 0,nop,wscale 7], length 0

Hmm...checksum problem?