Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release 1.24] Make flannel aware of node-external-ip parameter #6188

Closed
manuelbuil opened this issue Sep 29, 2022 · 3 comments
Closed

[Release 1.24] Make flannel aware of node-external-ip parameter #6188

manuelbuil opened this issue Sep 29, 2022 · 3 comments
Assignees
@VestigeJ @manuelbuil
Milestone
v1.24.6+k3s1

Comments

@manuelbuil
Copy link
Contributor

Backport: #6177
@manuelbuil manuelbuil self-assigned this Sep 29, 2022
@manuelbuil manuelbuil added this to the v1.24.6+k3s1 milestone Sep 29, 2022
@manuelbuil manuelbuil mentioned this issue Sep 29, 2022
Merged
@VestigeJ
Copy link

##Environment Details

Infrastructure

  • Cloud

Node(s) CPU architecture, OS, and Version:

Linux 5.11.0-1022-aws x86_64 GNU/Linux Ubuntu 20.04.3 LTS

Cluster Configuration:
1 server in AWS
1 agent in Linode

NAME                STATUS   ROLES                       AGE   VERSION
ip-2-16-1-250   Ready    control-plane,etcd,master   15m   v1.24.7-rc2+k3s1
localhost           Ready    <none>                      15m   v1.24.7-rc2+k3s1

Config.yaml:

write-kubeconfig-mode: 644
debug: true
token: coolclouds
protect-kernel-defaults: true
cluster-init: true
cluster-cidr: 1.42.1.0/16,2001:cafe:42:0::/56
service-cidr: 1.43.1.0/16,2001:cafe:42:1::/112
node-ip: 1.1.1.2,2600:1f1c:ab4:iced:ea13:cafe:651c:d00c
node-external-ip: 2.16.1.250,2600:1f1c:ab4:iced:ea13:cafe:651c:d00c
flannel-backend: wireguard-native

Replication Steps

- Installed k3s:
<!-- Provide the command to install k3s -->
   $  get.k3s; 
   $  set.figs 
   $  VERSION=v1.24.7-rc2+k3s1
   $  sudo INSTALL_K3S_VERSION=$VERSION INSTALL_K3S_EXEC=server ./install-k3s.sh 
   $  kgn
   $  kgp -A -o wide
   $  k exec -n kube-system -it svclb-traefik-d9f1ea3a-7zbq5 -- ping 10.42.0.7
   $  k exec -n kube-system -it svclb-traefik-d9f1ea3a-q6mkw -- ping 10.42.1.2
   $  cat /etc/os-release 
   $  k3s -v
   $  get.report"

Results:

v1.24.7-rc2+k3s1

$ kgn -o yaml | grep -i -e annotation -A 15

    annotations:
      etcd.k3s.cattle.io/node-address: 1.1.1.250
      etcd.k3s.cattle.io/node-name: ip-1-1-1-250-32381500
      flannel.alpha.coreos.com/backend-data: '{"PublicKey":"aUD0ly59UXpw="}'
      flannel.alpha.coreos.com/backend-type: wireguard
      flannel.alpha.coreos.com/backend-v6-data: '{"PublicKey":"aUD0ly59UXpw="}'
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: 5.2.2.2
      flannel.alpha.coreos.com/public-ip-overwrite: 5.2.2.2
      flannel.alpha.coreos.com/public-ipv6: 2600:1f1c:iced:ee48:cafe:b00c:951c:f17e
      flannel.alpha.coreos.com/public-ipv6-overwrite: 2600:1f1c:iced:ee48:cafe:b00c:951c:f17e
      k3s.io/external-ip: 5.2.2.2,2600:1f1c:iced:ee48:cafe:b00c:951c:f17e
      k3s.io/hostname: ip-1-1-1-250
      k3s.io/internal-ip: 1.1.1.250,2600:1f1c:iced:ee48:cafe:b00c:951c:f17e
      k3s.io/node-args: '["server","--write-kubeconfig-mode","644","--debug","true","--token","********","--protect-kernel-defaults","true","--cluster-init","true","--cluster-cidr","10.42.0.0/16,2001:cafe:42:0::/56","--service-cidr","10.43.0.0/16,2001:cafe:42:1::/112","--node-ip","192.168.10.250,2600:1f1c:iced:ee48:cafe:b00c:951c:f17e","--node-external-ip","5.2.2.2,2600:1f1c:iced:ee48:cafe:b00c:951c:f17e","--flannel-backend","wireguard-native"]'
      k3s.io/node-config-hash: SGS7L3VAF74RQ====
--
    annotations:
      flannel.alpha.coreos.com/backend-data: '{"PublicKey":"tS2yIZXIelY="}'
      flannel.alpha.coreos.com/backend-type: wireguard
      flannel.alpha.coreos.com/backend-v6-data: '{"PublicKey":"tS2yIZXIelY="}'
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: 9.7.7.7
      flannel.alpha.coreos.com/public-ip-overwrite: 9.7.7.7
      flannel.alpha.coreos.com/public-ipv6: 2300:cafe::iced:booc:fe15:db63
      flannel.alpha.coreos.com/public-ipv6-overwrite: 2300:cafe::iced:booc:fe15:db63
      k3s.io/external-ip: 9.7.7.7,2300:cafe::iced:booc:fe15:db63
      k3s.io/hostname: localhost
      k3s.io/internal-ip: 9.7.7.7,2300:cafe::iced:booc:fe15:db63
      k3s.io/node-args: '["agent","--token","********","--server","https://5.2.2.2:6443","--node-external-ip","9.7.7.7,2300:cafe::iced:booc:fe15:db63"]'
      k3s.io/node-config-hash: BPXACXGUNWA====
      k3s.io/node-env: '{"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/5f37b186566aa35f0a966d566dad0314a29bee23aa3e414e02e345c1eac0f762"}'
      node.alpha.kubernetes.io/ttl: "0"

$ k exec -n kube-system -it svclb-traefik-d9f1ea3a-7zbq5 -- ping 10.42.0.7

Defaulted container "lb-tcp-80" out of: lb-tcp-80, lb-tcp-443
PING 10.42.0.7 (10.42.0.7): 56 data bytes
64 bytes from 10.42.0.7: seq=0 ttl=62 time=2.104 ms
64 bytes from 10.42.0.7: seq=1 ttl=62 time=1.972 ms
64 bytes from 10.42.0.7: seq=2 ttl=62 time=1.944 ms
64 bytes from 10.42.0.7: seq=3 ttl=62 time=2.078 ms
64 bytes from 10.42.0.7: seq=4 ttl=62 time=2.024 ms
^C
--- 10.42.0.7 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 1.944/2.024/2.104 ms

$ k exec -n kube-system -it svclb-traefik-d9f1ea3a-q6mkw -- ping 10.42.1.2

Defaulted container "lb-tcp-80" out of: lb-tcp-80, lb-tcp-443
PING 10.42.1.2 (10.42.1.2): 56 data bytes
64 bytes from 10.42.1.2: seq=0 ttl=62 time=2.038 ms
64 bytes from 10.42.1.2: seq=1 ttl=62 time=2.052 ms
64 bytes from 10.42.1.2: seq=2 ttl=62 time=2.203 ms
64 bytes from 10.42.1.2: seq=3 ttl=62 time=2.050 ms
64 bytes from 10.42.1.2: seq=4 ttl=62 time=2.069 ms
64 bytes from 10.42.1.2: seq=5 ttl=62 time=2.371 ms
64 bytes from 10.42.1.2: seq=6 ttl=62 time=2.016 ms
^C
--- 10.42.1.2 ping statistics ---
7 packets transmitted, 7 packets received, 0% packet loss
round-trip min/avg/max = 2.016/2.114/2.371 ms

@rancher-max rancher-max reopened this Oct 24, 2022
@rancher-max
Copy link
Contributor

Reopened to re-validate with the newly added flannel-external-ip flag

@VestigeJ
Copy link

$ get.figs server

=========== k3s config =========== 
write-kubeconfig-mode: 644
debug: true
token: coolclouds
selinux: true
protect-kernel-defaults: true
profile: cis-1.6
node-external-ip: $SERVER_PUBLIC_IP
flannel-backend: wireguard-native
flannel-external-ip: true

$ get.figs agent

=========== k3s config =========== 
token: coolclouds
selinux: true
server: https://$SERVER_PUBLIC_IP:6443
node-external-ip: $AGENT_PUBLIC_IP

$ kgn

NAME               STATUS   ROLES                  AGE     VERSION
ip-SERVER_PUBLIC_IP   Ready    control-plane,master   3m45s   v1.24.7-rc4+k3s1
ip-AGENT_PUBLIC_IP    Ready    <none>                 2m21s   v1.24.7-rc4+k3s1

$ kgn -o yaml | grep -i -e annotation -A 15

    annotations:
      flannel.alpha.coreos.com/backend-data: '{"PublicKey":"ydAt61kA2YKL2BZx7iYos8gpyxIt/RmRnJOljOwjHhY="}'
      flannel.alpha.coreos.com/backend-type: wireguard
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: $SERVER_PUBLIC_IP
      flannel.alpha.coreos.com/public-ip-overwrite: $SERVER_PUBLIC_IP
      k3s.io/external-ip: $SERVER_PUBLIC_IP
      k3s.io/hostname: ip-$SERVER_IP
      k3s.io/internal-ip: $SERVER_IP
      k3s.io/node-args: '["server","--write-kubeconfig-mode","644","--debug","true","--token","********","--selinux","true","--protect-kernel-defaults","true","--profile","cis-1.6","--node-external-ip","$SERVER_PUBLIC_IP","--flannel-backend","wireguard-native","--flannel-external-ip","true"]'
      k3s.io/node-config-hash: 7JVWIVWCFG4G2GCRMUQERGBVD226KKLL3PW2I4PPV6MTHEDGDX5Q====
      k3s.io/node-env: '{"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/f693798eb26e0dfbfbd8e881866f157e6ba7ec7bdb7e11b9ac03caffc285c2b4"}'
      node.alpha.kubernetes.io/ttl: "0"
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2022-10-25T17:11:41Z"
    finalizers:
--
    annotations:
      flannel.alpha.coreos.com/backend-data: '{"PublicKey":"bI/quOo2s2KY6SvVA+QYAsTbmHd4TfVLBkreAXnJ/kY="}'
      flannel.alpha.coreos.com/backend-type: wireguard
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: $AGENT_PUBLIC_IP
      flannel.alpha.coreos.com/public-ip-overwrite: $AGENT_PUBLIC_IP
      k3s.io/external-ip: $AGENT_PUBLIC_IP
      k3s.io/hostname: ip-$AGENT_IP
      k3s.io/internal-ip: $AGENT_IP
      k3s.io/node-args: '["agent","--token","********","--selinux","true","--server","https://$SERVER_PUBLIC_IP:6443","--node-external-ip","$AGENT_PUBLIC_IP"]'
      k3s.io/node-config-hash: ML7XOOSYSHIQ67GUHLZ4JEDNR4IAYPALF5SXJZYD2ILJN5CMT76A====
      k3s.io/node-env: '{"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/f693798eb26e0dfbfbd8e881866f157e6ba7ec7bdb7e11b9ac03caffc285c2b4"}'
      node.alpha.kubernetes.io/ttl: "0"
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2022-10-25T17:13:05Z"
    finalizers:

$ kgp -n kube-system svclb-traefik-b974a32c-wzzb2 -o yaml | grep -e "podIP: " | awk '{print $2}'

10.42.1.2

$ kgp -n kube-system svclb-traefik-b974a32c-rxkp5 -o yaml | grep -e "podIP: " | awk '{print $2}'

10.42.0.7

$ k exec -n kube-system -it svclb-traefik-b974a32c-rxkp5 -- ping -c 5 10.42.1.2

Defaulted container "lb-tcp-80" out of: lb-tcp-80, lb-tcp-443
PING 10.42.1.2 (10.42.1.2): 56 data bytes
64 bytes from 10.42.1.2: seq=0 ttl=62 time=0.780 ms
64 bytes from 10.42.1.2: seq=1 ttl=62 time=0.784 ms
64 bytes from 10.42.1.2: seq=2 ttl=62 time=0.729 ms
64 bytes from 10.42.1.2: seq=3 ttl=62 time=0.696 ms
64 bytes from 10.42.1.2: seq=4 ttl=62 time=0.712 ms

--- 10.42.1.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.696/0.740/0.784 ms

$ k exec -n kube-system -it svclb-traefik-b974a32c-rxkp5 -- ping -c 5 10.42.0.7

Defaulted container "lb-tcp-80" out of: lb-tcp-80, lb-tcp-443
PING 10.42.0.7 (10.42.0.7): 56 data bytes
64 bytes from 10.42.0.7: seq=0 ttl=64 time=0.061 ms
64 bytes from 10.42.0.7: seq=1 ttl=64 time=0.066 ms
64 bytes from 10.42.0.7: seq=2 ttl=64 time=0.063 ms
64 bytes from 10.42.0.7: seq=3 ttl=64 time=0.074 ms
64 bytes from 10.42.0.7: seq=4 ttl=64 time=0.067 ms

--- 10.42.0.7 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.061/0.066/0.074 ms

Interfaces added to the server node
$ ip a

33: flannel-wg: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8921 qdisc noqueue state UNKNOWN group default 
    link/none 
    inet 10.42.0.0/32 scope global flannel-wg
       valid_lft forever preferred_lft forever
34: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8921 qdisc noqueue state UP group default qlen 1000
    link/ether c2:5a:29:fa:7e:f5 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::c05a:29ff:fefa:7ef5/64 scope link 
       valid_lft forever preferred_lft forever
35: veth5cb23a2c@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8921 qdisc noqueue master cni0 state UP group default 
    link/ether 9e:06:21:a9:43:13 brd ff:ff:ff:ff:ff:ff link-netns cni-53ab1b62-7e71-6e38-fe23-9f87f3321ff0
    inet6 fe80::9c06:21ff:fea9:4313/64 scope link 
       valid_lft forever preferred_lft forever
38: veth1758e59f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8921 qdisc noqueue master cni0 state UP group default 
    link/ether f2:13:95:da:47:4d brd ff:ff:ff:ff:ff:ff link-netns cni-08bf7aed-1bca-63dc-dba1-a7b758688315
    inet6 fe80::f013:95ff:feda:474d/64 scope link 
       valid_lft forever preferred_lft forever
39: vethe62a7f09@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8921 qdisc noqueue master cni0 state UP group default 
    link/ether fa:a8:1b:a4:c1:b9 brd ff:ff:ff:ff:ff:ff link-netns cni-e076f334-3c42-2eaa-1cb1-d1115da88a2f
    inet6 fe80::f8a8:1bff:fea4:c1b9/64 scope link 
       valid_lft forever preferred_lft forever
40: vetha957a935@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8921 qdisc noqueue master cni0 state UP group default 
    link/ether de:a1:fc:ef:0d:8b brd ff:ff:ff:ff:ff:ff link-netns cni-1f456024-0f92-6e58-4e67-ce0b01d84d70
    inet6 fe80::dca1:fcff:feef:d8b/64 scope link 
       valid_lft forever preferred_lft forever
41: veth16221375@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8921 qdisc noqueue master cni0 state UP group default 
    link/ether 16:ab:99:19:73:cd brd ff:ff:ff:ff:ff:ff link-netns cni-228c6bf2-b76f-da5d-cbac-d7761d6092c9
    inet6 fe80::14ab:99ff:fe19:73cd/64 scope link 
       valid_lft forever preferred_lft forever

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VestigeJ VestigeJ

@manuelbuil manuelbuil

Labels
None yet
Projects
None yet
Milestone
v1.24.6+k3s1
Development

No branches or pull requests
3 participants
@VestigeJ @manuelbuil @rancher-max