Wrong interface selection on Vagrant

[selcukusta]

When trying to install k3s via Vagrant, Flannel selects the first network interface but the first one is using to bridge guest machines on Vagrant.

First solution is using flannel with external installation (with —no-flannel flag). My suggestion is, run the k3s server with “—iface=enp0s8” flag which will be used for flannel configuration or something like that.

[ibuildthecloud]

@selcukusta There is a --node-ip x.x.x.x flag, would that work?

[selcukusta]

hi @ibuildthecloud! Unfortunately, doesn't work. My cluster info:

master01 192.168.0.10
worker01 192.168.0.11
worker02 192.168.0.12

vagrant@worker01:~$ ifconfig output is;

cni0      Link encap:Ethernet  HWaddr d6:40:4c:2e:17:eb
          inet addr:10.42.1.1  Bcast:10.42.1.255  Mask:255.255.255.0
          inet6 addr: fe80::d440:4cff:fe2e:17eb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:9 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:564 (564.0 B)  TX bytes:918 (918.0 B)

enp0s3    Link encap:Ethernet  HWaddr 02:0d:67:c1:5e:a8
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::d:67ff:fec1:5ea8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:76692 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14231 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:102405443 (102.4 MB)  TX bytes:1430370 (1.4 MB)

enp0s8    Link encap:Ethernet  HWaddr 08:00:27:1d:57:09
          inet addr:192.168.0.11  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe1d:5709/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:980 errors:0 dropped:0 overruns:0 frame:0
          TX packets:914 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:774461 (774.4 KB)  TX bytes:152067 (152.0 KB)

flannel.1 Link encap:Ethernet  HWaddr 72:60:cc:df:cb:e0
          inet addr:10.42.1.0  Bcast:10.42.1.0  Mask:255.255.255.255
          inet6 addr: fe80::7060:ccff:fedf:cbe0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:8 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1192 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1192 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:824713 (824.7 KB)  TX bytes:824713 (824.7 KB)

veth29e02690 Link encap:Ethernet  HWaddr 3a:82:92:a5:3c:16
          inet6 addr: fe80::3882:92ff:fea5:3c16/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:9 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:690 (690.0 B)  TX bytes:1476 (1.4 KB)

vagrant@worker01:~$ sudo cat /run/flannel/subnet.env output is;

FLANNEL_NETWORK=10.42.0.0/16
FLANNEL_SUBNET=10.42.1.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

It might be wrong, am I right?

[colby]

This is a documented behavior of kubelet, where it will select the default interface, which happens to be the NAT network of Vagrant. Try using KUBELET_EXTRA_ARGS to override that.

[selcukusta]

Yes, you're right. Two actions are needed. The first one is;

sudo sed -i '9s/^/Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false --node-ip='"$1"'"\n/' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf but kubelet service is embedded in k3s binary.

Another one is add --iface=enp0s8 flag to flannel manifest file like that;

      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.10.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=enp0s8

But the interface selection operation is embedded also. So I couldn't change them.

[aaliddell]

Setting --node-ip only provides half the required configurability, as @selcukusta mentioned.

For some clarification, consider the following setup:

              ^                                      ^
              |                                      |
              |                                      |
Default route |     10.2.0.1            10.2.0.2     | Default route
     10.1.0.1 |     +--------------------------+     | 10.1.0.2
              |     |       Internal Net       |     |
           +--+-----+--+                    +--+-----+--+
           |eth0| |eth1|                    |eth1| |eth0|
           +----+ +----+                    +----+ +----+
           |           |                    |           |
           |   node1   |                    |   node2   |
           |           |                    |           |
           +-----------+                    +-----------+

Each node has two interfaces, one to the wider network (eth0, set as default route) and one connected to a private internal network (eth1). The end goal is for the flannel VXLAN to sit on the private eth1 network, whilst the LoadBalancer services etc should sit on the eth0 interface.

To achieve this, each node should have their --node-ip set to their eth0 IP, which is what would occur by default as this is the default interface. However, the flannel VXLAN cannot be moved to the eth1 interface, as it is hardcoded to select the default route interface: https://github.com/rancher/k3s/blob/17a44a319700b69ad20ec13086e219bcceb24cb5/pkg/agent/flannel/flannel.go#L40

For evidence: setting --node-ip to 10.2.0.1 does not affect the VXLAN interface (and also obviously puts the services on the wrong ip):

ip -d l show flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
    ...
    vxlan id 1 local 10.1.0.1 dev eth0 srcport 0 0 dstport 8472 ...

To solve this, somehow the flannel interface should be configurable and it looks like the config options may be available in func flannel(...) as an argument? Perhaps --flannel-iface=... as an argument?

With standard Flannel this is available as an option, which could be used as inspiration: https://github.com/coreos/flannel/blob/0fb471046932ef5f8f363f7daace3d6f4d372dc9/main.go#L118 & https://github.com/coreos/flannel/blob/0fb471046932ef5f8f363f7daace3d6f4d372dc9/main.go#L195-L216

Also, tengentially related: the readme states the VXLAN port is on 4789, but flannel puts it on 8472, as you can see in the command output above. I'll send a PR to change the docs.

[yoink00]

I've also faced the same problem on a setup where I need to use the non-default interface to communicate between the nodes. I've made an attempt to allow this to be configured here: https://github.com/yoink00/k3s

It is completely untested but should be complete. I'll be testing this tomorrow. Any comments would be gratefully received.

To override the default interface specify '--flannel-iface=eth1' (for example).

[aaliddell]

@yoink00: That looks reasonable, you should open a PR to get others to take a look.

In the meantime, for people stuck with this: I just went the route of using --no-flannel to bypass the fixed internal config, then manually deployed Flannel as CNI with the correct --iface option passed in. Although this also requires setting up CNI plugins correctly manually in the right places and matching the Flannel config to the CIDRs etc.

[deniseschannon]

This is available in our latest RC (v0.4.0-rc3). If anyone could test it out, it'd be great!

[erikwilson]

Verified by setting --flannel-iface=... and checking that ip -d l show flannel.1 included the desired interface. Thanks for submitting a PR @yoink00, if this does not fill your needs feel free to re-open or submit a new issue @selcukusta & @aaliddell

[balthild]

Does this option (--flannel-iface) can be set by environment variable?

[costela]

@balthild currently no (see here), but I just opened #659 to see if we can address this.