Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

systemd networkd configuration broken #6655

Closed
afbjorklund opened this issue Feb 16, 2020 · 8 comments · Fixed by #6659
Closed

systemd networkd configuration broken #6655

afbjorklund opened this issue Feb 16, 2020 · 8 comments · Fixed by #6659
Labels
area/guest-vm General configuration issues with the minikube guest VM co/runtime/crio CRIO related issues co/runtime/docker Issues specific to a docker runtime kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@afbjorklund
Copy link
Collaborator

afbjorklund commented Feb 16, 2020 •
edited
Loading

The eth0 and eth1 don't come up correctly in networkctl, at least not in VirtualBox.

● systemd-networkd-wait-online.service - Wait for Network to be Configured
   Loaded: loaded (/usr/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:systemd-networkd-wait-online.service(8)

$ networkctl
IDX LINK             TYPE               OPERATIONAL SETUP     
  1 lo               loopback           carrier     unmanaged 
  2 eth0             ether              routable    configuring
  3 eth1             ether              routable    configuring
  4 sit0             sit                off         unmanaged 
  5 docker0          bridge             routable    unmanaged 
  7 veth7da0c9b      ether              carrier     unmanaged 
  9 veth03151ca      ether              carrier     unmanaged 

7 links listed.

This is a problem when you have another systemd unit that waits for network-online

It will wait for 2 minutes (120 seconds), and then fail with a timeout - the "dead" above.

For this reason, CRI-O is slow to start.

See cri-o/cri-o@ebf507d

Our Docker unit just waits on "network"

minikube docker.service:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target  minikube-automount.service docker.socket
Requires= minikube-automount.service docker.socket

docker.com docker.service:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket

See moby/moby@dfa4e77
@afbjorklund afbjorklund added area/guest-vm General configuration issues with the minikube guest VM kind/bug Categorizes issue or PR as related to a bug. co/runtime/docker Issues specific to a docker runtime co/runtime/crio CRIO related issues labels Feb 16, 2020
@afbjorklund afbjorklund mentioned this issue Feb 16, 2020
Merged
@afbjorklund
Copy link
Collaborator Author 

$ SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/systemd-networkd-wait-online
Found link 9
Found link 7
Found link 5
Found link 4
Found link 3
Found link 2
Found link 1
ignoring: lo
link eth1 is being processed by networkd
Event loop failed: Connection timed out

@afbjorklund afbjorklund added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Feb 16, 2020
@afbjorklund
Copy link
Collaborator Author

Something is buggy with the current configuration, since it works OK without it...

$ cat /etc/systemd/network/10-eth1.network 
[Match]
Name=eth1
Virtualization=qemu

[Network]
DHCP=ipv4

[DHCP]
UseDNS=false
$ cat /etc/systemd/network/20-dhcp.network 
[Match]
Name=eth*

[Network]
DHCP=ipv4

[DHCP]
ClientIdentifier=mac

https://github.com/kubernetes/minikube/tree/master/deploy/iso/minikube-iso/board/coreos/minikube/rootfs-overlay/etc/systemd/network

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Feb 16, 2020 •
edited
Loading 

Feb 16 11:07:42 minikube systemd-networkd[28381]: eth1: DHCPv4 address 192.168.99.100/24
Feb 16 11:07:42 minikube systemd-networkd[28381]: eth0: DHCPv4 address 10.0.2.15/24 via 10.0.2.2
Feb 16 11:07:42 minikube systemd-networkd[28381]: eth1: DHCP: No gateway received from DHCP server: No data available

This is probably not the configuration, but more like bugs in systemd 240.

@afbjorklund afbjorklund changed the title systemd network configuration broken systemd networkd configuration broken Feb 16, 2020
@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Feb 16, 2020 •
edited
Loading

Needs to have fixes backported, won't get an updated systemd until next OS upgrade.

https://github.com/systemd/systemd/commits/master/src/network/networkd-dhcp4.c

@afbjorklund
Copy link
Collaborator Author

Well, it is a little happier at least:

Feb 16 13:20:27 minikube systemd-networkd[2025]: eth1: Gained carrier
Feb 16 13:20:27 minikube systemd-networkd[2025]: eth0: Gained carrier
Feb 16 13:20:27 minikube systemd-networkd[2025]: eth1: DHCPv4 address 192.168.99.102/24
Feb 16 13:20:27 minikube systemd-networkd[2025]: eth1: DHCP: No gateway received from DHCP server.
Feb 16 13:20:27 minikube systemd-networkd[2025]: eth0: DHCPv4 address 10.0.2.15/24 via 10.0.2.2

But it still times out and fails. :-(

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Feb 16, 2020 •
edited
Loading

Another important fix:

EDIT: Did not seem to help, must be something else that is blocking the configure/complete.

@afbjorklund afbjorklund mentioned this issue Feb 16, 2020
Merged
@afbjorklund
Copy link
Collaborator Author

Probably same as systemd/systemd#8686

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Feb 16, 2020 •
edited
Loading

As per workarounds, this made it start:

[Network]
DHCP=ipv4
LinkLocalAddressing=no
$ networkctl 
IDX LINK             TYPE               OPERATIONAL SETUP     
  1 lo               loopback           carrier     unmanaged 
  2 eth0             ether              routable    configured
  3 eth1             ether              routable    configured
  4 sit0             sit                carrier     unmanaged 
  5 mybridge         bridge             carrier     unmanaged 
  6 veth205f338c     ether              carrier     unmanaged 
  7 veth9a5b12ae     ether              carrier     unmanaged 

7 links listed.

Apparently it defaults to ipv6 😵

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/guest-vm General configuration issues with the minikube guest VM co/runtime/crio CRIO related issues co/runtime/docker Issues specific to a docker runtime kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add systemd patch for handling DHCP router afbjorklund/minikube
1 participant
@afbjorklund