Creating complex network configurations

[daanbosch]

Hello,

I'm trying to install on bare metal with some customization. However I have a couple of questions.
Please allow me to elaborate:

I'm installing on bare-metal, the bare metals have:

• OOB management interface (For connecting to the machine while the main network is down, not to tbe used for any traffic)
• interface1
• interface2

What I am trying to achieve:

• OOB get's an IP from DHCP
• Bond interface 1 and 2 and add a vlan tag

I know the mac addresses of the interfaces, and I found this documentation:
https://docs.fedoraproject.org/en-US/fedora-coreos/customize-nic/
So that should settle the interface naming for the bond.

Next, the network configuration, I found this:
https://docs.fedoraproject.org/en-US/fedora-coreos/sysconfig-network-configuration/
It explains how to do vlan and bonds. That will work as well.

However, what I'm unsure about is the following:

• How do I tell OKD what interface/route to use as default? Should I add default routes? My goal is to make sure the OOB interface is not used by OKD.
• Do I need to create/modify an ISO per bare metal? As I'm configuring a static ip on a bond with no DHCP servers.

[forzamehlano]

Is the no DHCP servers a choice or a restriction?

Our network looks similar to your setup, and we build the nodes via DHCP and then statically assign them addressing / bonding / routing configuration.

Presumably your bonded interface will also be the default route? In which case OKD will never touch your OOB management interface unless you intentionally configure it (either by configuring the SDN to route via the host (default in OpenshiftSDN but not for OVNKubernetes), or implement some form of CNI plugin on the OOB interface (again, not by default)).

[daanbosch]

Is the no DHCP servers a choice or a restriction?

A restriction in this scenario

Our network looks similar to your setup, and we build the nodes via DHCP and then statically assign them addressing / bonding / routing configuration.

What do you use to configure this at a later stage?

Presumably your bonded interface will also be the default route?
It should be, yes

Currently what I use to configure the networking:

coreos-installer iso customize fedora-coreos-39.20231101.3.0-live.x86_64-my-machine.iso --dest-ignition ignition/master.ign --network-keyfile ../nmstate-files/my-machine/bond0.200.nmconnection --network-keyfile ../nmstate-files/my-machine/bond0.nmconnection --network-keyfile ../nmstate-files/my-machine/ens8f0np0.nmconnection --network-keyfile ../nmstate-files/my-machine/ens8f1np1.nmconnection  --network-keyfile ../nmstate-files/my-machine/eno1.nmconnection --dest-device /dev/sda -f

Then I boot from ISO.
Networking for Openshift:
networking:

  machineNetwork:
  - cidr: 10.0.0.0/24
  clusterNetwork:
  - cidr: 10.0.2.0/23
    hostPrefix: 26
  networkType: Calico
  serviceNetwork:
  - 10.0.1.0/24

I also tried with kubernetes-ovn instead of Calico (Preffered)