README_draft_1811

multus-cni Logo

<XXX: add ToC>

What is multus-cni?

Multus is a latin word for "Multi"
As the name suggests, it acts as a Multi plugin in Kubernetes and provides the multiple network interface support in a pod
This project is a reference implementation for Kubernetes Network Custom Resource Definition De-facto Standard. For more information refer Network Plumbing Working Group Agenda
Reference implementation support following modes as "section 3.2 Network Object Definition" in the spec
- CNI config JSON in network object
- Not using CNI config (“thick” plugin usecase)
- CNI configuration stored in on-disk file
refer the section 3.2 Network Object Definition for more details in Kubernetes Network Custom Resource Definition De-facto Standard
Multus supports all reference plugins (eg. Flannel, DHCP, Macvlan) that implement the CNI specification and all 3rd party plugins (eg. Calico, Weave, Cilium, Contiv). In addition to it, Multus supports SRIOV, SRIOV-DPDK, OVS-DPDK & VPP workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes
The default network gets "eth0" and additional network Pod interface name as “net0”, “net1”,… “netX and so on. Multus also support interface names from the user.
Multus is one of the projects in the Baremetal Container Experience kit.

Multi-Homed pod

This is the example pod with multus. The pod has three interface: eth0, net0 and net1. eth0 connects kubernetes cluster network to connect with kubernetes server/services (e.g. kubernetes api-server, kubelets and so on). net0 and net1 are network attachment and connect to other networks with other CNI networks (e.g. vlan/vxlan/ptp).

multus-pod-image

Version compatibility

Current version of multus-cni is '3.1'. v3.0 and above are not compatible with v1.0 and v2.0 version Network Object CRD.

Quickstart Guide

See QUICKSTART.md <-XXX:TBD

How to get?

Get binaries from release page
Get docker image from Docker Hub
Build from source

Build from source

This plugin requires Go 1.8 (or later) to build.

# ./build

How to use multus-cni?

(Prerequisite) XXX: TBD
(Install multus)
- copy multus binary
- use daemonset XXX: TBD
add/modify conf file in /etc/cni/net.d/ (XXX: need to add about delegate/defaultnet/clusternet) XXX: TBD
Setup CRDs (daemonset automatically does
- NetworkAttachmentDefinition
- ClusterRole
- ClusterRoleBinding XXX: TBD
Create network attachment definition
- net-attach-def with config
- net-attach-def with config file name in /etc/cni/multus.d
- thick plugin case (need to write now?) XXX: TBD
Run pod with network annotation
- text annotation
- text annotation with additional info (ifname)
- json annotation
- json annotation with additional info (ifname) XXX: TBD
Verifying pod network

XXX: TBD

Run ifconfig command in Pod:

# kubectl exec -it multus-multi-net-poc -- ifconfig
eth0      Link encap:Ethernet  HWaddr C6:43:7C:09:B4:9C
          inet addr:10.128.0.4  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:42 (42.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

net0      Link encap:Ethernet  HWaddr 06:21:91:2D:74:B9
          inet addr:192.168.42.3  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::421:91ff:fe2d:74b9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

net1      Link encap:Ethernet  HWaddr D2:94:98:82:00:00
          inet addr:10.56.217.171  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::d094:98ff:fe82:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:120 (120.0 B)  TX bytes:648 (648.0 B)

north     Link encap:Ethernet  HWaddr BE:F2:48:42:83:12
          inet6 addr: fe80::bcf2:48ff:fe42:8312/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1420 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1276 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:95956 (93.7 KiB)  TX bytes:82200 (80.2 KiB)

Interface name	Description
lo	loopback
eth0	weave network interface
net0	Flannel network tap interface
net1	VF0 of NIC 1 assigned to the container by Intel - SR-IOV CNI plugin
north	VF0 of NIC 2 assigned with VLAN ID 210 to the container by SR-IOV CNI plugin

Miscellaneous config

Default Network Readiness Indicator

You may wish for your "default network" (that is, the CNI plugin & its configuration you specify as your default delegate) to become ready before you attach networks with Multus. This is disabled by default and not used unless you add the readiness check option(s) to your CNI configuration file.

For example, if you use Flannel as a default network, the recommended method for Flannel to be installed is via a daemonset that also drops a configuration file in /etc/cni/net.d/. This may apply to other plugins that place that configuration file upon their readiness, hence, Multus uses their configuration filename as a semaphore and optionally waits to attach networks to pods until that file exists.

In this manner, you may prevent pods from crash looping, and instead wait for that default network to be ready.

Only one option is necessary to configure this functionality:

readinessindicatorfile: The path to a file whose existance denotes that the default network is ready.

NOTE: If readinessindicatorfile is unset, or is an empty string, this functionality will be disabled, and is disabled by default.

Logging

You may wish to enable some enhanced logging for Multus, especially during the process where you're configuring Multus and need to understand what is or isn't working with your particular configuration.

Multus will always log via STDERR, which is the standard method by which CNI plugins communicate errors, and these errors are logged by the Kubelet. This method is always enabled.

Writing to a Log File

Optionally, you may have Multus log to a file on the filesystem. This file will be written locally on each node where Multus is executed. You may configure this via the LogFile option in the CNI configuration. By default this additional logging to a flat file is disabled.

For example in your CNI configuration, you may set:

    "LogFile": "/var/log/multus.log",

Logging Level

The default logging level is set as panic -- this will log only the most critical errors, and is the least verbose logging level.

The available logging level values, in decreasing order of verbosity are:

debug
error
panic

You may configure the logging level by using the LogLevel option in your CNI configuration. For example:

    "LogLevel": "debug",

multus-cni Configuration Reference

Following is the example of multus config file, in /etc/cni/net.d/. ("XXX_Note" is just a comment, so you can ignore at your configuration)

{
    "name": "node-cni-network",
    "type": "multus",
    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
    "confDir": "/etc/cni/multus/net.d",
    "cniDir": "/var/lib/cni/multus",
    "binDir": "/opt/cni/bin",
    "logFile": "/var/log/multus.log",
    "logLevel": "debug",
    "capabilities": {
        "portMappings": true
    },    
    "readinessindicatorfile": "",
    
"XXX_Note1":"NOTE: you can set clusterNetwork+defaultNetworks OR delegates!!",
    "clusterNetwork": "defaultCRD",
    "defaultNetworks": ["sidecarCRD", "flannel"],
"XXX_Note2":"NOTE: If you use clusterNetwork/defaultNetworks, delegates is ignored",
    "delegates": [{
        "type": "weave-net",
        "hairpinMode": true
    }, {
        "type": "macvlan",
        ... (snip)
    }]
}

name (string, required): the name of the network
type (string, required): "multus"
confDir (string, optional): directory for CNI config file that multus reads. default /etc/cni/multus/net.d
cniDir (string, optional): Multus CNI data directory, default /var/lib/cni/multus
binDir (string, optional): directory for CNI plugins which multus calls. default /opt/cni/bin
kubeconfig (string, optional): kubeconfig file for the out of cluster communication with kube-apiserver. See the example kubeconfig. If you would like to use CRD (i.e. network attachment definition), this is required
logFile (string, optional): file path for log file. multus puts log in given file
logLevel (string, optional): logging level ("debug", "error" or "panic")
capabilities ({}list, optional): capabilities supported by at least one of the delegates. (NOTE: Multus only supports portMappings capability for now). See the example.
readinessindicatorfile: The path to a file whose existance denotes that the default network is ready

User should chose following parameters combination (clusterNetwork+defaultNetworks or delegates):

clusterNetwork (string, required): default CNI network for pods, used in kubernetes cluster (Pod IP and so on): name of network-attachment-definition, CNI json file name (without extention, .conf/.conflist) or directory for CNI config file
defaultNetworks ([]string, required): default CNI network attachment: name of network-attachment-definition, CNI json file name (without extention, .conf/.conflist) or directory for CNI config file
delegates ([]map,required): number of delegate details in the Multus

Network selection flow of clusterNetwork/defaultNetworks

Multus will find network for clusterNetwork/defaultNetworks as following sequences:

CRD object for given network name
CNI json config file in confDir. Given name should be without extention, like .conf/.conflist. (e.g. "test" for "test.conf")
Directory for CNI json config file. Multus will find alphabetically first file for the network
Multus failed to find network. Multus raise error message

Development

See 'doc/development_info.md' for the development information <- XXX:TBD

Need help

Read Containers Experience Kits
Try our container exp kit demo - KubeCon workshop on Enabling NFV Features in Kubernetes
Join us on #intel-sddsg-slack slack channel and ask question in #general-discussion
You can also email us
Feel free to submit an issue

Please fill in the Questions/feedback - google-form!

Contacts

For any questions about Multus CNI, please reach out on github issue or feel free to contact the developer @kural in our Intel-Corp Slack

Provide feedback

Saved searches

Use saved searches to filter your results more quickly