-
Notifications
You must be signed in to change notification settings - Fork 589
how to use
Tomofumi Hayashi edited this page Nov 7, 2018
·
9 revisions
Kubelet must be configured to run with the CNI network plugin. Edit /etc/kubernetes/kubelet file and add --network-plugin=cni flags in KUBELET\_OPTS as shown below:
KUBELET_OPTS="...
--network-plugin-dir=/etc/cni/net.d
--network-plugin=cni
"
You could copy binary directory or could use daemonset yaml in multus repository.
- copy multus binary
Copy multus binary into CNI binary directory, usually /opt/cni/bin.
# Execute following commands at all Kubernetes nodes (i.e. master and minions)
$ cp multus /opt/cni/bin
- use daemonset
As Quickstart, you could apply as such:
# Execute following command at Kubernetes master
$ cat ./images/{multus-daemonset.yml,flannel-daemonset.yml} | kubectl apply -f -
You put CNI config file in /etc/cni/net.d. Kubernetes CNI runtime uses the alphabetically first file in the directory. ("Note1", "Note2" are just comments, you can remove them at your configuration)
# Execute following commands at all Kubernetes nodes (i.e. master and minions)
$ mkdir -p /etc/cni/net.d
$ cat >/etc/cni/net.d/30-multus.conf <<EOF
{
"name": "multus-cni-network",
"type": "multus",
"readinessindicatorfile": "/var/run/flannel/subnet.env",
"delegates": [
{
"Note1": "This is example, wrote your CNI config in delegates",
"Note2": "If you use flannel, you also need to run flannel daemonset before!",
"type": "flannel",
"name": "flannel.1",
"delegate": {
"isDefaultGateway": true
}
}
],
"kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
}
EOF
For the detail, please take a look into Configuration Reference
You can use "clusterNetwork"/"defaultNetworks" instead of "delegates", see for the detail
As above config, you need to set "kubeconfig" in the config file for NetworkAttachmentDefinition(CRD).
In case of "delegates", the first delegates network will be used for "Pod IP". Otherwise, "clusterNetwork" will be used for "Pod IP".
Create resources for multus to access CRD objects as folloiwng command:
# Execute following commands at Kubernetes master
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: multus
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: multus
rules:
- apiGroups: ["k8s.cni.cncf.io"]
resources:
- '*'
verbs:
- '*'
- apiGroups:
- ""
resources:
- pods
- pods/status
verbs:
- get
- update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: multus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: multus
subjects:
- kind: ServiceAccount
name: multus
namespace: kube-system
EOF
Create kubeconfig at master node as following commands:
# Execute following command at Kubernetes master
$ mkdir -p /etc/cni/multus.d
$ SERVICEACCOUNT_CA=$(kubectl get secrets -n=kube-system -o json | jq -r '.items[]|select(.metadata.annotations."kubernetes.io/service-account.name"=="multus")| .data."ca.crt"')
$ SERVICEACCOUNT_TOKEN=$(kubectl get secrets -n=kube-system -o json | jq -r '.items[]|select(.metadata.annotations."kubernetes.io/service-account.name"=="multus")| .data.token' | base64 -d )
$ KUBERNETES_SERVICE_PROTO=$(kubectl get all -o json | jq -r .items[0].spec.ports[0].name)
$ KUBERNETES_SERVICE_HOST=$(kubectl get all -o json | jq -r .items[0].spec.clusterIP)
$ KUBERNETES_SERVICE_PORT=$(kubectl get all -o json | jq -r .items[0].spec.ports[0].port)
$ cat > /etc/cni/net.d/multus.d/multus.kubeconfig <<EOF
# Kubeconfig file for Multus CNI plugin.
apiVersion: v1
kind: Config
clusters:
- name: local
cluster:
server: ${KUBERNETES_SERVICE_PROTOCOL:-https}://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}
certificate-authority-data: ${SERVICEACCOUNT_CA}
users:
- name: multus
user:
token: "${SERVICEACCOUNT_TOKEN}"
contexts:
- name: multus-context
context:
cluster: local
user: multus
current-context: multus-context
EOF
Copy /etc/cni/net.d/multus.d/multus.kubeconfig into other Kubernetes nodes
Note: Recommend to exec 'chmod 600 /etc/cni/net.d/multus.d/multus.kubeconfig' to keep secure
$ scp /etc/cni/net.d/multus.d/multus.kubeconfig ...
If you use daemonset to install multus, skip this section and go to "Create network attachment"
Create CRD definition in Kubernetes as following command at master node:
# Execute following command at Kubernetes master
$ cat <<EOF | kubectl create -f -
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: network-attachment-definitions.k8s.cni.cncf.io
spec:
group: k8s.cni.cncf.io
version: v1
scope: Namespaced
names:
plural: network-attachment-definitions
singular: network-attachment-definition
kind: NetworkAttachmentDefinition
shortNames:
- net-attach-def
validation:
openAPIV3Schema:
properties:
spec:
properties:
config:
type: string
EOF
Following command creates NetworkAttachmentDefinition. CNI config is in config: field.
# Execute following command at Kubernetes master
$ cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf-1
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "eth1",
"mode": "bridge",
"ipam": {
"type": "host-local",
"ranges": [
[ {
"subnet": "10.10.0.0/16",
"rangeStart": "10.10.1.20",
"rangeEnd": "10.10.3.50",
"gateway": "10.10.0.254"
} ]
]
}
}'
EOF
If NetworkAttachmentDefinition has no spec, multus find a file in defaultConfDir ('/etc/cni/multus/net.d', with same name in the 'name' field of CNI config.
# Execute following command at Kubernetes master
$ cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf-2
EOF
# Execute following commands at all Kubernetes nodes (i.e. master and minions)
$ cat <<EOF > /etc/cni/multus/net.d/macvlan2.conf
{
"cniVersion": "0.3.0",
"type": "macvlan",
"name": "macvlan-conf-2",
"master": "eth1",
"mode": "bridge",
"ipam": {
"type": "host-local",
"ranges": [
[ {
"subnet": "11.10.0.0/16",
"rangeStart": "11.10.1.20",
"rangeEnd": "11.10.3.50",
"gateway": "11.10.0.254"
} ]
]
}
}
# Execute following command at Kubernetes master
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: pod-case-01
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf-1, macvlan-conf-2
spec:
containers:
- name: pod-case-01
image: docker.io/centos/tools:latest
command:
- /sbin/init
EOF
You can also specify interface name as adding @<ifname>.
# Execute following command at Kubernetes master
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: pod-case-02
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf-1@macvlan1
spec:
containers:
- name: pod-case-02
image: docker.io/centos/tools:latest
command:
- /sbin/init
EOF
# Execute following command at Kubernetes master
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: pod-case-03
annotations:
k8s.v1.cni.cncf.io/networks: '[
{ "name" : "macvlan-conf-1" },
{ "name" : "macvlan-conf-2" }
]'
spec:
containers:
- name: pod-case-03
image: docker.io/centos/tools:latest
command:
- /sbin/init
EOF
You can also specify interface name as adding "interfaceRequest": "<ifname>".
# Execute following command at Kubernetes master
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: pod-case-04
annotations:
k8s.v1.cni.cncf.io/networks: '[
{ "name" : "macvlan-conf-1",
"interfaceRequest": "macvlan1" },
{ "name" : "macvlan-conf-2" }
]'
spec:
containers:
- name: pod-case-04
image: docker.io/centos/tools:latest
command:
- /sbin/init
EOF
Following the example of ip -d address output of above pod, "pod-case-04":
# Execute following command at Kubernetes master
$ kubectl exec -it pod-case-04 -- ip -d address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 0a:58:0a:f4:02:06 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
veth numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet 10.244.2.6/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::ac66:45ff:fe7c:3a19/64 scope link
valid_lft forever preferred_lft forever
4: macvlan1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 4e:6d:7a:4e:14:87 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
macvlan mode bridge numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet 10.10.1.22/16 scope global macvlan1
valid_lft forever preferred_lft forever
inet6 fe80::4c6d:7aff:fe4e:1487/64 scope link
valid_lft forever preferred_lft forever
5: net2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 6e:e3:71:7f:86:f7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
macvlan mode bridge numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet 11.10.1.22/16 scope global net2
valid_lft forever preferred_lft forever
inet6 fe80::6ce3:71ff:fe7f:86f7/64 scope link
valid_lft forever preferred_lft forever
| Interface name | Description |
|---|---|
| lo | loopback |
| eth0 | Default network interface (flannel) |
| macvlan1 | macvlan interface (macvlan-conf-1) |
| net2 | macvlan interface (macvlan-conf-2) |