implement DSA plugin

Signed-off-by: Ed Bartosh <[email protected]>

.github/workflows/ci.yaml

@@ -75,6 +75,7 @@ jobs:
           - intel-deviceplugin-operator
           - intel-sgx-plugin
           - intel-sgx-initcontainer
+          - intel-dsa-plugin
 
           # Demo images
           - crypto-perf

README.md

@@ -19,6 +19,7 @@ Table of Contents
     * [QAT device plugin](#qat-device-plugin)
     * [VPU device plugin](#vpu-device-plugin)
     * [SGX device plugin](#sgx-device-plugin)
+    * [DSA device pugin](#dsa-device-plugin)
 * [Device Plugins Operator](#device-plugins-operator)
 * [Demos](#demos)
 * [Developers](#developers)
@@ -163,6 +164,11 @@ operator deployment and NFD is configured to register the SGX EPC memory extende
 Containers requesting SGX EPC resources in the cluster use `sgx.intel.com/epc` resource which is of
 type [memory](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory).
 
+
+### DSA device plugin
+
+The [DSA device plugin](cmd/dsa_plugin/README.md) supports acceleration using the Intel Data Streaming accelerator(DSA).
+
 ## Device Plugins Operator
 
 Currently the operator has limited support for the QAT, GPU, FPGA and SGX device plugins:

build/docker/intel-dsa-plugin.Dockerfile

@@ -0,0 +1,38 @@
+# CLEAR_LINUX_BASE and CLEAR_LINUX_VERSION can be used to make the build
+# reproducible by choosing an image by its hash and installing an OS version
+# with --version=:
+# CLEAR_LINUX_BASE=clearlinux@sha256:b8e5d3b2576eb6d868f8d52e401f678c873264d349e469637f98ee2adf7b33d4
+# CLEAR_LINUX_VERSION="--version=29970"
+#
+# This is used on release branches before tagging a stable version.
+# The master branch defaults to using the latest Clear Linux.
+ARG CLEAR_LINUX_BASE=clearlinux/golang:latest
+
+FROM ${CLEAR_LINUX_BASE} as builder
+
+ARG CLEAR_LINUX_VERSION=
+
+RUN swupd update --no-boot-update ${CLEAR_LINUX_VERSION}
+
+ARG DIR=/intel-device-plugins-for-kubernetes
+ARG GO111MODULE=on
+WORKDIR $DIR
+COPY . .
+
+RUN mkdir /install_root \
+    && swupd os-install \
+    ${CLEAR_LINUX_VERSION} \
+    --path /install_root \
+    --statedir /swupd-state \
+    --no-boot-update \
+    && rm -rf /install_root/var/lib/swupd/*
+
+RUN cd cmd/dsa_plugin; GO111MODULE=${GO111MODULE} go install; cd -
+RUN chmod a+x /go/bin/dsa_plugin \
+    && install -D /go/bin/dsa_plugin /install_root/usr/local/bin/intel_dsa_device_plugin \
+    && install -D ${DIR}/LICENSE /install_root/usr/local/share/package-licenses/intel-device-plugins-for-kubernetes/LICENSE \
+    && scripts/copy-modules-licenses.sh ./cmd/dsa_plugin /install_root/usr/local/share/
+
+FROM scratch as final
+COPY --from=builder /install_root /
+ENTRYPOINT ["/usr/local/bin/intel_dsa_device_plugin"]

cmd/dsa_plugin/README.md

@@ -0,0 +1,130 @@
+# Intel DSA device plugin for Kubernetes
+
+Table of Contents
+
+* [Introduction](#introduction)
+* [Installation](#installation)
+    * [Deploy with pre-built container image](#deploy-with-pre-built-container-image)
+    * [Getting the source code](#getting-the-source-code)
+    * [Verify node kubelet config](#verify-node-kubelet-config)
+    * [Deploying as a DaemonSet](#deploying-as-a-daemonset)
+        * [Build the plugin image](#build-the-plugin-image)
+        * [Deploy plugin DaemonSet](#deploy-plugin-daemonset)
+    * [Deploy by hand](#deploy-by-hand)
+        * [Build the plugin](#build-the-plugin)
+        * [Run the plugin as administrator](#run-the-plugin-as-administrator)
+    * [Verify plugin registration](#verify-plugin-registration)
+
+## Introduction
+
+The DSA device plugin for Kubernetes supports acceleration using the Intel Data Streaming accelerator(DSA).
+
+The DSA plugin discovers DSA work queues and presents them as a node resources.
+
+## Installation
+
+The following sections detail how to obtain, build, deploy and test the DSA device plugin.
+
+Examples are provided showing how to deploy the plugin either using a DaemonSet or by hand on a per-node basis.
+
+### Deploy with pre-built container image
+
+[Pre-built images](https://hub.docker.com/r/intel/intel-dsa-plugin)
+of this component are available on the Docker hub. These images are automatically built and uploaded
+to the hub from the latest master branch of this repository.
+
+Release tagged images of the components are also available on the Docker hub, tagged with their
+release version numbers in the format `x.y.z`, corresponding to the branches and releases in this
+repository. Thus the easiest way to deploy the plugin in your cluster is to run this command
+
+```bash
+$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/dsa_plugin?ref=<REF>
+daemonset.apps/intel-dsa-plugin created
+```
+
+Where `<REF>` needs to be substituted with the desired git ref, e.g. `master`.
+
+Nothing else is needed. But if you want to deploy a customized version of the plugin read further.
+
+### Getting the source code
+
+```bash
+$ export INTEL_DEVICE_PLUGINS_SRC=/path/to/intel-device-plugins-for-kubernetes
+$ git clone https://github.com/intel/intel-device-plugins-for-kubernetes ${INTEL_DEVICE_PLUGINS_SRC}
+```
+
+### Verify node kubelet config
+
+Every node that will be running the dsa plugin must have the
+[kubelet device-plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)
+configured. For each node, check that the kubelet device plugin socket exists:
+
+```bash
+$ ls /var/lib/kubelet/device-plugins/kubelet.sock
+/var/lib/kubelet/device-plugins/kubelet.sock
+```
+
+### Deploying as a DaemonSet
+
+To deploy the dsa plugin as a daemonset, you first need to build a container image for the
+plugin and ensure that is visible to your nodes.
+
+#### Build the plugin image
+
+The following will use `docker` to build a local container image called
+`intel/intel-dsa-plugin` with the tag `devel`.
+
+The image build tool can be changed from the default `docker` by setting the `BUILDER` argument
+to the [`Makefile`](Makefile).
+
+```bash
+$ cd ${INTEL_DEVICE_PLUGINS_SRC}
+$ make intel-dsa-plugin
+...
+Successfully tagged intel/intel-dsa-plugin:devel
+```
+
+#### Deploy plugin DaemonSet
+
+You can then use the [example DaemonSet YAML](/deployments/dsa_plugin/base/intel-dsa-plugin.yaml)
+file provided to deploy the plugin. The default kustomization that deploys the YAML as is:
+
+```bash
+$ kubectl apply -k deployments/dsa_plugin
+daemonset.apps/intel-dsa-plugin created
+```
+
+### Deploy by hand
+
+For development purposes, it is sometimes convenient to deploy the plugin 'by hand' on a node.
+In this case, you do not need to build the complete container image, and can build just the plugin.
+
+#### Build the plugin
+
+First we build the plugin:
+
+```bash
+$ cd ${INTEL_DEVICE_PLUGINS_SRC}
+$ make dsa_plugin
+```
+
+#### Run the plugin as administrator
+
+Now we can run the plugin directly on the node:
+
+```bash
+$ sudo -E ${INTEL_DEVICE_PLUGINS_SRC}/cmd/dsa_plugin/dsa_plugin
+device-plugin registered
+```
+
+### Verify plugin registration
+
+You can verify the plugin has been registered with the expected nodes by searching for the relevant
+resource allocation status on the nodes:
+
+```bash
+$ kubectl get nodes -o=jsonpath="{range .items[*]}{.metadata.name}{'\n'}{' i915: '}{.status.allocatable.dsa\.intel\.com/*}{'\n'}"
+master
+  dsa.intel.com/wq-user-dedicated:  1
+  dsa.intel.com/wq-user-shared:     1
+```

cmd/dsa_plugin/dsa_plugin.go

@@ -0,0 +1,55 @@
+// Copyright 2020 Intel Corporation. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	"flag"
+	"os"
+
+	dpapi "github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin"
+	"github.com/intel/intel-device-plugins-for-kubernetes/pkg/idxd"
+
+	"k8s.io/klog"
+)
+
+const (
+	// Device plugin settings.
+	namespace = "dsa.intel.com"
+	// SysFS directory.
+	sysfsDir = "/sys/bus/dsa/devices"
+	// Device directories.
+	devDir = "/dev/dsa"
+	// Glob pattern for the state sysfs entry.
+	statePattern = "/sys/bus/dsa/devices/dsa*/wq*/state"
+)
+
+func main() {
+	var sharedDevNum int
+
+	flag.IntVar(&sharedDevNum, "shared-dev-num", 1, "number of containers sharing the same work queue")
+	flag.Parse()
+
+	if sharedDevNum < 1 {
+		klog.Warning("The number of containers sharing the same work queue must be greater than zero")
+		os.Exit(1)
+	}
+
+	plugin := idxd.NewDevicePlugin(sysfsDir, statePattern, devDir, sharedDevNum)
+	if plugin == nil {
+		klog.Fatal("Cannot create device plugin, please check above error messages.")
+	}
+	manager := dpapi.NewManager(namespace, plugin)
+	manager.Run()
+}

deployments/dsa_plugin/base/intel-dsa-plugin.yaml

@@ -0,0 +1,53 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: intel-dsa-plugin
+  labels:
+    app: intel-dsa-plugin
+spec:
+  selector:
+    matchLabels:
+      app: intel-dsa-plugin
+  template:
+    metadata:
+      labels:
+        app: intel-dsa-plugin
+    spec:
+      containers:
+      - name: intel-dsa-plugin
+        env:
+          - name: NODE_NAME
+            valueFrom:
+              fieldRef:
+                fieldPath: spec.nodeName
+        image: intel/intel-dsa-plugin:devel
+        imagePullPolicy: IfNotPresent
+        securityContext:
+          readOnlyRootFilesystem: true
+        volumeMounts:
+        - name: devfs
+          mountPath: /dev/dsa
+          readOnly: true
+        - name: chardevs
+          mountPath: /dev/char
+          readOnly: true
+        - name: sysfs
+          mountPath: /sys/bus/dsa
+          readOnly: true
+        - name: kubeletsockets
+          mountPath: /var/lib/kubelet/device-plugins
+      volumes:
+      - name: devfs
+        hostPath:
+          path: /dev/dsa
+      - name: chardevs
+        hostPath:
+          path: /dev/char
+      - name: sysfs
+        hostPath:
+          path: /sys/bus/dsa
+      - name: kubeletsockets
+        hostPath:
+          path: /var/lib/kubelet/device-plugins
+      nodeSelector:
+        kubernetes.io/arch: amd64

deployments/dsa_plugin/base/kustomization.yaml

@@ -0,0 +1,2 @@
+resources:
+  - intel-dsa-plugin.yaml

deployments/dsa_plugin/kustomization.yaml

@@ -0,0 +1,2 @@
+bases:
+  - base

deployments/dsa_plugin/overlays/namespace_kube-system/add-namespace-kube-system.yaml

@@ -0,0 +1,5 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: intel-dsa-plugin
+  namespace: kube-system

deployments/dsa_plugin/overlays/namespace_kube-system/kustomization.yaml

@@ -0,0 +1,4 @@
+bases:
+  - ../../base
+patches:
+  - add-namespace-kube-system.yaml