Implement DSA plugin

[bart0sh]

This plugin discovers DSA work queues in /sys/bus/dsa/devices/

For the queues in "user" mode it looks for /dev/dsa/wqN.Ndevice nodes and correspondent /dev/char/<major>.<minor> symlinks

Here is how the work queue resources are described in the kubectl describe node output:

$ k describe node dhcp0 | grep -A5 Allocatable
Allocatable:
  cpu:                  2
  dsa.intel.com/wq-kernel-dedicated:  1
  dsa.intel.com/wq-mdev-dedicated:    6
  dsa.intel.com/wq-mdev-shared:       5
  dsa.intel.com/wq-user-shared:       45

Ref: #442

[codecov-io]

Codecov Report

Merging #494 (5677540) into master (09d213d) will increase coverage by 0.09%.
The diff coverage is 56.96%.

@@            Coverage Diff             @@
##           master     #494      +/-   ##
==========================================
+ Coverage   56.70%   56.79%   +0.09%     
==========================================
  Files          30       31       +1     
  Lines        2028     2111      +83     
==========================================
+ Hits         1150     1199      +49     
- Misses        807      838      +31     
- Partials       71       74       +3     

Impacted Files	Coverage Δ
pkg/idxd/plugin.go	56.96% <56.96%> (ø)
cmd/sgx_plugin/sgx_plugin.go	76.00% <0.00%> (+2.08%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09d213d...62fb5fc. Read the comment docs.

[mythi]

@bart0sh nice! can you capture this as an open item:
"For example, Intel® DSA devices can use the IDXD kernel driver or DPDK-supported drivers, such as vfio-pci."

we may need to have something similar what QAT plugin does with regards to binding the device to a certain driver?

[bart0sh]

@mythi Thanks for pointing this out. That definitely makes sense for future work.

On the current stage, however, we'll be fine with supporting 3 modes: user, kernel and mdev in my opinion. All of them are provided by idxd kernel driver and idxd is supported by DPDK IOAT Rawdev driver. That can cover most of use cases.

Current draft PR is a POC supporting only user mode. I'm planning to extend it in the next 3-5 days to support kernel and mdev modes + add support for shared and dedicated work queues. So, the device resource would look like dsa.intel.com/<queue>-<mode>-<queue type>, e.g. dsa.intel.com/wq0.0-user-shared. For shared queues amount of resources can be set by plugin command line parameter similar to GPU/VPU plugins.
Does this make sense for you?

[mythi]

@bart0sh is <queue> meaningful for the user? also, is kernel mode something that needs to be exposed to containers?

[bart0sh]

@mythi work queue is the only consumable DSA resource as far as I understand. Other DSA resources (engines and devices) are used mostly for configuration purposes. All userspace libraries I've examined work with queues, i.e. with /dev/dsa/wqN.N device nodes and symlinks.

kernel mode resources don't need to be exposed to the containers as far as I can see. However, they can ensure that either other workflows are not using DSA device in dedicated mode or that amount of other workflows using the device is not higher than plugin would allow in shared mode. It can only happen if all workflows working with the devices only after requesting them.

[rojkov]

As a user I would probably be not interested in a particular queue, e.g. wq0.1, but in just any queue available. Then at runtime I'd configure my software to work with the allocated queue. The name of the queue could be passed with an env variable. Does it make sense?

[bart0sh]

@rojkov users might be interested in using multiple queues on the same device for performance reasons. On a workload level queue names are usually hidden from users by high level userspace APIs.

Configuration of the queues should be done by cluster admin from my point of view. User would just specify which queues are needed for the workflow.

[bart0sh]

@rojkov I should've explained queue naming scheme better. Its format is wqD.N, where D is a device number and N is a queue number.

[mythi]

@bart0sh: @rojkov highlighted the same question what I had in mind: are all the devices the same or would user care about device "topology". It looks that there are N amount of wqD devices so would it make sense to make it: dsa.intel.com/wq0: 2. etc..

[bart0sh]

@mythi That would make sense for the same type of queues. However, I'd not suggest doing this unless we want to restrict plugin to share only one type of queues or something similar.
For example: wq0.0 can be configured as dedicated user queue and wq0.1 - as shared mdev queue. So, for the naming scheme I'm proposing the correspondent resources would be: dsa.intel.com/wq0.0-user-dedicated and dsa.intel.com/wq0.1-mdev-shared respectively.

[rojkov]

It sounds like a node is going to have only one available resource per type (the types are 4 dimensional: 1) device, 2) queue, 3) user/mdev, 4) dedicated/shared) and users are supposed to know which particular queue to use. Right?

Is it possible to have dsa.intel.com/wq0.0-user-dedicated and dsa.intel.com/wq0.0-mdev-shared on the same node? What about different nodes?

[bart0sh]

@rojkov queue can't have 2 types at the same time, so having dsa.intel.com/wq0.0-user-dedicated and dsa.intel.com/wq0.0-mdev-shared or even dsa.intel.com/wq0.0-user-shared is not possible on the same node.
However, it's possible on 2 different nodes.

After giving your idea more thought I think it makes sense to group work queues by their types, i.e. having resourse names dsa.intel.com/wqD-<user or mdev or kernel>-<shared or dedicted> or even without device number: dsa.intel.com/wq-<user or mdev or kernel>-<shared or dedicted>.

[bart0sh]

@mythi @rojkov just discovered that queues can be given a name. Not sure it matters anyhow as device nodes will be named the same way, e.g. /dev/dsa0/wq0.1. Do you think those should be also part of resource names?

Another thing is grouping. DSA engines and queues can be grouped. I was not going to expose this info at least in the first plugin version. What's your opinion?

I'm still thinking that exposing 2 types (shared-dedicated and kernel-user-mdev) should be enough for majority of use cases. Thoughts?

[rojkov]

I'd start with dsa.intel.com/wq-<user or mdev or kernel>-<shared or dedicted> and probably wait for users' feedback.

[mythi]

I'd start with dsa.intel.com/wq-<user or mdev or kernel>-<shared or dedicted> and probably wait for users' feedback.

+1

[mythi]

Another thing is grouping. DSA engines and queues can be grouped. I was not going to expose this info at least in the first plugin version. What's your opinion?

@bart0sh let's start with basic resources as discussed earlier and add configurability/grouping later.