Nvidia driver loading appears to be different in Cobia

[rowanmoul]

In the main readme, it suggests creating a startup task to load the nvidia drivers, however the command given there is trying to run a binary that doesn't exist on my Cobia system (/usr/bin/nvidia-modprobe) . I don't have any bluefin systems to compare against, but I assume that binary existed on bluefin?

jailmaker/README.md

Line 143 in 56d53a1

[ ! -f /dev/nvidia-uvm ] && modprobe nvidia-current-uvm && /usr/bin/nvidia-modprobe -c0 -u

That said, it appears that running nvidia-modprobe is not needed. If you load the kernel module (modprobe nvidia-current-uvm), the necessary device nodes will be created when jailmaker runs nvidia-smi -f /dev/null as part of the nvidia passthrough method on jail startup.

jailmaker/jlmkr.py

Line 111 in 56d53a1

subprocess.run(['nvidia-smi', '-f', '/dev/null'], check=True)

[Jip-Hop]

Looks like you got it working. Would you mind adding these instructions for Cobia and creating a pull request? I don't have a nvidia GPU so can't test this myself. Please also mention the version number besides Cobia in the readme. Thank you!

[rowanmoul]

I haven't actually got everything running, but I expect it to work since the device nodes are created in time for jailmaker to pass them through. The next step is to get docker running inside the jail and pass the gpu into the containers that will use it and verify that they are able to do so. I will do some more testing, and maybe I'll even install bluefin in a VM to compare against. It's entirely possible that nvidia-modprobe was never included in SCALE, but the startup task still did enough by loading the kernel module.

[rowanmoul]

I dug through the bluefin installer disk image and found the root filesystem squashfs. It does indeed include /usr/bin/nvidia-modprobe.
It looks like it is installable as a separate package, not bundled with the driver (this was the case under Debian Bullseye too), so I guess IX intentionally removed it for one reason or another.
I haven't had a chance to work on the rest of it just yet.

[Jip-Hop]

Could you please try the jlmkr.py file from the develop branch? I think it should now load the nvidia kernel module automatically.

[rowanmoul]

Sorry for the long delay, I haven't been able to put as much time into this as I would have liked but I finally got back to it. I can confirm that your changes in the develop branch are currently enough (in Cobia 2023.10.1.3) to load the drivers and pass the GPU through without any extra startup scripts.
One other thing that I found. If someone is passing through their Nvidia GPU so that they can use it in a docker container, they will need to install the nvidia-container-toolkit in the jail. I followed this documentation: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-with-apt
Following that, I installed version 1.14.4 of the toolkit (at the time of writing). The TrueNAS Scale version noted above has v1.13.4 installed, but since this isn't a driver component, I don't think it needs to perfectly match, like was discovered here. If that ever changes, we may want to look at bind mounting the container toolkit components too.

While I am usually more comfortable in Arch Linux (that is where I first got started running Linux), I used a debian bookworm based jail for this so that (in theory) shared library versions would be in sync with those found on the host.