How to get GPU passthrough working on WSL2 with Windows 11 as hosts , Ubuntu 22.10 as guest os and FreeBSD virtualized with qemu-kvm

[Marietto2008]

Windows Version

Windows 11 PRO Version 10.0.22000.2176

WSL Version

Versione WSL: 1.3.11.0

Are you using WSL 1 or WSL 2?

• WSL 2
• WSL 1

Kernel Version

5.15.90.2-3

Distro Version

Ubuntu 22.10

Other Software

No response

Repro Steps

After a week of hard work,I've just configured my WSL2 subsystem as I want : I have enabled the ZFS support,I can start the graphical applications directly from the Ubuntu command line ; I've installed the nvidia driver and the CUDA libraries within Ubuntu and finally I've configured qemu/kvm and virt-manager so that I can virtualize FreeBSD on top of Ubuntu. Now,the problem is : how can I "tell" to FreeBSD to run Xorg with one of the GPUs that I have on the PC. Actually I have two GPUs on the system. The GTX 1060,that I use as default,where I have attached my monitor and the RTX 2080 ti,that usually is used to be passed into another OS. When I'm using Linux as host os,I would like to virtualize Windows and I passthru the RTX 2080 ti in Windows using qemu-kvm and virt-manager ; when I use FreeBSD as host os,I want to pass thru the RTX 2080 ti to a Linux guest os using bhyve. But when I use Windows 11 as host os and Ubuntu as guest,how can I pass thru my RTX 2080 ti with qemu-kvm and virt-manager to FreeBSD for example ? Below you can see all the PCI devices that are recognized within Ubuntu 22.04 at the moment :

# lspci
0301:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
05c0:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
117a:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
242b:00:00.0 3D controller: Microsoft Corporation Basic Render Driver
4e53:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
5dc2:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
6164:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
6251:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
7c22:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
840d:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
94be:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
ad48:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
ae16:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
af96:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
baff:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
d5cc:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
d9a5:00:00.0 System peripheral: Red Hat, Inc. Virtio file system (rev 01)
de65:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio console (rev 01)
fd8d:00:00.0 3D controller: Microsoft Corporation Basic Render Driver

the only two virtual graphics adapter that can be used are the following :

 242b:00:00.0 3D controller: Microsoft Corporation Basic Render Driver
 fd8d:00:00.0 3D controller: Microsoft Corporation Basic Render Driver

I've already tried to configure Xorg in FreeBSD declaring inside the xorg.conf the one and only graphic "card" I see,the "00:01.0 VGA Compatible Controller RedHat QXL paravirtual graphic card" :

like this :

unfortunately it is not accepted as a valid graphic card :

What shoud I do to have a working graphic enviroment running on FreeBSD ? Maybe should I try to pass-thru one of these two graphic adapters

242b:00:00.0 3D controller: Microsoft Corporation Basic Render Driver
fd8d:00:00.0 3D controller: Microsoft Corporation Basic Render Driver

in virt-manager like they were a physical graphic cards ? I'm very doubtful that it works like this.

Anyway,these are the physical GPUs that are on my system :

both are recognized on Windows 11 and on Ubuntu :

# nvidia-smi

+--------------------------------------------------------------------------------------------+
| NVIDIA-SMI 536.40                 Driver Version: 536.40       CUDA Version: 12.2               
|-----------------------------------------+----------------------+---------------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC          
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M.  
|                                         |                      |               MIG M.                                              
|=========================================+=========================================
|   0  NVIDIA GeForce GTX 1060 3GB  WDDM  | 00000000:01:00.0  On |                  N/A  
| 57%   41C    P2              25W / 120W |    710MiB /  3072MiB |      0%      Default            
|                                         |                      |                  N/A                                                  
+-----------------------------------------+----------------------+---------------------------+
|   1  NVIDIA GeForce RTX 2080 Ti   WDDM  | 00000000:02:00.0 Off |                  N/A      
| 31%   40C    P8              21W / 250W |     74MiB / 11264MiB |      0%                              
+_________________________________________+______________________+____________________________

Some background :

    Windows Version = Windows 11 PRO Version 10.0.22000.2176

    WSL2 Version = 1.3.11.0

    Kernel Version = 5.15.90.2-3-microsoft-standard

    Distro Version = Ubuntu 22.10

    Version kernel: 5.15.90.2-3

    Version WSLg: 1.0.54

    Version MSRDC: 1.2.4240

    Version Direct3D: 1.608.2-61064218

    Version DXCore: 10.0.25880.1000-230602-1350.main

Expected Behavior

That I can use the RTX 2080 ti within FreeBSD os as default gpu.

Actual Behavior

Unfortunately my gpus aren't exposed on the WSL2 sub system.

Diagnostic Logs

No response

[moekhalil]

IIRC, WSL2 uses GPU partitioning (GPU-PV) instead of PCIe passthrough.

nvidia-SMI works and CUDA works via that GPU "Wrapper", but it's not actual passthrough.

If the host is required to be Windows, I believe Hyper-V (and maybe -- depending on needs -- Windows Server) might be needed here.

[Marietto2008]

Hello.

I'm trying to use Ubuntu VM on a Hyper-V with Microsoft GPU-P support. The tutorial that I have used to reach the goal is the one below :

https://github.com/brokeDude2901/dxgkrnl_ubuntu

The result that I have achieved has been to enable the nVidia driver and CUDA libraries within an Ubuntu 20.04 VM,but Blender Cycles does not recognize my GPUs :

Do you know the reason ? Instead,Blender Cycles recognizes at least one GPU (I have 2 GPUs,but it recognizes only one : but I have an idea about the reason : on the script I have declared 16 GB of memory instead of the default,32 GB) if the VM is Windows 11 :

Something is missing or it does not work well if the VM is Ubuntu 20.04. I know what it is. While Windows 11 recognizes the GPU directly on the Device Manager,Ubuntu does not recognise it on the terminal when I issue the command "lspci" as you can see below :

root@marietto-HyperV:/home/marietto# lspci

7ea5:00:00.0 3D controller: Microsoft Corporation Device 008e
ec5c:00:00.0 3D controller: Microsoft Corporation Device 008e

what I can do to allow Blender Cycles to recognize my GPU as a good GPU for rendering ?

[moekhalil]

@Marietto2008 -- Have you tried using Ubuntu under WSL2 with Cuda installed?

Just installed Blender on my WSL2 Ubuntu distro, it is showing CUDA GPU just fine.
Windows 11 23593 + WSL2 + Ubuntu 23.04

[Marietto2008]

yes,on the WSL2 my GPUs are recognized by Blender Cycles :

but look at the bottom of the photo : OpenGL renderer string should shows the name of my GPU,but it isn't there.

[Marietto2008]

I don't understand. Why my GPUs are recognized within Ubuntu by BLender cycles but it is not inside Ubuntu installed with Hyper-V ?They don't use the same technology ?

[moekhalil]

No. WSL2 does not support hardware pass-thru like Hyper-V. 
The Nvidia + Cuda drivers on WSL2 are just lightweight wrappers/linkers that send off GPU instructions to the windows graphics driver through symlinking: /lib/wsl/lib/libcuda or libnvidia to C:\Windows\System32\lxss\lib\libcuda or libnvidia

[Marietto2008]

ok,but my motherboard does not support SR-IOV,I can't pass thru my graphic card using the Discrete Device Assignement. For this reason I'm trying to share the GPU between the host os (WIndows 11) and the guest os (in this case Ubuntu,because if I use Windows 11 as guest os,it works). This is the tutorial that allows to do that :

https://github.com/brokeDude2901/dxgkrnl_ubuntu

unfortunately Blender Cycles can't render anything because it does not recognize the GPUs,even if the cuda libraries works os,they are able to recognize both my GPUs. I don't know if it is natural of I'm making some mistake. This bring us to one of my old comment :

#10294 (comment)

[Marietto2008]

# glxinfo | grep "OpenGL renderer string" display D3D12 (My GPU Name) :