Skip to content

Latest commit

 

History

History
262 lines (239 loc) · 8.44 KB

README.md

File metadata and controls

262 lines (239 loc) · 8.44 KB
Raw

Device Passthrough

Device passthrough for TDX guests requires updates to the host kernel and QEMU. The patch sets below track the bleeding edge state of pre-release Kernel and QEMU development branches. They are provided here for preview and test purposes only, and may be update frequently. Any feedback or issues should be reported as a reply to the latest upstream posting of the given patch. See the linux-coco, linux-kvm, and qemu-devel archives on lore.kernel.org for the most recent public posting of these patches. Note that there some temporary workarounds and shortcuts included while formal replacements are in development.

Wait for these patches to be accepted by their respective upstream projects, Next, wait for new releases of those upstream project versions to be picked up by your chosen Linux distribution provider before using them for any production use case.

Host setup for device passthrough

  • This guide assume the host machine is already setup with wiki. Complete the steps till https://github.com/canonical/tdx#44-verify-intel-tdx-is-enabled-on-host-os.

    mkdir ~/nvidia_setup
git config --global user.email [email protected]
git config --global user.name "Your Name"
cd ~/nvidia_setup
git clone https://github.com/amathew3/devicepassthrough.git
git clone https://github.com/canonical/tdx.git
cd tdx/
git checkout -b tag2.0 2.0
cp ../devicepassthrough/canonical_passthrough.patch .
git apply canonical_passthrough.patch
cd ..

  • tdx-linux - Contains patch for kvm and qemu:

  git clone -b device-passthrough https://github.com/intel/tdx-linux.git

  • tdx-kvm - Contains an mbox file with host side TDX patches for KVM. This can be applied using:

    git clone -b kvm-coco-queue-20240512 https://git.kernel.org/pub/scm/linux/kernel/git/vishal/kvm.git
cd ~/nvidia_setup/kvm
git checkout -b 07082024 origin/kvm-coco-queue-20240807
cp -rf ../tdx-linux/tdx-kvm .
git am --empty=drop tdx-kvm/tdx_kvm_baseline_<sha>.mbox

    The baseline (as noted in the filename above) for these patches is the kvm-coco-queue branch in the KVM repo. Since this is a rebasing branch, the commit is not guaranteed to be present in kvm.git. A snapshot of the older version of kvm-coco-queue can be found, for example, here.

  • tdx-qemu - Contains an mbox file with TDX patches for QEMU. This can be applied using:

    cd ~/nvidia_setup
git clone https://gitlab.com/qemu-project/qemu
cd qemu
git checkout -b rc0 v9.1.0-rc0
cp -rf ../tdx-linux/tdx-qemu .
git am --empty=drop tdx-qemu/tdx_qemu_baseline_<sha>.mbox

    The baseline for these should be found in qemu.git.

  • tdx-edk2 - The EDK2 enabling we need so far is already in an upstream tag. This simply contains a file with the stable tag name. from the EDK2 repo

Building the components

Installing the packages needed for kernel build.

  sudo apt update
  sudo apt install -y build-essential libncurses-dev bison flex libssl-dev libelf-dev debhelper-compat=12 meson ninja-build \ 
  libglib2.0-dev python3-pip nasm iasl
  cd ~/nvidia_setup/kvm
  make menuconfig
  • Save the default config file.
  • Check the below config values.

  • tdx-kvm:

    • Config options:

      Enables:
CONFIG_INTEL_TDX_HOST
CONFIG_KVM
CONFIG_KVM_INTEL
CONFIG_TDX_GUEST_DRIVER
CONFIG_HYPERV

Disables:
CONFIG_KEXEC
CONFIG_CRASH_DUMP

    • Build and install the kernel on the host machine. Add module options:

      make -j$(nproc)
make modules -j$(nproc)
sudo make modules_install 
sudo make install
MID=$(sudo awk '/Advanced options for Ubuntu/{print $(NF-1)}' /boot/grub/grub.cfg | cut -d\' -f2)
KID=$(sudo awk "/with Linux $KERNELVER/"'{print $(NF-1)}' /boot/grub/grub.cfg | cut -d\' -f2 | head -n1)
sudo grub-editenv /boot/grub/grubenv set saved_entry="${MID}>${KID}"
sudo update-grub

  • tdx-qemu:

    • Config options: 
      cd ~/nvidia_setup/qemu
./configure --enable-slirp --enable-kvm --target-list=x86_64-softmmu
make -j$(nproc)
sudo make install

  • tdx-edk2:

    • Clone the EDK2 repo and checkout the tag as noted in this repo.
    • Build the OVMF image: 
      cd ~/nvidia_setup
git clone -b edk2-stable202405 https://github.com/tianocore/edk2
cd edk2
git submodule update --init
  • Copy the below content to a file build_ovmf.sh 
    #!/bin/bash
rm -rf Build
make -C BaseTools
. edksetup.sh
cat <<-EOF > Conf/target.txt
	ACTIVE_PLATFORM = OvmfPkg/OvmfPkgX64.dsc
	TARGET = DEBUG
	TARGET_ARCH = X64
	TOOL_CHAIN_CONF = Conf/tools_def.txt
	TOOL_CHAIN_TAG = GCC6
	BUILD_RULE_CONF = Conf/build_rule.txt
	MAX_CONCURRENT_THREAD_NUMBER = $(nproc)
EOF
build clean
build

if [ ! -f Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd ]; then
	echo "Build failed, OVMF.fd not found"
	exit 1
fi

cp Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd ./OVMF.fd
  • Run the script. 
     chmod +x ./build_ovmf.sh
 ./build_ovmf.sh
 sudo reboot
  • Check the system booted up with newly built kernel. 
    uname -r
# Expected result
6.10.0-rc7+

Libvirt cofiguration for using the newly built qemu.

  • Apply the following settings to the file /etc/libvirt/qemu.conf
 sudo vi /etc/libvirt/qemu.conf
 user = <your_user_name>
 group = <your_group>
 dynamic_ownership = 0
 security_driver = "none"
  • Restart the libvirtd service.
  sudo aa-teardown /usr/sbin/libvirtd
  systemctl restart libvirtd

Enabling CC mode on GPU.

  cd ~/nvidia_setup
  git clone https://github.com/NVIDIA/nvtrust.git
  cd ~/nvidia_setup/nvtrust 
  git submodule update --init 
  cd host_tools/python

  sudo python3 ./nvidia_gpu_tools.py --gpu-name=H100 --query-cc-mode
  sudo python3 ./nvidia_gpu_tools.py --gpu-name=H100 --set-cc-mode=on --reset-after-cc-mode-switch
  lspci -d 10de: -nn
  sudo modprobe vfio
  sudo modprobe vfio_pci
  sudo sh -c "echo 10de 2331 > /sys/bus/pci/drivers/vfio-pci/new_id"

  • Please use device number returned in lspci -d 10de: -nn for setting the new_id.

  • Follow the steps mentioned in wiki to prepare a guest image.

  cd ~/nvidia_setup/tdx/guest_tools/image
  sudo ./create-td-image.sh

Boot TD with GPU passthrough

  • To passthrough the GPU card to TD, say 38:00.0, run the run_td.sh script with -d as an argument. 
    cd ~/nvidia_setup/tdx/guest-tools/
sudo ./run_td.sh -d
  • If you are login as normal user, use sudo for all the commands need root privilage

Enabling LKCA on the TDVM

vi /etc/modprobe.d/nvidia-lkca.conf
install nvidia /sbin/modprobe ecdsa_generic; /sbin/modprobe ecdh; /sbin/modprobe --ignore-install nvidia
update-initramfs -u

Install NVIDIA Driver and CUDA Toolkit

  • Setup proxy inside TDVM, if the environment is behind a proxy
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb 
apt-get update
apt-get -y install cuda-toolkit-12-5
apt install -y nvidia-driver-550-server-open
reboot
  • After successful installation for nvidia driver and reboot, check whether nvidia-persistenced is running 
    ps -aux | grep nvidia-persistenced
  • Modify the service that automatically launches nvidia-persistenced 
    vi /usr/lib/systemd/system/nvidia-persistenced.service
# Change:
ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --no-persistence-mode --verbose
# to this:
ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --uvm-persistence-mode --verbose
  • Tell systemd to reload its modules, and reboot the guest 
    systemctl daemon-reload
reboot
  • Validating State and Versions
    • Checking GPU CC mode status
    nvidia-smi conf-compute -f
    • Checking VBIOS version, Minimum version needed is 96.00.5E.00.00
     nvidia-smi -q | grep VBIOS
    • Checking GPU Ready state
    nvidia-smi conf-compute -grs
    • Changing GPU to ready state
    nvidia-smi conf-compute -srs 1
    Now the guest VM is ready to operate in CC mode with secured H100 accelerator.