Skip to content

KVM RISC V 64bit on FireSim

Jump to bottom
Atish Patra edited this page Feb 11, 2021 · 17 revisions

Currently, we can boot RISC-V 64bit SMP Guest using KVM RISC-V on FireSim. The FPGA design is based on Rocket core with H extension enabled. The Rocket-H design is developed by @sandro2pinto & @josecm and available in Firesim Framework. This document will focus on the instructions on how to build/run various components to boot Guest Linux using KVM RISC-V.

To achieve this, we need following components:

  1. Rocket core RISC-V Hypervisor Extension enabled image in FireSim
  2. OpenSBI Firmware
  3. Host & Guest Kernel Image
  4. KVMTOOL
  5. Host RootFS with KVMTOOL and Guest Kernel

The below sub-sections provide detailed steps to build and run RISC-V KVM on FireSim. All components except FireSim require a RISC-V 64bit cross-compiler so for we use Linux multilib toolchain from RISC-V GNU Compiler Toolchain project (Refer, https://github.com/riscv/riscv-gnu-toolchain). If you have a different RISC-V 64bit cross-compiler then set CROSS_COMPILE environment variable appropriately in steps below.

The Rocket-H design has limited memory and doesn't have any external IO support that can be used load rootfs. Thus, the host image has to include a initramfs while the guest can use the pre-built rootfs. Thus, we can't use use same RISC-V 64bit Linux kernel as Guest and Host kernel. We need to compile them separately.

1. Build Guest Linux Kernel Image

git clone https://github.com/kvm-riscv/linux.git
export ARCH=riscv
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
mkdir build-riscv64
make -C linux O=`pwd`/build-riscv64 defconfig
make -C linux O=`pwd`/build-riscv64

The above commands will create build-riscv64/arch/riscv/boot/Image which will be our Guest kernel.

2. Add libfdt library to CROSS_COMPILE SYSROOT directory

We need libfdt library in the cross-compile toolchain for compiling KVMTOOL RISC-V (described in next step). The libfdt library is generally not available in the cross-compile toolchain so we need to explicitly compile libfdt from DTC project and add it to CROSS_COMPILE SYSROOT directory.

git clone git://git.kernel.org/pub/scm/utils/dtc/dtc.git
cd dtc
export ARCH=riscv
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
export CC="${CROSS_COMPILE}gcc -mabi=lp64d -march=rv64gc"
TRIPLET=$($CC -dumpmachine)
SYSROOT=$($CC -print-sysroot)
make libfdt
make EXTRA_CFLAGS="-mabi=lp64d" DESTDIR=$SYSROOT PREFIX=/usr LIBDIR=/usr/lib64/lp64d install-lib install-includes
cd ..

The above commands will install cross-compiled libfdt library at $SYSROOT/usr/lib64/lp64d directory of cross-compile toolchain.

3. Build KVMTOOL

git clone https://github.com/kvm-riscv/kvmtool.git
export ARCH=riscv
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
cd kvmtool
make lkvm-static
${CROSS_COMPILE}strip lkvm-static
cd ..

The above commands will create kvmtool/lkvm-static which will be our user-space tool for KVM RISC-V.

4. Build Host RootFS containing KVMTOOL and Guest Linux

export ARCH=riscv
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
git clone https://github.com/kvm-riscv/howto.git
wget https://busybox.net/downloads/busybox-1.27.2.tar.bz2
tar -C . -xvf ./busybox-1.27.2.tar.bz2
mv ./busybox-1.27.2 ./busybox-1.27.2-kvm-riscv64
cp -f ./howto/configs/busybox-1.27.2_defconfig busybox-1.27.2-kvm-riscv64/.config
make -C busybox-1.27.2-kvm-riscv64 oldconfig
make -C busybox-1.27.2-kvm-riscv64 install
mkdir -p busybox-1.27.2-kvm-riscv64/_install/etc/init.d
mkdir -p busybox-1.27.2-kvm-riscv64/_install/dev
mkdir -p busybox-1.27.2-kvm-riscv64/_install/proc
mkdir -p busybox-1.27.2-kvm-riscv64/_install/sys
mkdir -p busybox-1.27.2-kvm-riscv64/_install/apps
ln -sf /sbin/init busybox-1.27.2-kvm-riscv64/_install/init
cp -f ./howto/configs/busybox/fstab busybox-1.27.2-kvm-riscv64/_install/etc/fstab
cp -f ./howto/configs/busybox/rcS busybox-1.27.2-kvm-riscv64/_install/etc/init.d/rcS
cp -f ./howto/configs/busybox/motd busybox-1.27.2-kvm-riscv64/_install/etc/motd
cp -f ./kvmtool/lkvm-static busybox-1.27.2-kvm-riscv64/_install/apps
cp -f ./build-riscv64/arch/riscv/boot/Image busybox-1.27.2-kvm-riscv64/_install/apps
cd busybox-1.27.2-kvm-riscv64/_install; find ./ | cpio -o -H newc > ../../rootfs_kvm_riscv64.cpio; cd -

The above commands will create rootfs_kvm_riscv64.cpio which will be our Host RootFS containing KVMTOOL and Guest Linux.

5. Build Host Linux Kernel Image

cd build-riscv64
make -C linux O=`pwd`/build-riscv64 defconfig

Include the compiled rootfs in kernel config by adding following lines to the .config parameter or from menuconfig.

CONFIG_INITRAMFS_SOURCE="rootfs_kvm_riscv64.cpio" 
CONFIG_INITRAMFS_ROOT_UID=0
CONFIG_INITRAMFS_ROOT_GID=0
CONFIG_INITRAMFS_COMPRESSION_NONE=y

Build the linux kernel image now.

make -C linux O=`pwd`/build-riscv64

##7. Compile the device tree

git clone https://github.com/kvm-riscv/firesim.git
cd firesim
dtc rocket-firesim-minimal.dts > rocket-firesim-minimal.dtb

8. Build OpenSBI Firmware

git clone https://github.com/riscv/opensbi.git
cd opensbi
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
PLATFORM=generic FW_PAYLOAD_PATH=../build-riscv64/arch/riscv/boot/Image FW_FDT_PATH=../build-riscv64/firesim/rocket-firesim-minimal.dtb
cd ..

The above commands will create opensbi/build/platform/generic/firmware/fw_payload.elf which will be the workload used for firesim.

Reference

  1. https://github.com/sandro2pinto/bao-rocket-h-firesim
Clone this wiki locally