Skip to content

KVM RISC V 64bit on FireSim

Jump to bottom
Atish Patra edited this page Feb 11, 2021 · 17 revisions

Currently, we can boot RISC-V 64bit SMP Guest using KVM RISC-V on FireSim. The FPGA design is based on Rocket core with H extension enabled. The Rocket-H design is developed by @sandro2pinto & @josecm and available in Firesim Framework. This document will focus on the instructions on how to build/run various components to boot Guest Linux using KVM RISC-V.

To achieve this, we need following components:

  1. Rocket core RISC-V Hypervisor Extension enabled image in FireSim
  2. OpenSBI Firmware
  3. Host & Guest Kernel Image
  4. KVMTOOL
  5. Host RootFS with KVMTOOL and Guest Kernel
  6. AWS account with F1 access. The "eu-west-1" or "us-west-2" region must be selected as the AFI image is available in those two regions only.

The below sub-sections provide detailed steps to build and run RISC-V KVM on FireSim. All components except FireSim require a RISC-V 64bit cross-compiler so for we use Linux multilib toolchain from RISC-V GNU Compiler Toolchain project (Refer, https://github.com/riscv/riscv-gnu-toolchain). If you have a different RISC-V 64bit cross-compiler then set CROSS_COMPILE environment variable appropriately in steps below.

The Rocket-H design has limited memory and doesn't have any external IO support that can be used load rootfs. Thus, the host image has to include a initramfs while the guest can use the pre-built rootfs. Thus, we can't use use same RISC-V 64bit Linux kernel as Guest and Host kernel. We need to compile them separately.

Building the software image

1. Build Guest Linux Kernel Image

git clone https://github.com/kvm-riscv/linux.git
export ARCH=riscv
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
mkdir build-riscv64
make -C linux O=`pwd`/build-riscv64 defconfig
make -C linux O=`pwd`/build-riscv64

The above commands will create build-riscv64/arch/riscv/boot/Image which will be our Guest kernel.

2. Add libfdt library to CROSS_COMPILE SYSROOT directory

We need libfdt library in the cross-compile toolchain for compiling KVMTOOL RISC-V (described in next step). The libfdt library is generally not available in the cross-compile toolchain so we need to explicitly compile libfdt from DTC project and add it to CROSS_COMPILE SYSROOT directory.

git clone git://git.kernel.org/pub/scm/utils/dtc/dtc.git
cd dtc
export ARCH=riscv
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
export CC="${CROSS_COMPILE}gcc -mabi=lp64d -march=rv64gc"
TRIPLET=$($CC -dumpmachine)
SYSROOT=$($CC -print-sysroot)
make libfdt
make EXTRA_CFLAGS="-mabi=lp64d" DESTDIR=$SYSROOT PREFIX=/usr LIBDIR=/usr/lib64/lp64d install-lib install-includes
cd ..

The above commands will install cross-compiled libfdt library at $SYSROOT/usr/lib64/lp64d directory of cross-compile toolchain.

3. Build KVMTOOL

git clone https://github.com/kvm-riscv/kvmtool.git
export ARCH=riscv
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
cd kvmtool
make lkvm-static
${CROSS_COMPILE}strip lkvm-static
cd ..

The above commands will create kvmtool/lkvm-static which will be our user-space tool for KVM RISC-V.

4. Build Host RootFS containing KVMTOOL and Guest Linux

export ARCH=riscv
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
git clone https://github.com/kvm-riscv/howto.git
wget https://busybox.net/downloads/busybox-1.27.2.tar.bz2
tar -C . -xvf ./busybox-1.27.2.tar.bz2
mv ./busybox-1.27.2 ./busybox-1.27.2-kvm-riscv64
cp -f ./howto/configs/busybox-1.27.2_defconfig busybox-1.27.2-kvm-riscv64/.config
make -C busybox-1.27.2-kvm-riscv64 oldconfig
make -C busybox-1.27.2-kvm-riscv64 install
mkdir -p busybox-1.27.2-kvm-riscv64/_install/etc/init.d
mkdir -p busybox-1.27.2-kvm-riscv64/_install/dev
mkdir -p busybox-1.27.2-kvm-riscv64/_install/proc
mkdir -p busybox-1.27.2-kvm-riscv64/_install/sys
mkdir -p busybox-1.27.2-kvm-riscv64/_install/apps
ln -sf /sbin/init busybox-1.27.2-kvm-riscv64/_install/init
cp -f ./howto/configs/busybox/fstab busybox-1.27.2-kvm-riscv64/_install/etc/fstab
cp -f ./howto/configs/busybox/rcS busybox-1.27.2-kvm-riscv64/_install/etc/init.d/rcS
cp -f ./howto/configs/busybox/motd busybox-1.27.2-kvm-riscv64/_install/etc/motd
cp -f ./kvmtool/lkvm-static busybox-1.27.2-kvm-riscv64/_install/apps
cp -f ./build-riscv64/arch/riscv/boot/Image busybox-1.27.2-kvm-riscv64/_install/apps
cd busybox-1.27.2-kvm-riscv64/_install; find ./ | cpio -o -H newc > ../../rootfs_kvm_riscv64.cpio; cd -

The above commands will create rootfs_kvm_riscv64.cpio which will be our Host RootFS containing KVMTOOL and Guest Linux.

5. Build Host Linux Kernel Image

git clone https://github.com/kvm-riscv/firesim.git
cd linux
git am ../firesim/0001-force-sifive-uart-config.patch
cd build-riscv64
make -C linux O=`pwd`/build-riscv64 defconfig

Include the compiled rootfs in kernel config by adding following lines to the .config parameter or from menuconfig.

CONFIG_INITRAMFS_SOURCE="rootfs_kvm_riscv64.cpio" 
CONFIG_INITRAMFS_ROOT_UID=0
CONFIG_INITRAMFS_ROOT_GID=0
CONFIG_INITRAMFS_COMPRESSION_NONE=y

Build the linux kernel image now.

make -C linux O=`pwd`/build-riscv64

##7. Compile the device tree

cd firesim
dtc rocket-firesim-minimal.dts > rocket-firesim-minimal.dtb

8. Build OpenSBI Firmware

git clone https://github.com/riscv/opensbi.git
cd opensbi
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
PLATFORM=generic FW_PAYLOAD_PATH=../build-riscv64/arch/riscv/boot/Image FW_FDT_PATH=../build-riscv64/firesim/rocket-firesim-minimal.dtb
cd ..

The above commands will create opensbi/build/platform/generic/firmware/fw_payload.elf which will be the workload used for firesim.

Using the Firesim to load the software image

These instructions are based on bao hypervisor instructions[1]

For FireSim Initial Setup/Installation please refer to the FireSim documentation:

https://docs.fires.im/en/latest/Initial-Setup/index.html

Note: While selecting the AMI, you can use the FPGA Developer AMI - 1.6.1

1. Add Rocket-H to Chipyard

On your AWS machine, add our rocket-chip design:

cd firesim
git checkout eb7d1af4964814270b176d4e682fc999e8db561a
cd firesim/target-design/chipyard
git checkout 0e0c89cb1fa64eda74fa8e7dac438eae54a9df56
cd firesim/target-design/chipyard/generators/rocket-chip/
git remote add josecm https://github.com/josecm/rocket-chip.git
git fetch josecm
git checkout hyp

2. Configure useHype parameter on Boom and Ariane

2.1 - Boom

Edit the 'parameters.scala' file to include the useHype parameter:

firesim/target-design/chipyard/generators/boom/src/main/scala/common/parameters.scala

useSupervisor: Boolean = false,
useHype: Boolean = false,
useVM: Boolean = true,

2.2 - Ariane

Edit the 'ArianeTile.scala' file to include the useHype parameter:

firesim/target-design/chipyard/generators/ariane/src/main/scala/ArianeTile.scala

'val useSupervisor: Boolean = false'
'val useHype: Boolean = false'
'val useDebug: Boolean = true'

2.3 Extend Chipyard rocket configurations

Edit the 'RocketConfigs.scala' file to define the QuadRocketHypConfig class:

firesim/target-design/chipyard/generators/chipyard/src/main/scala/config/RocketConfigs.scala

class QuadRocketHypConfig extends Config(
new freechips.rocketchip.subsystem.WithHyp ++
new QuadRocketConfig)

Note: As a sugestion, add the class at the top of the file

Edit the 'TargetConfigs.scala' file to include the QuadRocketHypConfig class:

firesim/target-design/chipyard/generators/firechip/src/main/scala/TargetConfigs.scala

class FireSimQuadRocketHypConfig extends Config(
new WithDefaultFireSimBridges ++
new WithDefaultMemModel ++
new WithFireSimConfigTweaks ++
new chipyard.QuadRocketHypConfig)

Note: As a sugestion, add the class after the definition of the FireSimQuadRocketConfig class

Reference

  1. https://github.com/sandro2pinto/bao-rocket-h-firesim
Clone this wiki locally