Access to peripherals from enclaves

[grg-haas]

This PR implements (still experimental!) support for accessing peripherals from Keystone enclaves (as requested in #414). This is accomplished through several steps:

1. In the SM, as part of the boot process, we parse the device tree and look for nodes marked with status="disabled" and secure-status="okay", following the Linux kernel spec for firmware-secured devices. Such devices are allocated a PMP register.
2. If an enclave then wants to claim exclusive access to this device for a time, it can use the new claim_mmio runtime syscall to do so. This assigns the device's PMP region to the requesting struct enclave, ensuring that physical access to it is enabled at context switch time.
3. In order to make use of the device, however, the runtime must still map it virtually. To that extent, we've (@chungmcl and I) implemented a basic driver IO system that allows for redirecting calls to read and write to a driver instead. To open a driver, call openat(-2, "<name of the driver>", 0, 0) to get a file descriptor. This file descriptor can then be passed to the read, write, fprintf, fflush, etc functions in the eapp.

[its-valentinvp]

Hi @grg-haas,
We (@f1bu and I) experimented with this PR during our case study. We needed some changes to get it to work correctly and want to contribute them here. We provide the necessary patch at the bottom of this comment.

Important notes:
The DRIVERS option needs to be set to ON. We also recommend to turn on advanced debugging for the runtime. This makes it easier to see what is going on in the runtime. For example, this allows us to see if a syscall is implemented in the runtime or if page faults occur. To do all this, change these three options in runtime/CMakeLists.txt to ON:

rt_option(DRIVERS "Include support for hardware drivers" OFF)
rt_option(INTERNAL_STRACE "Debug syscalls" OFF)
rt_option(DEBUG "Enable debugging" OFF)

Changes:

• kernel_va_to_pa was not working correctly on claiming/releasing the device, so we replaced it with translate in runtime/call/syscall.c
• the device name did not match the driver name and also did not match the device tree node, which was created for UART (it was named /soc/serial@10001000 in Qemu), so the device was therefore not found and opened at failed too. We changed the driver and device name to serial and applied the new name in examples/devshare/eapp/devshare.c too.
• In examples/devshare/eapp/devshare.c we also need to replace fprintf with write, otherwise we get page faults. We didnt investigated this further, because write worked fine.
• Optional: Some QoL changes like a devshare enclave startup script (just run ./startup.sh after Qemu boot) and a modified inittab to skip buildroot login.

Demo:
The output of the devshare enclave looks like this:

Additional Infos:
This PR only works with Qemu, to try it on real hardware boards, we need to paste in the device driver code (in our case, it was a u-boot driver for the network card of the VF2 board) and also need to modify the device tree of Linux to mark the device as a secure device (using flags: secure-status and status). An example patch for the VF2 and NIC port 0 looks like this:

--- a/arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive-2.dtsi	2024-06-24 17:10:56.102667746 +0000
+++ "b/arch/riscv/boot/dts/starfive/jh7110-starfive-visionfive-2.dtsi"	2024-06-24 17:15:17.500790752 +0000
@@ -128,7 +128,8 @@
 &gmac0 {
 	phy-handle = <&phy0>;
 	phy-mode = "rgmii-id";
-	status = "okay";
+	status = "disabled";
+	secure-status = "okay";
 
 	mdio {
 		#address-cells = <1>;

Now, the device should be claimable under this name:

#define SECURE_DEVICE "ethernet@16030000"

Once the driver code is copied to the runtime, we also need to provide a driver definition. The name is the driver name that needs to be passed to the openat call in the EAPP. For VF2 this could look like this:

driver_instance dwmac_ethernet_driver = {
  .name = "ethernet",
  .init = ethernet_init,
  .fini = ethernet_fini,
  .read = ethernet_read,
  .write = ethernet_write
};

The drivers init function is a perfect place to map the physical mmio region to the runtime. The needed size and the base address can be found in the Linux device tree. For VF2 and NIC port 0 it looks like this:

gmac0: ethernet@16030000 {
			compatible = "starfive,jh7110-dwmac", "snps,dwmac-5.20";
			reg = <0x0 0x16030000 0x0 0x10000>;
...

So, the base address is 0x1603000 and the size is 0x10000. Now, the mapping in the driver can be done like this:

void *base = (void *) map_anywhere_with_dynamic_page_table((uintptr_t) 0x16030000, 0x10000);

This mapping allows us to read/write mmio registers inside the runtime.

It is important to note that the eyrie runtime is currently pretty limited for more complex driver development, but it is still possible to integrate more complex drivers like an NIC driver (like we did for our case study). However, this involves much manual work (In the case of the U-Boot driver, we had to decouple it completely from U-Boot and also needed to parse the device tree "manually").

---

Github seems to not allow .patch files from a Linux machine. We need to paste the patch here:

diff --git a/examples/devshare/eapp/devshare.c b/examples/devshare/eapp/devshare.c
index ce6b2ca..ca47a07 100644
--- a/examples/devshare/eapp/devshare.c
+++ b/examples/devshare/eapp/devshare.c
@@ -3,28 +3,34 @@
 
 #include "app/syscall.h"
 
-#define SECURE_DEVICE  "uart@10001000"
+// Define the device which we want to claim
+#define SECURE_DEVICE  "serial@10001000"
 
 int main()
 {
   int ret, fd, i;
   ret = claim_mmio(SECURE_DEVICE,
                    strlen(SECURE_DEVICE));
+
   if(ret < 0) {
     printf("Failed to claim " SECURE_DEVICE "\n");
     return -1;
   }
 
-  fd = openat(-2, "uart8250", 0, 0);
+  printf("Successfully claimed " SECURE_DEVICE "\n");
+
+  // Get file descriptor for uart8250 driver
+  fd = openat(-2, "serial", 0, 0);
   if(fd < 0) {
     printf("Failed to get fd for device\n");
     return -1;
   }
 
-  for(i = 0; i < 1000; i++) {
-    fprintf(fd, "Writing to UART: %i!\n", i);
-    fflush(fd);
-  }
+  printf("Got FD for uart8250 driver\n");
+
+  // Write to file descriptor/device
+  write(fd, "Hello, World!\n", 15);
+  printf("Writing to FD, see /tmp/serial.out for output.\n");
 
   // todo do something with the device
 
@@ -32,7 +38,10 @@ int main()
                      strlen(SECURE_DEVICE));
   if(ret < 0) {
     printf("Failed to release " SECURE_DEVICE "\n");
+    return -1;
   }
 
+  printf("Successfully released " SECURE_DEVICE "\n");
+
   return 0;
 }
diff --git a/overlays/keystone/board/generic/etc/inittab b/overlays/keystone/board/generic/etc/inittab
new file mode 100644
index 0000000..5e4d040
--- /dev/null
+++ b/overlays/keystone/board/generic/etc/inittab
@@ -0,0 +1,41 @@
+# /etc/inittab
+#
+# Copyright (C) 2001 Erik Andersen <[email protected]>
+#
+# Note: BusyBox init doesn't support runlevels.  The runlevels field is
+# completely ignored by BusyBox init. If you want runlevels, use
+# sysvinit.
+#
+# Format for each entry: <id>:<runlevels>:<action>:<process>
+#
+# id        == tty to run on, or empty for /dev/console
+# runlevels == ignored
+# action    == one of sysinit, respawn, askfirst, wait, and once
+# process   == program to run
+
+# Startup the system
+::sysinit:/bin/mount -t proc proc /proc
+::sysinit:/bin/mount -o remount,rw /
+::sysinit:/bin/mkdir -p /dev/pts /dev/shm
+::sysinit:/bin/mount -a
+::sysinit:/bin/mkdir -p /run/lock/subsys
+::sysinit:/sbin/swapon -a
+null::sysinit:/bin/ln -sf /proc/self/fd /dev/fd
+null::sysinit:/bin/ln -sf /proc/self/fd/0 /dev/stdin
+null::sysinit:/bin/ln -sf /proc/self/fd/1 /dev/stdout
+null::sysinit:/bin/ln -sf /proc/self/fd/2 /dev/stderr
+::sysinit:/bin/hostname -F /etc/hostname
+# now run any rc scripts
+::sysinit:/etc/init.d/rcS
+
+# Put a getty on the serial port, change dir to /root and autologin with root
+::respawn:-/bin/sh -c "cd /root;. /etc/profile;exec /bin/sh"
+
+# Stuff to do for the 3-finger salute
+#::ctrlaltdel:/sbin/reboot
+
+# Stuff to do before rebooting
+::shutdown:/etc/init.d/rcK
+::shutdown:/sbin/swapoff -a
+::shutdown:/bin/umount -a -r
+
diff --git a/overlays/keystone/board/generic/post-build.sh b/overlays/keystone/board/generic/post-build.sh
new file mode 100755
index 0000000..d174666
--- /dev/null
+++ b/overlays/keystone/board/generic/post-build.sh
@@ -0,0 +1,14 @@
+#!/bin/sh
+
+# Ensure the overlay directory structure exists
+mkdir -p $TARGET_DIR/etc
+
+# Copy the custom inittab to the target filesystem
+cp $BR2_EXTERNAL_KEYSTONE_PATH/board/generic/etc/inittab $TARGET_DIR/etc/inittab
+
+
+# Copy the script to the target directory
+cp $BR2_EXTERNAL_KEYSTONE_PATH/board/generic/startup.sh $TARGET_DIR/root/startup.sh
+
+# Make the script executable
+chmod +x $TARGET_DIR/root/startup.sh
diff --git a/overlays/keystone/board/generic/startup.sh b/overlays/keystone/board/generic/startup.sh
new file mode 100755
index 0000000..fc60768
--- /dev/null
+++ b/overlays/keystone/board/generic/startup.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+
+# Load the Keystone driver
+modprobe keystone-driver
+
+# Start devshare enclave
+/usr/share/keystone/examples/devshare.ke
\ No newline at end of file
diff --git a/overlays/keystone/configs/riscv64_generic_defconfig b/overlays/keystone/configs/riscv64_generic_defconfig
index 89e02f3..e266774 100644
--- a/overlays/keystone/configs/riscv64_generic_defconfig
+++ b/overlays/keystone/configs/riscv64_generic_defconfig
@@ -7,6 +7,7 @@ BR2_PACKAGE_HOST_GDB_PYTHON3=y
 BR2_CCACHE=y
 BR2_CCACHE_INITIAL_SETUP="-M0 -F0"
 BR2_GLOBAL_PATCH_DIR="$(BR2_EXTERNAL_KEYSTONE_PATH)/patches"
+BR2_ROOTFS_POST_BUILD_SCRIPT="$(BR2_EXTERNAL_KEYSTONE_PATH)/board/generic/post-build.sh"
 BR2_PER_PACKAGE_DIRECTORIES=y
 BR2_SSP_NONE=y
 BR2_TARGET_GENERIC_ROOT_PASSWD="sifive"
diff --git a/runtime/call/syscall.c b/runtime/call/syscall.c
index fa9e422..dc94b82 100644
--- a/runtime/call/syscall.c
+++ b/runtime/call/syscall.c
@@ -219,7 +219,7 @@ void handle_syscall(struct encl_ctx* ctx)
       break;
     }
 
-    uintptr_t devstr_claim_pa = kernel_va_to_pa(rt_copy_buffer_1);
+    uintptr_t devstr_claim_pa = translate((uintptr_t) rt_copy_buffer_1);
     copy_from_user(rt_copy_buffer_1, (void *) arg0, arg1);
 
     ret = sbi_claim_mmio(devstr_claim_pa);
@@ -239,7 +239,7 @@ void handle_syscall(struct encl_ctx* ctx)
       break;
     }
 
-    uintptr_t devstr_release_pa = kernel_va_to_pa(rt_copy_buffer_1);
+    uintptr_t devstr_release_pa = translate((uintptr_t) rt_copy_buffer_1);
     copy_from_user(rt_copy_buffer_1, (void *) arg0, arg1);
 
 #ifdef USE_DRIVERS
diff --git a/runtime/drivers/serial.c b/runtime/drivers/serial.c
index 0215344..a077941 100644
--- a/runtime/drivers/serial.c
+++ b/runtime/drivers/serial.c
@@ -218,7 +218,7 @@ size_t uart8250_write(void* buf, size_t len);
 size_t uart8250_read(void* buf, size_t len);
 
 driver_instance uart8250_driver = {
-  .name = "uart8250",
+  .name = "serial",
   .init = uart8250_init,
   .fini = NULL,
   .read = uart8250_read,