RPI 5 Zephyr Dom0 exercise

RPI 5 Zephyr Dom0 exercise is intended to demonstrate of basic functionality of Zephyr running as thin Xen Dom0 on Raspberry Pi 5 platform.

The thin Zephyr Xen Dom0 allows to start at boot two Xen domains and, if enabled, manage them through shell commands. The Zephyr Xen Dom0 is implemented by zephyr-dom0-xt application developed by Xen Troops tem.

Note

The work is in active development.

              Start                                              
            ┌────────────────────────┬───────────────────┐       
            │                        │                   │       
            │                        │                   │       
            │                        │                   │       
┌───────────┴─────────────┐   ┌──────▼─────┐   ┌─────────▼──────┐
│                         │   │ Dom1       │   │ Dom2           │
│   zephyr-dom0-xt        │   │ (GPIO)     │   │ (xenvm)        │
│                         │   │            │   │                │
│                         │   │ samples/   │   │     samples/   │
│                         │   │            │   │                │
└─────────────────────────┘   │ Blinky     │   │ synchronization│
                              │            │   │                │
┌──┬────────────────┬─────┐   │            │   │                │
│  │                │     │   │            │   │                │
│  │ zephyr-xenlib  │     │   │            │   │                │
│  └────────────────┘     │   │            │   │                │
│                         │   │            │   │                │
│                         │   │            │   │                │
│ Zephyr OS               │   │Zephyr OS   │   │    Zephyr OS   │
└─────────────────────────┘   └────────────┘   └────────────────┘
                                                                 
┌───────────────────────────────────────────────────────────────┐
│                  XEN                                          │
└───────────────────────────────────────────────────────────────┘
                                                                 
┌───────────────────────────────────────────────────────────────┐
│                   Raspberry Pi 5                              │
└───────────────────────────────────────────────────────────────┘

The zephyr-dom0-xt application contains binary images of Dom1 and Dom2 and domains configuration. It may contain DTBs for domains also, but it is not actual for this exercise as both domains are Zephyr.

Boot process:

RPI5 bootloder started u-boot
u-boot executes boot script:
- loads Zephyr Dom0 binary
- adds Xen data to the System device tree
- loads Xen binary and starts Xen
Xen is booted and starts Zephyr Dom0
Zephyr Dom0 processing domains configuration which is statically defined
Zephyr Dom0 creates and starts Dom1
Zephyr Dom0 creates and starts Dom2
Dom1 and Dom2 can be paused/unpaused by using shell commands

Prerequisites

check and follow RPI 5 u‐boot
check and follow RPI 5 Xen
check and follow RPI 5 Zephyr
enable RPI 5 Zephyr xenlib

Zephyr sources

Update Zephyr sources from:

repo: https://github.com/xen-troops/zephyr.git

branch: rpi5_dev

Zephyr zephyr-dom0-xt Dom0 application

zephyr-dom0-xt sources

repo: https://github.com/GrygiriiS/zephyr-dom0-xt.git

branch: dom0-xt_rpi_5_dev_v1

Checkout and add zephyr-dom0-xt in Zephyr as Git submodule:

git submodule add -b dom0-xt_rpi_5_dev_v1 https://github.com/GrygiriiS/zephyr-dom0-xt.git

The zephyr-dom0-xt will be placed at:

<zephyr_root>
|- zephyr
   |- zephyr-dom0-xt  <-------
|- modules/
| ...
|- zephyr-xenlib/

Dom1 preparation

The Xen Dom1 is Zephyr samples/basic/blinky application, which shell be built using test RPI 5 snippet rpi_5_xen_domd, as it uses real GPIO HW "/soc/gpio@7d517c00" which is passed through to Dom1 by Zephyr Dom0 and Xen.

Use below command to build Dom1 Zephyr Blinky application:

west build -b xenvm -p always -S rpi_5_xen_domd samples/basic/blinky

Once build is finished copy zephyr.bin into the zephyr/zephyr-dom0-xt/prebuilt/domd_img.bin

Dom2 preparation

The Xen Dom2 is pure Xen DomU which doesn't use any real SoC HW. The Dom2 is Zephyr samples/synchronization application, which shell be built using xenvm platform.

Use below command to build Dom2 Zephyr Synchronization application:

west build -b xenvm -p always samples/synchronization

Once build is finished copy zephyr.bin into the zephyr/zephyr-dom0-xt/prebuilt/domu_img.bin

Dom0 zephyr-dom0-xt preparation

Use below command to build Zephyr Dom0 zephyr-dom0-xt application:

    west build -b rpi_5 -p always -S xen_dom0 zephyr-dom0-xt -- \
    -DCONFIG_USE_DEFAULT_DOM_CFG=n -DCONFIG_DOM_CFG_NUMBER=2

Once build is finished copy zephyr.bin to RPI5 bootfs boot partition.

Dom0 zephyr-dom0-xt boot

Follow instructions in Zephyr boot under XEN from u-boot using boot script to boot the system.

Dom0 get list of available configurations

Run xu config_list to see list of available configurations:

    uart:~$ xu config_list
    rpi_5_domd
    rpi_5_domu

Dom0 create Dom1

Create Dom1 by using xu create with DomD configuration and Xen domainId -d 1. Note the zephyr-xenlib_ starts automatically domain with Xen domainId 1.

    uart:~$ xu create rpi_5_domd -d 1
    W: Domain device tree generation is not supported
    I: rambase = 40000000, ramsize = 16777216
    I: kernbase = 40000000 kernsize = 360452, dtbsize = 11058
    I: kernsize_aligned = 2097152
    I: DTB will be placed on addr = 0x40e00000
    (XEN) memory_map:add: dom1 gfn=107d517 mfn=107d517 nr=1
    uart:~$ (XEN) d1v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
    (XEN) common/grant_table.c:1909:d1v0 Expanding d1 grant table from 1 to 2 frames
    ...
    (XEN) common/grant_table.c:1909:d1v0 Expanding d1 grant table from 15 to 16 frames

Dom0 create Dom2

Create Dom2 by using xu create with DomU configuration and Xen domainId -d 2.

    uart:~$ xu create rpi_5_domu -d 2
    W: Domain device tree generation is not supported
    I: rambase = 40000000, ramsize = 16777216
    I: kernbase = 40000000 kernsize = 364548, dtbsize = 11058
    I: kernsize_aligned = 2097152
    I: DTB will be placed on addr = 0x40e00000
    I: Created domain is paused
    To unpause issue: xu unpause 2

    uart:~$ xu unpause 2

Dom0 zephyr-dom0-xt attach Dom1/2 console

To see Dom1 console run command (to exit console press Ctrl-]):

xu console 1

The console output should be:

Attached to a domain console
*** Booting Zephyr OS build v3.6.0-65-gf93361527725 ***
LED state: OFF
LED state: ON
LED state: OFF
LED state: ON
LED state: OFF
LED state: ON
LED state: OFF
LED state: ON
LED state: OFF
LED state: ON
LED state: OFF
LED state: ON
LED state: OFF
Detached from console

To see Dom2 console run command (to exit console press Ctrl-]):

xu console 2