Skip to content

Running Qflex

Jump to bottom
UlisesLuzius edited this page Oct 11, 2023 · 18 revisions

Running QFlex in multi-nodes

In this case, QFlex is meant to be running in a virtual cluster on the same host. For this reason, the first step is to build QFlex with appropriate features, then to build a virtual network inside the host machine, to finally add the guest instance to the virtual network.

Requirements:

  • Running a Linux distribution
  • iproute2 is installed
  • Being inside the Nix environment

Building QFlex using NixOS

NixOS is a platform that allows reproducible builds and deployments such as you won't need to manage any dependencies for running QFlex.

Please first clone the qflex main respository (at the moment all branches of qflex, libqflex, flexus, qemu need to checkout to qflex-8.0):

git clone https://github.com/parsa-epfl/qflex.git
cd qflex
git submodule update --init

To install nix, and enable Nix Flakes (~1 minute):

sh <(curl -L https://nixos.org/nix/install) --daemon # install nix
Welcome to the Multi-User Nix Installation
...
Would you like to see a more detailed list of what I will do?
[y/n] n
...
Can I use sudo?
[y/n] y
... nixbldXX users ...
Ready to continue?
[y/n] y
...
Alright! We're done!

Finally, to enable Nix Flakes:

mkdir -p ~/.config/nix && echo "experimental-features = nix-command flakes" >> ~/.config/nix/nix.conf

Finally you can enter the nix environment and start building QEMU:

nix develop -i -k HOME # Entered the enviroment
cd qemu 
./configure --target-list=aarch64-softmmu --enable-qflex --enable-snapext --enable-capstone --enable-multinode
make -j12

And to build both QFlex Timing and Trace simulators:

cd ../libqflex/qflex-trace
make
cd ../../
cd flexus
mkdir build && cd build
cmake -DSIMULATOR=KeenKraken .. && make -j12
cmake -DSIMULATOR=KnottyKraken.Boom .. && make -j12

Always make sure to be inside the nix environment running cmake or ./configure and building make -jX.

Notes:

  • -i = --ignore-environment: Clear the entire environment (except those specified with --keep), it won't try to parse any local packages and libraries.
  • -k NAME = --keep: Keep the environment variable name.
  • --enable-snapext is for allowing QEMU to save incremental and external snapshots, check SMARTS methodology for more details.
  • --enable-multinode is for allowing two or more QEMU instances to advance in sychronicity in the notion of time.
  • --enable-qflex is for building QEMU modifications allowing to run with the simulator attached.
  • --enable-capstone is for QEMU and QFlex to use capstone when logging executed instructions type.
  • KeenKraken is our QFlex Trace simulator.
  • KnottyKraken.Boom is our QFlex Timing simulator.

Building virtual network

The network build for this purpose is a simple star topology network, as in the following figure. The next point are to be executed on the host machine.

Virtual bridged network to connect multiple QEMU instances

Creation of a virtual bridge

The following bash commands create a new interface, then add an IP address to it, and finally mount the virtual interface. Mounting the interface also create a static route to it.

ip link add br0 type bridge # Where `br0` is the bridge interface's name.
ip addr add [BRIDGE IP]/[SUBNET MASK] dev br0 # ex. ip addr add 192.168.1.100/24 dev br0
ip link set br0 up

Allowing QFlex acess to virtual bridge

To allow QFlex to freely access the bridge as well as connecting itself the devices to it, the following must be appended to the bridge.conf

allow br0

The file can be found in the following directory:

  1. /etc/qemu/bridge.conf if you're running a globally installed QEMU/QFlex
  2. [QFLEX ROOT]/qemu/build/qemu-bundle/usr/local/etc/bridge.conf if you're running a locally compiled version of QFlex

Modifying forwarding policies

If the QFlex guest instances have to communicate in-between them. The iptables policies have to be modified to avoid any packet drop in the bridge.

iptables -I FORWARD -i br0 -j ACCEPT
iptables -I FORWARD -o br0 -j ACCEPT

Starting the SyncServer

The SyncServer code source find itself in the [QFLEX ROOT]/libqflex/multi-nodes directory. Foremost, compile the code source onto a binary.

cargo build --release

Then start the server on your host.

[MULTI-NODES ROOT]/target/release/multi-node -m -f [SOCKET PATH] -n [NB OF SLAVE] -b [BUDGET]
  • -m indicate a master type instance
  • -f a location to create a UNIX-type socket
  • -n the number of QFlex slave expected
  • -b a budget/quantum quantity.

Starting QFlex instances

According to the number of QFlex instances expected, start your guest machine with your usual parameter, with the following device and network configuration.

-netdev bridge,br=br0,id=hn1
-device virtio-net,netdev=hn1,mac=e6:c8:ff:09:76:9c
-multinode slave-idx=[SLAVE ID],socket-path=[SOCKET PATH],start=[AUTOSTART]

# The MAC adress choosen above should be unique for each guest machine and follow the Administratively Assigned (AAI) convention
# see: https://en.wikipedia.org/wiki/MAC_address#IEEE_802c_local_MAC_address_usage

# Exemple
 sudo [QFLEX ROOT]/qemu/build/qemu-system-aarch64 --cpu max \
 -machine virt,gic-version=3 \
 -smp 1 -m 4G -rtc clock=vm \
 -drive file=images/qemu-efi.img,format=raw,if=pflash,readonly=on \
 -drive file=images/varstore.qcow2,if=pflash,readonly=on \
 -drive file=images/root.qcow2.base2,format=qcow2,if=virtio \
 -object rng-random,filename=/dev/urandom,id=rng0 \
 -device virtio-rng-pci,rng=rng0 \

 -netdev bridge,br=br0,id=hn1
 -device virtio-net,netdev=hn1,mac=e6:c8:ff:09:76:9c
 -multinode slave-idx=0,socket-path=/tmp/qflex,start=off \

 -icount shift=0,sleep=on,align=off \
 -nographic
  • [SLAVE ID] is a unique identifier for your salve
  • [SOCKET PATH] the same socket that was opened by the server
  • [AUTOSTART] a switch with value on or off. The former start to consume the budget right after the initial connection with the SyncServer.

Interacting with QFlex instance's behaviour

To access the QEMU prompt of any QFlex instance, press Ctrl-A then C. From the prompt, two additional commands were implemented.

  • savevm-externale [NAME]: Save an incremental snapshot of the running machine
  • multinodes-start/multinodes-stop: Start or stop consuming the budget, required if start=off was passed for arguments.
Clone this wiki locally