Multinode support

We have implemented features require to synchronize two QEMU instances.

Running QFlex in multi-nodes

Building QFlex

See here

Building virtual network

The network build for this purpose is a simple star topology network, as in the following figure. The next point are to be executed on the host machine.

Creation of a virtual bridge

The following bash commands create a new interface, then add an IP address to it, and finally mount the virtual interface. Mounting the interface also create a static route to it.

ip link add br0 type bridge # Where `br0` is the bridge interface's name.
ip addr add [BRIDGE IP]/[SUBNET MASK] dev br0 # ex. ip addr add 192.168.1.100/24 dev br0
ip link set br0 up

Allowing QFlex acess to virtual bridge

To allow QFlex to freely access the bridge as well as connecting itself the devices to it, the following must be appended to the bridge.conf

allow br0

The file can be found in the following directory:

1. /etc/qemu/bridge.conf if you're running a globally installed QEMU/QFlex
2. [QFLEX ROOT]/qemu/build/qemu-bundle/usr/local/etc/bridge.conf if you're running a locally compiled version of QFlex

Modifying forwarding policies

If the QFlex guest instances have to communicate in-between them. The iptables policies have to be modified to avoid any packet drop in the bridge.

iptables -I FORWARD -i br0 -j ACCEPT
iptables -I FORWARD -o br0 -j ACCEPT

Starting the SyncServer

The SyncServer code source find itself in the [QFLEX ROOT]/libqflex/multi-nodes directory. Foremost, compile the code source onto a binary.

cargo build --release

Then start the server on your host.

[MULTI-NODES ROOT]/target/release/multi-node -m -f [SOCKET PATH] -n [NB OF SLAVE] -b [BUDGET]

• -m indicate a master type instance
• -f a location to create a UNIX-type socket
• -n the number of QFlex slave expected
• -b a budget/quantum quantity.

Starting QFlex instances

According to the number of QFlex instances expected, start your guest machine with your usual parameter, with the following device and network configuration.

-netdev bridge,br=br0,id=hn1
-device virtio-net,netdev=hn1,mac=e6:c8:ff:09:76:9c
-multinode slave-idx=[SLAVE ID],socket-path=[SOCKET PATH],start=[AUTOSTART]

##################################################################################
# The MAC adress choosen above HAVE TO BE UNIQUE FOR EACH GUEST MACHINE 
# and follow the Administratively Assigned (AAI) convention
# see: https://en.wikipedia.org/wiki/MAC_address#IEEE_802c_local_MAC_address_usage
##################################################################################

# Exemple
 sudo [QFLEX ROOT]/qemu/build/qemu-system-aarch64 --cpu max \
 -machine virt,gic-version=3 \
 -smp 1 -m 4G -rtc clock=vm \
 -drive file=images/qemu-efi.img,format=raw,if=pflash,readonly=on \
 -drive file=images/varstore.qcow2,if=pflash,readonly=on \
 -drive file=images/root.qcow2,format=qcow2,if=virtio \
 -netdev bridge,br=br0,id=hn1
 -device virtio-net,netdev=hn1,mac=e6:c8:ff:09:76:9c
 -multinode slave-idx=0,socket-path=/tmp/qflex,start=off \
 -icount shift=0,sleep=on,align=off \
 -nographic

• [SLAVE ID] is a unique identifier for your salve
• [SOCKET PATH] the same socket that was opened by the server
• [AUTOSTART] a switch with value on or off. The former start to consume the budget right after the initial connection with the SyncServer.

Configuring QFlex instance's IP network configuration

Previously, the MAC address was configured though QEMU CLI. The IP address of each guest instances have to setted up manually, with the following command

ip addr add [INSTANCE IP]/[MASK] dev etho
ip link set eth0 up

Where [INSTANCE IP]/[MASK] is a valid and unique address that belong to the same network as the bridge br0

Interacting with QFlex instance's behaviour

To access the QEMU prompt of any QFlex instance, press Ctrl-A then C. From the prompt, two additional commands were implemented.

• savevm-externale [NAME]: Save an incremental snapshot of the running machine
• multinodes-start/multinodes-stop: Start or stop consuming the budget, required if start=off was passed for arguments.