Multi-Agent RL for DFaaS by Emanuele Petriglia

This repository is in a working in progress state.

If you are looking for the source code of the experiments of Emanuele Petriglia's master's thesis, discussed in October 2024, see the petriglia-thesis-2024 branch.

The thesis, a summary and the presentation slides are available in another repository hosted on GitLab, but they are written in Italian.

How to set up the environment

The experiments are run and tested on Ubuntu 24.04 using Python 3.12. For a reproducible development environment, it is preferable to install the dependencies in a virtual environment (see the venv module). The venv module is not installed by default in Ubuntu, it must be installed using sudo apt install python3.12-venv.

The complete list of Python dependencies can be found in the requirements.txt file. However, the most important dependencies are:

• Ray RLlib (version 2.40): this is a reinforcement learning library used to define the DFaaS custom environment, run the experiments by training the models with the implemented algorithms.

• PyTorch (version 2.5.1): is a library for deep learning on GPUs and CPUs. It is used by Ray RLlib when training models with deep learning reinforcement learning algorithms.

For plotting only, the following dependencies are required:

• Matplotlib (version 3.9.3): is a plot generation library used in the scripts in the plots and notebooks directory.

• orjson (version 3.10.13): is a JSON library that is faster than the standard library. Used because the experiments generate large JSON files that slow down the encoding/decoding processes.

• Jupyter Notebook (version 7.3.2): Jupyter notebooks are used to create plots and explore results and statistics from the experiments. The notebooks are stored in the notebooks directory.

• ipympl (version 0.9.6): this is an extension for Jupyter Notebook to support interactive Matplotlib using Jupyter Widgets ipywidgets.

Note that Jupyter Notebook and ipympl are only needed if you are using the notebooks. Scripts in the plots directory are not interactive and only require Matplotlib.

When installing Ray RLlib, pip automatically installs its dependencies, which are also used by the experiment scripts (like NumPy or Gymnasium). This means that the environment can be easily set up by installing the following packages:

ray[rllib]==2.40.0
torch==2.5.1
gputil==1.4.0  # Required by RLlib (GPU system monitoring).
matplotlib==3.9.3
orjson==3.10.13
ipympl==0.9.6
notebook==7.3.2

Run the following commands to set up the development environment with Ubuntu:

$ sudo apt install python3.12-venv
$ git clone https://github.com/unimib-datAI/marl-dfaas.git
$ cd marl-dfaas
$ python3.12 -m venv .env
$ source .env/bin/activate
$ pip install <packages above>

For perfect reproducibility, there is a requirements.txt file that can be used instead of the previous command:

$ pip install --requirement requirements.txt

Please note that both the requirements file and the command line suggestions expect a machine with an NVIDIA GPU and CUDA (at least 12.4) installed for PyTorch. PyTorch can also be used with a CPU, in this case follow the instructions on the official website.

The requirements file also contains black (a development tool for automatically formatting source code and Jupyter notebooks), pylint (a static code analyser) and pre-commit packages. The latter run automatically black when doing commits.

How to run the experiments

WIP

Important: always run Python scripts from the project root directory to allow loading of commonly used modules (dfaas_env.py...). As example, if you need to run a test script:

$ python tests/env/local_strategy.py

How to run Jupyter notebooks

Just run:

$ jupyter notebook --no-browser notebooks/

Then open the link in the output in a browser.

Patching Ray

The selected version of Ray RLlib needs to be patched to fix some bugs or undesirable behaviour that has not yet been addressed upstream. The patches are collected in the patches directory and can be applied using the patch command:

patch -p0 < patches/NAME.patch

The patches have only been tested with Ray 2.40.0. They will only work if the virtual environment is named .env and the Python version is 3.12, as the file path is hardcoded into the patch file.

Note: The patch binary is required and preinstalled on Ubuntu. If not available, it can be installed with apt install patch.

The patches are created using the standard diff tool:

diff -Naru .env/.../rllib/example.py .env/.../rllib/example_new.py > patches/NAME.patch

See this reply on StackExchange for more information.

License

Copyright © 2024-2025 Emanuele Petriglia

The source code in this repository is licensed under the Apache License, version 2.0. See the LICENSE file for more information.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.