Skip to content

Commit

Permalink
add requirements
Browse files Browse the repository at this point in the history
  • Loading branch information
mattiasakesson committed Jun 19, 2024
1 parent 7607b67 commit 73659c0
Show file tree
Hide file tree
Showing 2 changed files with 36 additions and 52 deletions.
85 changes: 33 additions & 52 deletions examples/monai-2D-mednist/README.rst
Original file line number Diff line number Diff line change
@@ -1,34 +1,31 @@
FEDn Project: MonAI 2D Classification with the MedNIST Dataset (PyTorch)
------------------------------------------------------------------------
FEDn Project: MNIST (PyTorch)
-----------------------------

This is an example FEDn Project based on the MonAI 2D Classification with the MedNIST Dataset.
This is an example FEDn Project based on the classic hand-written text recognition dataset MNIST.
The example is intented as a minimalistic quickstart and automates the handling of training data
by letting the client download and create its partition of the dataset as it starts up.
by letting the client download and create its partition of the dataset as it starts up.

Links:

- MonAI: https://monai.io/
- Base example notebook: https://github.com/Project-MONAI/tutorials/blob/main/2d_classification/mednist_tutorial.ipynb
- MedNIST dataset: https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/MedNIST.tar.gz
**Note: These instructions are geared towards users seeking to learn how to work
with FEDn in local development mode using Docker/docker-compose. We recommend all new users
to start by following the Quickstart Tutorial: https://fedn.readthedocs.io/en/stable/quickstart.html**

Prerequisites
-------------

Using FEDn Studio:

- `Python 3.8, 3.9, 3.10 or 3.11 <https://www.python.org/downloads>`__
- `A FEDn Studio account <https://fedn.scaleoutsystems.com/signup>`__
- `A FEDn Studio account <https://fedn.scaleoutsystems.com/signup>`__

If using pseudo-distributed mode with docker-compose:

- `Docker <https://docs.docker.com/get-docker>`__
- `Docker Compose <https://docs.docker.com/compose/install>`__


Creating the compute package and seed model
-------------------------------------------

Install fedn:
Install fedn:

.. code-block::
Expand All @@ -39,7 +36,7 @@ Clone this repository, then locate into this directory:
.. code-block::
git clone https://github.com/scaleoutsystems/fedn.git
cd fedn/examples/monai-2D-mednist
cd fedn/examples/mnist-pytorch
Create the compute package:

Expand All @@ -55,63 +52,47 @@ Next, generate a seed model (the first model in a global model trail):
fedn run build --path client
This will create a seed model called 'seed.npz' in the root of the project. This step will take a few minutes, depending on hardware and internet connection (builds a virtualenv).

Download and Prepare the data
-------------------------------------------

Install monai

.. code-block::
pip install monai
Download and divide the data into parts. Set the number of
data parts as an arguments python prepare_data.py NR-OF-DATAPARTS. In the
below command we divide the dataset into 10 parts.
.. code-block::
python prepare_data.py 10
This will create a seed model called 'seed.npz' in the root of the project. This step will take a few minutes, depending on hardware and internet connection (builds a virtualenv).

Using FEDn Studio
-----------------

Follow the guide here to set up your FEDn Studio project and learn how to connect clients (using token authentication): `Studio guide <https://fedn.readthedocs.io/en/stable/studio.html>`__.
On the step "Upload Files", upload 'package.tgz' and 'seed.npz' created above.
On the step "Upload Files", upload 'package.tgz' and 'seed.npz' created above.


Connecting clients:
===================
Modifing the data split:
========================

**NOTE: In case a different data path needs to be set, use the env variable FEDN_DATA_PATH.**
The default traning and test data for this example is downloaded and split direcly by the client when it starts up (see 'startup' entrypoint).
The number of splits and which split used by a client can be controlled via the environment variables ``FEDN_NUM_DATA_SPLITS`` and ``FEDN_DATA_PATH``.
For example, to split the data in 10 parts and start a client using the 8th partiton:

.. code-block::
export FEDN_PACKAGE_EXTRACT_DIR=package
export FEDN_DATA_PATH=./data/
export FEDN_CLIENT_SETTINGS_PATH=<full_path_to_the_dir>/client_settings.yaml
export FEDN_NUM_DATA_SPLITS=10
export FEDN_DATA_PATH=./data/clients/8/mnist.pt
fedn client start -in client.yaml --secure=True --force-ssl
The default is to split the data into 2 partitions and use the first partition.


Connecting clients using Docker:
================================

For convenience, there is a Docker image hosted on ghrc.io with fedn preinstalled. To start a client using Docker:
For convenience, there is a Docker image hosted on ghrc.io with fedn preinstalled. To start a client using Docker:

.. code-block::
docker run \
-v $PWD/client.yaml:/app/client.yaml \
-v $PWD/client_settings.yaml:/app/client_settings.yaml \
-e FEDN_PACKAGE_EXTRACT_DIR=package \
-e FEDN_DATA_PATH=./data/ \
-e FEDN_CLIENT_SETTINGS_PATH=/app/client_settings.yaml \
-e FEDN_NUM_DATA_SPLITS=2 \
-e FEDN_DATA_PATH=/app/package/data/clients/1/mnist.pt \
ghcr.io/scaleoutsystems/fedn/fedn:0.9.0 run client -in client.yaml --force-ssl --secure=True
**NOTE: The following instructions are only for SDK-based client communication and for local development environments using Docker.**


Local development mode using Docker/docker compose
--------------------------------------------------

Expand All @@ -126,8 +107,8 @@ Start a pseudo-distributed FEDn network using docker-compose:
-f docker-compose.override.yaml \
up
This starts up local services for MongoDB, Minio, the API Server, one Combiner and two clients.
You can verify the deployment using these urls:
This starts up local services for MongoDB, Minio, the API Server, one Combiner and two clients.
You can verify the deployment using these urls:

- API Server: http://localhost:8092/get_controller_status
- Minio: http://localhost:9000
Expand All @@ -142,18 +123,18 @@ Upload the package and seed model to FEDn controller using the APIClient. In Pyt
client.set_active_package("package.tgz", helper="numpyhelper")
client.set_active_model("seed.npz")
You can now start a training session with 5 rounds (default):
You can now start a training session with 5 rounds (default):

.. code-block::
client.start_session()
Automate experimentation with several clients
Automate experimentation with several clients
=============================================

If you want to scale the number of clients, you can do so by modifying ``docker-compose.override.yaml``. For example,
in order to run with 3 clients, change the environment variable ``FEDN_NUM_DATA_SPLITS`` to 3, and add one more client
by copying ``client1`` and setting ``FEDN_DATA_PATH`` to ``/app/package/data3/``
If you want to scale the number of clients, you can do so by modifying ``docker-compose.override.yaml``. For example,
in order to run with 3 clients, change the environment variable ``FEDN_NUM_DATA_SPLITS`` to 3, and add one more client
by copying ``client1`` and setting ``FEDN_DATA_PATH`` to ``/app/package/data/clients/3/mnist.pt``


Access message logs and validation data from MongoDB
Expand Down
3 changes: 3 additions & 0 deletions examples/monai-2D-mednist/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
monai
PyYAML
numpy==1.26.4

0 comments on commit 73659c0

Please sign in to comment.