From f1b9384ed17dc54c842a96eded55088fcccc0a90 Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Mon, 1 Jul 2024 16:29:14 +0200 Subject: [PATCH 01/14] Reorganize menu, remove local dev instructions from readme in git repo --- docs/distributed.rst | 100 ++++++++++++++++++++- docs/index.rst | 6 +- docs/projects.rst | 2 +- docs/quickstart.rst | 97 +++------------------ examples/mnist-keras/README.rst | 14 ++- examples/mnist-pytorch/README.rst | 140 +++--------------------------- 6 files changed, 131 insertions(+), 228 deletions(-) diff --git a/docs/distributed.rst b/docs/distributed.rst index 13803dd3f..d53df2d3c 100644 --- a/docs/distributed.rst +++ b/docs/distributed.rst @@ -1,9 +1,103 @@ + +Local development +================= + +.. note:: + These instructions are for users wanting to set up a local development deployment of FEDn (i.e. without FEDn Studio). + This requires practical knowledge of Docker and docker-compose. + +Running the FEDn development sandbox (docker-compose) +===================================================== + +During development on FEDn, and when working on own aggregators/helpers, it is +useful to have a local development setup of the core FEDn services (controller, combiner, database, object store). +For this, we provide Dockerfiles and docker-compose template. + +To start a development sandbox for FEDn using docker-compose: + +.. code-block:: + + docker compose \ + -f ../../docker-compose.yaml \ + -f docker-compose.override.yaml \ + up + +This starts up local services for MongoDB, Minio, the API Server, one Combiner and two clients. +You can verify the deployment using these urls: + +- API Server: http://localhost:8092/get_controller_status +- Minio: http://localhost:9000 +- Mongo Express: http://localhost:8081 + +This setup does not include the security features of Studio, and thus will not require authentication of clients. +To use the APIClient to test a compute package and seed model against a local FEDn deployment: + +.. code-block:: + + from fedn import APIClient + client = APIClient(host="localhost", port=8092) + client.set_active_package("package.tgz", helper="numpyhelper") + client.set_active_model("seed.npz") + + +To connect a native FEDn client, you need to make sure that the combiner service can be resolved using the name "combiner". +One way to achieve this is to edit your '/etc/hosts' and add a line '127.0.0.1 combiner'. + +Access message logs and validation data from MongoDB +==================================================== + +You can access and download event logs and validation data via the API, and you can also as a developer obtain +the MongoDB backend data using pymongo or via the MongoExpress interface: + +- http://localhost:8081/db/fedn-network/ + +Username and password are found in 'docker-compose.yaml'. + +Access global models +==================== + +You can obtain global model updates from the 'fedn-models' bucket in Minio: + +- http://localhost:9000 + +Username and password are found in 'docker-compose.yaml'. + +Reset the FEDn deployment +========================= + +To purge all data from a deployment incuding all session and round data, access the MongoExpress UI interface and +delete the entire ``fedn-network`` collection. Then restart all services. + +Clean up +======== +You can clean up by running + +.. code-block:: + + docker-compose -f ../../docker-compose.yaml -f docker-compose.override.yaml down -v + + +Connecting clients using Docker: +================================ + +For convenience, we distribute a Docker image hosted on ghrc.io with FEDn preinstalled. For example, to start a client for the MNIST PyTorch example using Docker +and FEDN 0.10.0, run this from the example folder: + +.. code-block:: + + docker run \ + -v $PWD/client.yaml:/app/client.yaml \ + -e FEDN_PACKAGE_EXTRACT_DIR=package \ + -e FEDN_NUM_DATA_SPLITS=2 \ + -e FEDN_DATA_PATH=/app/package/data/clients/1/mnist.pt \ + ghcr.io/scaleoutsystems/fedn/fedn:0.10.0 run client -in client.yaml --force-ssl --secure=True + + Self-managed distributed deployment =================================== -This tutorial outlines the steps for deploying the FEDn framework over a **local network**, using a single workstation or laptop as -the host for the servier-side components, and other hosts or devices as clients. For general steps on how to run FEDn, see the quickstart tutorials. - +You can use different hosts for the various FEDn services. These instructions shows how to set up FEDn on a **local network** using a single workstation or laptop as +the host for the servier-side components, and other hosts or devices as clients. .. note:: For a secure and production-grade deployment solution over **public networks**, explore the FEDn Studio service at diff --git a/docs/index.rst b/docs/index.rst index f73a9aa5c..cb011fa3b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -3,20 +3,20 @@ :caption: Introduction introduction - quickstart .. toctree:: :maxdepth: 1 :caption: Documentation - studio - distributed + quickstart apiclient projects + studio architecture aggregators helpers auth + distributed .. toctree:: :maxdepth: 1 diff --git a/docs/projects.rst b/docs/projects.rst index 6397a3a56..507b0c610 100644 --- a/docs/projects.rst +++ b/docs/projects.rst @@ -1,6 +1,6 @@ .. _projects-label: -FEDn Projects +Creating your own FEDn Projects ================================================ A FEDn project is a convention for packaging/wrapping machine learning code to be used for federated learning with FEDn. At the core, diff --git a/docs/quickstart.rst b/docs/quickstart.rst index d231aca9e..65e22d8ad 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -7,21 +7,30 @@ Getting started with FEDn **Prerequisites** -- `Python >=3.8, <=3.11 `__ +- `Python >=3.8, <=3.12 `__ - `A FEDn Studio account `__ Set up a FEDn Studio Project ---------------------------- -Start by creating an account in FEDn Studio and set up a project by following the instruction here: :ref:`studio`. +Start by creating an account in Studio. Head over to `fedn.scaleoutsystems.com/signup `_ and sign up. + +Logged into Studio, do: + +1. Click on the "New Project" button in the top right corner of the screen. +2. Continue by clicking the "Create button". The FEDn template contains all the services necessary to start a federation. +3. Enter the project name (mandatory). The project description is optional. +4. Click the "Create" button to create the project. + +You Studio project provides all server side components. Next, you will set up your local machine / client and create a FEDn project. Install FEDn ------------ **Using pip** -Install the FEDn package using pip: +On you local machine/client, install the FEDn package using pip: .. code-block:: bash @@ -101,10 +110,10 @@ Then start the client by running the following command in the root of the projec Repeat the above for the number of clients you want to use. A normal laptop should be able to handle several clients for this example. -**Modifing the data split:** +**Modifying the data split:** The default traning and test data for this example (MNIST) is for convenience downloaded and split by the client when it starts up (see 'startup' entrypoint). -The number of splits and which split used by a client can be controlled via the environment variables ``FEDN_NUM_DATA_SPLITS`` and ``FEDN_DATA_PATH``. +The number of splits and which split is used by a client can be controlled via the environment variables ``FEDN_NUM_DATA_SPLITS`` and ``FEDN_DATA_PATH``. For example, to split the data in 10 parts and start a client using the 8th partiton: .. tabs:: @@ -188,81 +197,3 @@ With you first FEDn federation set up, we suggest that you take a close look at and how you develop your own FEDn projects: - :ref:`projects-label` - - -Local development deployment (using docker compose) ----------------------------------------------------------- - -.. note:: - These instructions are for users wanting to set up a local development deployment of FEDn (wihout Studio). - This requires basic knowledge of Docker and docker-compose. - The main use-case for this is rapid iteration while developing the FEDn Project, - development of aggregator plugins, etc. - -Follow the steps above to install FEDn, generate 'package.tgz' and 'seed.tgz'. Then, instead of -using a Studio project for a managed FEDn server-side, start a local FEDn network -using docker-compose: - -.. code-block:: - - docker compose \ - -f ../../docker-compose.yaml \ - -f docker-compose.override.yaml \ - up - -This starts up local services for MongoDB, Minio, the API Server, one Combiner and two clients. -You can verify the deployment using these urls: - -- API Server: http://localhost:8092/get_controller_status -- Minio: http://localhost:9000 -- Mongo Express: http://localhost:8081 - -Upload the package and seed model to FEDn controller using the APIClient. In Python: - -.. code-block:: - - from fedn import APIClient - client = APIClient(host="localhost", port=8092) - client.set_active_package("package.tgz", helper="numpyhelper") - client.set_active_model("seed.npz") - -You can now start a training session with 5 rounds (default): - -.. code-block:: - - client.start_session() - -**Automate experimentation with several clients** - -If you want to scale the number of clients, you can do so by modifying ``docker-compose.override.yaml``. For example, -in order to run with 3 clients, change the environment variable ``FEDN_NUM_DATA_SPLITS`` to 3, and add one more client -by copying ``client1`` and setting ``FEDN_DATA_PATH`` to ``/app/package/data/clients/3/mnist.pt`` - - -**Access message logs and validation data from MongoDB** - -You can access and download event logs and validation data via the API, and you can also as a developer obtain -the MongoDB backend data using pymongo or via the MongoExpress interface: - -- http://localhost:8081/db/fedn-network/ - -The credentials are as set in docker-compose.yaml in the root of the repository. - -**Access global models** - -You can obtain global model updates from the 'fedn-models' bucket in Minio: - -- http://localhost:9000 - -**Reset the FEDn deployment** - -To purge all data from a deployment incuding all session and round data, access the MongoExpress UI interface and -delete the entire ``fedn-network`` collection. Then restart all services. - -**Clean up** - -You can clean up by running - -.. code-block:: - - docker-compose -f ../../docker-compose.yaml -f docker-compose.override.yaml down -v diff --git a/examples/mnist-keras/README.rst b/examples/mnist-keras/README.rst index 0f2ec9cf1..22bf61c8d 100644 --- a/examples/mnist-keras/README.rst +++ b/examples/mnist-keras/README.rst @@ -8,7 +8,7 @@ This is a TF/Keras version of the Quickstart Tutorial (PyTorch) FEDn Project. Fo Prerequisites ------------------------------------------- -- `Python 3.8, 3.9, 3.10 or 3.11 `__ +- `Python >=3.8, <=3.12 `__ Creating the compute package and seed model ------------------------------------------- @@ -24,7 +24,7 @@ Clone this repository, then locate into this directory: .. code-block:: git clone https://github.com/scaleoutsystems/fedn.git - cd fedn/examples/mnist-pytorch + cd fedn/examples/mnist-keras Create the compute package: @@ -42,13 +42,9 @@ Next, generate a seed model (the first model in a global model trail): This step will take a few minutes, depending on hardware and internet connection (builds a virtualenv). -Using FEDn Studio ------------------- +Running the project on FEDn +---------------------------- -To set up your FEDn Studio project and learn how to connect clients (using JWT token authentication), follow this guide: https://fedn.readthedocs.io/en/latest/studio.html. On the +To set up your FEDn Studio project and connect clients, follow this guide: https://fedn.readthedocs.io/en/latest/studio.html. On the step "Upload Files", upload 'package.tgz' and 'seed.npz' created above. -Using pseudo-distributed mode (local development) -------------------------------------------------- - -See the PyTorch version of this example for detailed instructions on how to deploy FEDn in `local development mode <../mnist-pytorch/README.rst>`_. diff --git a/examples/mnist-pytorch/README.rst b/examples/mnist-pytorch/README.rst index 71e5de2d1..0d8044327 100644 --- a/examples/mnist-pytorch/README.rst +++ b/examples/mnist-pytorch/README.rst @@ -2,25 +2,15 @@ FEDn Project: MNIST (PyTorch) ----------------------------- This is an example FEDn Project based on the classic hand-written text recognition dataset MNIST. -The example is intented as a minimalistic quickstart and automates the handling of training data -by letting the client download and create its partition of the dataset as it starts up. +The example is intented as a minimalistic quickstart. - **Note: These instructions are geared towards users seeking to learn how to work - with FEDn in local development mode using Docker/docker-compose. We recommend all new users - to start by following the Quickstart Tutorial: https://fedn.readthedocs.io/en/stable/quickstart.html** + **Note: We recommend that all new users start by following the Quickstart Tutorial: https://fedn.readthedocs.io/en/stable/quickstart.html** Prerequisites ------------- -Using FEDn Studio: - -- `Python 3.8, 3.9, 3.10 or 3.11 `__ -- `A FEDn Studio account `__ - -If using pseudo-distributed mode with docker-compose: - -- `Docker `__ -- `Docker Compose `__ +- `Python >=3.8, <=3.12 `__ +- `A project in FEDn Studio `__ Creating the compute package and seed model ------------------------------------------- @@ -44,126 +34,18 @@ Create the compute package: fedn package create --path client -This should create a file 'package.tgz' in the project folder. +This creates a file 'package.tgz' in the project folder. -Next, generate a seed model (the first model in a global model trail): +Next, generate the seed model: .. code-block:: fedn run build --path client -This will create a seed model called 'seed.npz' in the root of the project. This step will take a few minutes, depending on hardware and internet connection (builds a virtualenv). - -Using FEDn Studio ------------------ - -Follow the guide here to set up your FEDn Studio project and learn how to connect clients (using token authentication): `Studio guide `__. -On the step "Upload Files", upload 'package.tgz' and 'seed.npz' created above. - - -Modifing the data split: -======================== - -The default traning and test data for this example is downloaded and split direcly by the client when it starts up (see 'startup' entrypoint). -The number of splits and which split used by a client can be controlled via the environment variables ``FEDN_NUM_DATA_SPLITS`` and ``FEDN_DATA_PATH``. -For example, to split the data in 10 parts and start a client using the 8th partiton: - -.. code-block:: - - export FEDN_PACKAGE_EXTRACT_DIR=package - export FEDN_NUM_DATA_SPLITS=10 - export FEDN_DATA_PATH=./data/clients/8/mnist.pt - fedn client start -in client.yaml --secure=True --force-ssl - -The default is to split the data into 2 partitions and use the first partition. - - -Connecting clients using Docker: -================================ - -For convenience, there is a Docker image hosted on ghrc.io with fedn preinstalled. To start a client using Docker: - -.. code-block:: - - docker run \ - -v $PWD/client.yaml:/app/client.yaml \ - -e FEDN_PACKAGE_EXTRACT_DIR=package \ - -e FEDN_NUM_DATA_SPLITS=2 \ - -e FEDN_DATA_PATH=/app/package/data/clients/1/mnist.pt \ - ghcr.io/scaleoutsystems/fedn/fedn:0.9.0 run client -in client.yaml --force-ssl --secure=True - - -Local development mode using Docker/docker compose --------------------------------------------------- - -Follow the steps above to install FEDn, generate 'package.tgz' and 'seed.tgz'. - -Start a pseudo-distributed FEDn network using docker-compose: - -.. code-block:: - - docker compose \ - -f ../../docker-compose.yaml \ - -f docker-compose.override.yaml \ - up +This will create a model file 'seed.npz' in the root of the project. This step will take a few minutes, depending on hardware and internet connection (builds a virtualenv). -This starts up local services for MongoDB, Minio, the API Server, one Combiner and two clients. -You can verify the deployment using these urls: - -- API Server: http://localhost:8092/get_controller_status -- Minio: http://localhost:9000 -- Mongo Express: http://localhost:8081 - -Upload the package and seed model to FEDn controller using the APIClient. In Python: - -.. code-block:: - - from fedn import APIClient - client = APIClient(host="localhost", port=8092) - client.set_active_package("package.tgz", helper="numpyhelper") - client.set_active_model("seed.npz") - -You can now start a training session with 5 rounds (default): - -.. code-block:: - - client.start_session() - -Automate experimentation with several clients -============================================= - -If you want to scale the number of clients, you can do so by modifying ``docker-compose.override.yaml``. For example, -in order to run with 3 clients, change the environment variable ``FEDN_NUM_DATA_SPLITS`` to 3, and add one more client -by copying ``client1`` and setting ``FEDN_DATA_PATH`` to ``/app/package/data/clients/3/mnist.pt`` - - -Access message logs and validation data from MongoDB -==================================================== - -You can access and download event logs and validation data via the API, and you can also as a developer obtain -the MongoDB backend data using pymongo or via the MongoExpress interface: - -- http://localhost:8081/db/fedn-network/ - -The credentials are as set in docker-compose.yaml in the root of the repository. - -Access global models -==================== - -You can obtain global model updates from the 'fedn-models' bucket in Minio: - -- http://localhost:9000 - -Reset the FEDn deployment -========================= - -To purge all data from a deployment incuding all session and round data, access the MongoExpress UI interface and -delete the entire ``fedn-network`` collection. Then restart all services. - -Clean up -======== -You can clean up by running - -.. code-block:: +Running the project on FEDn +---------------------------- - docker-compose -f ../../docker-compose.yaml -f docker-compose.override.yaml down -v +To set up your FEDn Studio project and connect clients, follow this guide: https://fedn.readthedocs.io/en/latest/studio.html. On the +step "Upload Files", upload 'package.tgz' and 'seed.npz' created above. From 96da4eefbb3b519935f837bebb8987791dabd995 Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Mon, 1 Jul 2024 16:31:06 +0200 Subject: [PATCH 02/14] Rename distributed.rst to developer.rst --- docs/{distributed.rst => developer.rst} | 0 docs/index.rst | 4 ++-- 2 files changed, 2 insertions(+), 2 deletions(-) rename docs/{distributed.rst => developer.rst} (100%) diff --git a/docs/distributed.rst b/docs/developer.rst similarity index 100% rename from docs/distributed.rst rename to docs/developer.rst diff --git a/docs/index.rst b/docs/index.rst index cb011fa3b..04ff85b07 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -9,14 +9,14 @@ :caption: Documentation quickstart - apiclient projects studio + apiclient architecture aggregators helpers auth - distributed + developer .. toctree:: :maxdepth: 1 From fa4b928a562a96cf2ae22002627703816efcac41 Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Thu, 4 Jul 2024 15:09:20 +0200 Subject: [PATCH 03/14] Improved quickstart --- docs/architecture.rst | 2 + docs/index.rst | 4 +- docs/projects.rst | 22 ++++++---- docs/quickstart.rst | 97 ++++++++++++++++++++++++++++--------------- 4 files changed, 80 insertions(+), 45 deletions(-) diff --git a/docs/architecture.rst b/docs/architecture.rst index a820e7e20..85e2430da 100644 --- a/docs/architecture.rst +++ b/docs/architecture.rst @@ -1,3 +1,5 @@ +.. _architecture-label: + Architecture overview ===================== diff --git a/docs/index.rst b/docs/index.rst index 04ff85b07..7cda27188 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -3,13 +3,13 @@ :caption: Introduction introduction + quickstart + projects .. toctree:: :maxdepth: 1 :caption: Documentation - quickstart - projects studio apiclient architecture diff --git a/docs/projects.rst b/docs/projects.rst index 507b0c610..1c9a078e2 100644 --- a/docs/projects.rst +++ b/docs/projects.rst @@ -1,8 +1,11 @@ .. _projects-label: -Creating your own FEDn Projects +Building your own projects ================================================ +This guide explains how a FEDn project is structured, and details how to develop your own +projects for your own use-cases. + A FEDn project is a convention for packaging/wrapping machine learning code to be used for federated learning with FEDn. At the core, a project is a directory of files (often a Git repository), containing your machine learning code, FEDn entry points, and a specification of the runtime environment (python environment or a Docker image). The FEDn API and command-line tools provides functionality @@ -28,9 +31,9 @@ We recommend that projects have roughly the following folder and file structure: | └ Dockerfile / docker-compose.yaml | -The "client" folder is referred to as the *compute package*. The file fedn.yaml is the FEDn Project File. It informs the FEDn Client of the code entry points to execute when computing model updates (local training) and validating models (optionally) . -When deploying the project to FEDn, the client folder will be compressed as a .tgz bundle and uploaded to the FEDn controller. FEDn can then manage the distribution of the compute package to each client/data provider when they connect. -Upon recipt of the bundle, the client will unpack it and stage it locally. +The ``client`` folder is commonly referred to as the *compute package*. The file ``fedn.yaml`` is the FEDn Project File. It contains information about the ``entry points``. The entry points are used by the client to compute model updates (local training) and local validations (optional) . +To run a project in FEDn, the client folder is compressed as a .tgz bundle and pushed to the FEDn controller. FEDn then manages the distribution of the compute package to each client. +Upon recipt of the package, a client will unpack it and stage it locally. .. image:: img/ComputePackageOverview.png :alt: Compute package overview @@ -62,11 +65,12 @@ what environment to execute those entrypoints in. Environment ^^^^^^^^^^^ - -The software environment to be used to exectute the entry points. This should specify all client side dependencies of the project. -FEDn currently supports Virtualenv environments, with packages on PyPI. When a project specifies a **python_env**, the FEDn -client will create an isolated virtual environment and install the project dependencies into it before starting up the client. +It is assumed that all entry points are executable within the client runtime environment. As a user, you have two main options +to specify the environment: + + 1. Provide a ``python_env`` in the ``fedn.yaml`` file. In this case, FEDn will create an isolated virtual environment and install the project dependencies into it before starting up the client. FEDn currently supports Virtualenv environments, with packages on PyPI. + 2. Manage the environment manually. Here you have several options, such as managing your own virtualenv, running in a Docker container, etc. Remove the ``python_env`` tag from ``fedn.yaml`` to handle the environment manually. Entry Points ^^^^^^^^^^^^ @@ -75,7 +79,7 @@ There are up to four Entry Points to be specified. **Build Entrypoint (build, optional):** -This entrypoint is usually called **once** for building artifacts such as initial seed models. However, it not limited to artifacts, and can be used for any kind of setup that needs to be done before the client starts up. +This entrypoint is intended to be called **once** for building artifacts such as initial seed models. However, it not limited to artifacts, and can be used for any kind of setup that needs to be done before the client starts up. **Startup Entrypoint (startup, optional):** diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 65e22d8ad..1c880ece4 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -2,7 +2,7 @@ Getting started with FEDn ========================= .. note:: - This tutorial is a quickstart guide to FEDn based on a pre-made FEDn Project. It is designed to serve as a minimalistic starting point for developers. + This tutorial is a quickstart guide to FEDn based on a pre-made FEDn Project. It is designed to serve as a starting point for new developers. To learn about FEDn Projects in order to develop your own federated machine learning projects, see :ref:`projects-label`. **Prerequisites** @@ -11,7 +11,7 @@ Getting started with FEDn - `A FEDn Studio account `__ -Set up a FEDn Studio Project +Start a FEDn Studio Project ---------------------------- Start by creating an account in Studio. Head over to `fedn.scaleoutsystems.com/signup `_ and sign up. @@ -23,10 +23,13 @@ Logged into Studio, do: 3. Enter the project name (mandatory). The project description is optional. 4. Click the "Create" button to create the project. -You Studio project provides all server side components. Next, you will set up your local machine / client and create a FEDn project. +.. image:: img/studio_project_overview.png -Install FEDn ------------- +When these steps are complete, you will see a Studio project similar to the above image. The Studio project provides all server side components of FEDn needed to manage +federated training. We will use this project in a later stage to run the federated experiments. But first, we will set up the local client. + +Install FEDn on your client +---------------------------- **Using pip** @@ -50,35 +53,45 @@ It is recommended to use a virtual environment when installing FEDn. .. _package-creation: -Initialize FEDn with the client code bundle and seed model ----------------------------------------------------------- -Next, we will prepare the client. The key part of a FEDn Project is the client definition - -code that contains entrypoints for training and (optionally) validating a model update on the client. +Create the compute package and seed model +-------------------------------------------- + +Next, we will prepare the client. For illustrative purposes, we use one of the pre-defined projects in the FEDn repository, ``minst-pytorch``. -Locate into ``examples/mnist-pytorch`` and familiarize yourself with the project structure. The dependencies needed in the client environment are specified -in ``client/python_env.yaml``. +In order to train a federated model using FEDn, your Studio project needs to be initialized with a ``compute package`` and a ``seed model``. The compute package is a code bundle containing the +code used by the client to execute local training and local validation. The seed model is a first version of the global model. -In order to train a federated model using FEDn, your Studio project needs to be initialized with a compute package and a seed model. The compute package is a bundle -of the client specification, and the seed model is a first version of the global model. +Locate into ``examples/mnist-pytorch`` folder in the cloned fedn repository. The compute package is located in the folder ``client``. -Create a package of the fedn project (assumes your current working directory is in the root of the project /examples/mnist-pytorch): +Create a package of the fedn project. Standing in ``examples/mnist-pytorch``: .. code-block:: fedn package create --path client -This will create a package called 'package.tgz' in the root of the project. +This will create a package called ``package.tgz`` in the root of the project. -Next, run the build entrypoint defined in ``client/fedn.yaml`` to build the model artifact. +Next, create the seed model: .. code-block:: fedn run build --path client -This will create a seed model called 'seed.npz' in the root of the project. We will now upload these to your Studio project using the FEDn APIClient. +This will create a seed model called ``seed.npz`` in the root of the project. We will now upload these to your Studio project using the FEDn APIClient. + +For a detailed explaination of the FEDn Project with instructions for how to create your own project, see this guide: :ref:`projects-label` + +Initialize your FEDn Studio Project +------------------------------------ + +In the Studio UI, navigate to the project you created above and click on the "Sessions" tab. Click on the "New Session" button. Under the "Compute package" tab, select a name and upload the generated package file. Under the "Seed model" tab, upload the generated seed file: -**Upload the package and seed model** +.. image:: img/upload_package.png + +**Upload the package and seed model using the Python APIClient** + +It is also possible to upload a package and seed model using the Python API Client. .. note:: You need to create an API admin token and use the token to authenticate the APIClient. @@ -100,9 +113,9 @@ Upload the package and seed model using the APIClient: Configure and attach clients ---------------------------- -Each local client needs an access token in order to connect. These tokens are issued from your Studio Project. Go to the 'Clients' tab and click 'Connect client'. -Download a client configuration file and save it to the root of the examples/mnist-pytorch folder. Rename the file to 'client.yaml'. -Then start the client by running the following command in the root of the project: +Each local client needs an access token in order to connect. These tokens are issued from your Studio Project. Go to the Clients' tab and click 'Connect client'. +Download a client configuration file and save it to the root of the ``examples/mnist-pytorch folder``. Rename the file to 'client.yaml'. +Then start the client by running the following command: .. code-block:: @@ -110,7 +123,7 @@ Then start the client by running the following command in the root of the projec Repeat the above for the number of clients you want to use. A normal laptop should be able to handle several clients for this example. -**Modifying the data split:** +**Modifying the data split (multiple-clients, optional):** The default traning and test data for this example (MNIST) is for convenience downloaded and split by the client when it starts up (see 'startup' entrypoint). The number of splits and which split is used by a client can be controlled via the environment variables ``FEDN_NUM_DATA_SPLITS`` and ``FEDN_DATA_PATH``. @@ -138,7 +151,21 @@ For example, to split the data in 10 parts and start a client using the 8th part Start a training session ------------------------ -You are now ready to start training the model using the APIClient: +In Studio click on the "Sessions" link, then the "New session" button in the upper right corner. Click the "Start session" tab and enter your desirable settings (or use default) and hit the "Start run" button. In the terminal where your are running your client you should now see some activity. When the round is completed, you can see the results in the FEDn Studio UI on the "Models" page. + +**Watch the training progress** + +Once a training session is started, you can monitor the progress of the training by navigating to "Sessions" and click on the "Open" button of the active session. The session page will list the models as soon as they are generated. To get more information about a particular model, navigate to the model page by clicking the model name. From the model page you can download the model weights and get validation metrics. + +To get an overview of how the models have evolved over time, navigate to the "Models" tab in the sidebar. Here you can see a list of all models generated across sessions along with a graph showing some metrics of how the models are performing. + +.. image:: img/studio_model_overview.png + +.. _studio-api: + +**Control training sessions using the Python APIClient** + +You can also issue training sessions using the APIClient: .. code:: python @@ -153,12 +180,7 @@ You are now ready to start training the model using the APIClient: >>> validations = client.get_validations(model_id=model_id) -Please see :py:mod:`fedn.network.api` for more details on the APIClient. - -.. note:: - - In FEDn Studio, you can start a training session by going to the 'Sessions' tab and click 'Start session'. See :ref:`studio` for a - step-by-step guide for how to control experiments using the UI. +Please see :py:mod:`fedn.network.api` for more details on how to use the APIClient. Access model updates -------------------- @@ -167,7 +189,7 @@ Access model updates In FEDn Studio, you can access global model updates by going to the 'Models' or 'Sessions' tab. Here you can download model updates, metrics (as csv) and view the model trail. -You can access global model updates via the APIClient: +You can also access global model updates via the APIClient: .. code:: python @@ -175,7 +197,8 @@ You can access global model updates via the APIClient: >>> client.download_model("", path="model.npz") -**Connecting clients using Docker** +Connecting clients using Docker +-------------------------------- You can also use Docker to containerize the client. For convenience, there is a Docker image hosted on ghrc.io with fedn preinstalled. @@ -188,12 +211,18 @@ To start a client using Docker: -e FEDN_PACKAGE_EXTRACT_DIR=package \ -e FEDN_NUM_DATA_SPLITS=2 \ -e FEDN_DATA_PATH=/app/package/data/clients/1/mnist.pt \ - ghcr.io/scaleoutsystems/fedn/fedn:0.9.0 run client -in client.yaml --force-ssl --secure=True + ghcr.io/scaleoutsystems/fedn/fedn:0.10.0 run client -in client.yaml --force-ssl --secure=True -**Where to go from here?** +Where to go from here? +------------------------ -With you first FEDn federation set up, we suggest that you take a close look at how a FEDn project is structured +With you first FEDn federated project set up, we suggest that you take a close look at how a FEDn project is structured and how you develop your own FEDn projects: - :ref:`projects-label` + +You can also dive into the architecture overview to learn more about how FEDn is designed and works under the hood: +- :ref:`architecture-label` + + From 4b16ea83af77490863117dad6a43e162672e0e57 Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Sat, 6 Jul 2024 08:36:08 +0200 Subject: [PATCH 04/14] Updated project page --- docs/projects.rst | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/projects.rst b/docs/projects.rst index 1c9a078e2..46e3630a0 100644 --- a/docs/projects.rst +++ b/docs/projects.rst @@ -63,8 +63,7 @@ what environment to execute those entrypoints in. command: python validate.py -Environment -^^^^^^^^^^^ +**Environment** It is assumed that all entry points are executable within the client runtime environment. As a user, you have two main options to specify the environment: @@ -72,8 +71,7 @@ to specify the environment: 1. Provide a ``python_env`` in the ``fedn.yaml`` file. In this case, FEDn will create an isolated virtual environment and install the project dependencies into it before starting up the client. FEDn currently supports Virtualenv environments, with packages on PyPI. 2. Manage the environment manually. Here you have several options, such as managing your own virtualenv, running in a Docker container, etc. Remove the ``python_env`` tag from ``fedn.yaml`` to handle the environment manually. -Entry Points -^^^^^^^^^^^^ +**Entry Points** There are up to four Entry Points to be specified. @@ -81,6 +79,13 @@ There are up to four Entry Points to be specified. This entrypoint is intended to be called **once** for building artifacts such as initial seed models. However, it not limited to artifacts, and can be used for any kind of setup that needs to be done before the client starts up. +To invoke the build entrypoint using the CLI: + +.. code-block:: bash + fedn build -- + + + **Startup Entrypoint (startup, optional):** @@ -111,8 +116,7 @@ model on local test/validation data. It should read a model update from file, va The validate entrypoint is optional. -Example train entry point -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +**Example train entry point** Below is an example training entry point taken from the PyTorch getting stated project. @@ -206,7 +210,7 @@ using the pytorch helper module. The fifth function (_init_seed) is used to init The seventh function (_validate) is used to validate the model, again observe the two first arguments which will be set by the FEDn client. -Packaging for distribution +Build a compute package -------------------------- To deploy a project to FEDn (Studio or pseudo-local) we simply compress the *client* folder as .tgz file. using fedn command line tool or manually: @@ -227,15 +231,15 @@ by looking at the code above. Here we assume that the dataset is present in a fi the exection of entrypoint.py. Then, independent on the preferred way to run the client (native, Docker, K8s etc) this structure needs to be maintained for this particular compute package. Note however, that there are many ways to accompish this on a local operational level. -Testing the entry points before deploying the package to FEDn --------------------------------------------------------------- +Testing the entry points locally +--------------------------------- -We recommend you to test your code before deploying it to FEDn for distibution to clients. You can conveniently test *train* and *validate* by: +We recommend you to test your entrypoints locally before uploading the compute package to Studio. You can test *train* and *validate* by (example for the mnist-keras +project): .. code-block:: bash python train.py ../seed.npz ../model_update.npz --data_path ../data/mnist.npz python validate.py ../model_update.npz ../validation.json --data_path ../data/mnist.npz -Once everything works as expected you can start the federated network, upload the .tgz compute package and the initial model (use :py:meth:`fedn.network.api.client.APIClient.set_initial_model` for uploading an initial model). - +Note that we here assume execution in the correct Python environment. From cbcc752abd0b7be6c9b4205ed29e9cad8b0b3239 Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Sat, 6 Jul 2024 08:56:47 +0200 Subject: [PATCH 05/14] Improved readibility a bit --- docs/projects.rst | 4 +-- docs/quickstart.rst | 79 ++++++++++++++++++++++++--------------------- 2 files changed, 45 insertions(+), 38 deletions(-) diff --git a/docs/projects.rst b/docs/projects.rst index 46e3630a0..2b86faa62 100644 --- a/docs/projects.rst +++ b/docs/projects.rst @@ -1,6 +1,6 @@ .. _projects-label: -Building your own projects +Develop your own project ================================================ This guide explains how a FEDn project is structured, and details how to develop your own @@ -44,7 +44,7 @@ The above figure provides a logical view of how FEDn uses the compute package (c recieves a model update request, it calls upon a Dispatcher that looks up entry point definitions in the compute package from the FEDn Project File. -FEDn Project File (fedn.yaml) +The Project File (fedn.yaml) ------------------------------ FEDn uses on a project file named 'fedn.yaml' to specify which entrypoints to execute when the client recieves a training or validation request, and diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 1c880ece4..a73434328 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -3,7 +3,7 @@ Getting started with FEDn .. note:: This tutorial is a quickstart guide to FEDn based on a pre-made FEDn Project. It is designed to serve as a starting point for new developers. - To learn about FEDn Projects in order to develop your own federated machine learning projects, see :ref:`projects-label`. + To learn how to develop your own project from scratch, see :ref:`projects-label`. **Prerequisites** @@ -11,8 +11,8 @@ Getting started with FEDn - `A FEDn Studio account `__ -Start a FEDn Studio Project ----------------------------- +1. Start a FEDn Studio Project +------------------------------ Start by creating an account in Studio. Head over to `fedn.scaleoutsystems.com/signup `_ and sign up. @@ -28,8 +28,8 @@ Logged into Studio, do: When these steps are complete, you will see a Studio project similar to the above image. The Studio project provides all server side components of FEDn needed to manage federated training. We will use this project in a later stage to run the federated experiments. But first, we will set up the local client. -Install FEDn on your client ----------------------------- +2. Install FEDn on your client +------------------------------- **Using pip** @@ -53,24 +53,23 @@ It is recommended to use a virtual environment when installing FEDn. .. _package-creation: +Next, we will prepare the client. For illustrative purposes, we use one of the pre-defined projects in the FEDn repository, ``minst-pytorch``. -Create the compute package and seed model +3. Create the compute package and seed model -------------------------------------------- -Next, we will prepare the client. For illustrative purposes, we use one of the pre-defined projects in the FEDn repository, ``minst-pytorch``. - In order to train a federated model using FEDn, your Studio project needs to be initialized with a ``compute package`` and a ``seed model``. The compute package is a code bundle containing the -code used by the client to execute local training and local validation. The seed model is a first version of the global model. - +code used by the client to execute local training and local validation. The seed model is a first version of the global model. For a detailed explaination of the compute package and seed model, see this guide: :ref:`projects-label` + Locate into ``examples/mnist-pytorch`` folder in the cloned fedn repository. The compute package is located in the folder ``client``. -Create a package of the fedn project. Standing in ``examples/mnist-pytorch``: +Create a compute package: .. code-block:: fedn package create --path client -This will create a package called ``package.tgz`` in the root of the project. +This will create a file called ``package.tgz`` in the root of the project. Next, create the seed model: @@ -78,12 +77,13 @@ Next, create the seed model: fedn run build --path client -This will create a seed model called ``seed.npz`` in the root of the project. We will now upload these to your Studio project using the FEDn APIClient. +This will create a file called ``seed.npz`` in the root of the project. + +Next will now upload these files to your Studio project: -For a detailed explaination of the FEDn Project with instructions for how to create your own project, see this guide: :ref:`projects-label` -Initialize your FEDn Studio Project ------------------------------------- +4. Initialize your FEDn Studio Project +-------------------------------------- In the Studio UI, navigate to the project you created above and click on the "Sessions" tab. Click on the "New Session" button. Under the "Compute package" tab, select a name and upload the generated package file. Under the "Seed model" tab, upload the generated seed file: @@ -110,23 +110,30 @@ Upload the package and seed model using the APIClient: >>> client.set_active_model("seed.npz") -Configure and attach clients ----------------------------- +5. Configure and attach clients +------------------------------- -Each local client needs an access token in order to connect. These tokens are issued from your Studio Project. Go to the Clients' tab and click 'Connect client'. -Download a client configuration file and save it to the root of the ``examples/mnist-pytorch folder``. Rename the file to 'client.yaml'. -Then start the client by running the following command: +**Generate an access token for the client (in Studio)** + +Each local client needs an access token in order to connect securely to the FEDn server. These tokens are issued from your Studio Project. +Go to the Clients' tab and click 'Connect client'. Download a client configuration file and save it to the root of the ``examples/mnist-pytorch folder``. +Rename the file to 'client.yaml'. + +**Start the client (on your local machine)** + +Now we can start the client by running the following command: .. code-block:: fedn run client -in client.yaml --secure=True --force-ssl -Repeat the above for the number of clients you want to use. A normal laptop should be able to handle several clients for this example. +Repeat these two steps (generate an access token and start a local client) for the number of clients you want to use. +A normal laptop should be able to handle several clients for this example. **Modifying the data split (multiple-clients, optional):** -The default traning and test data for this example (MNIST) is for convenience downloaded and split by the client when it starts up (see 'startup' entrypoint). -The number of splits and which split is used by a client can be controlled via the environment variables ``FEDN_NUM_DATA_SPLITS`` and ``FEDN_DATA_PATH``. +The default traning and test data for this particular example (mnist-pytorch) is for convenience downloaded and split automatically by the client when it starts up (see the 'startup' entrypoint). +The number of splits and which split to use by a client can be controlled via the environment variables ``FEDN_NUM_DATA_SPLITS`` and ``FEDN_DATA_PATH``. For example, to split the data in 10 parts and start a client using the 8th partiton: .. tabs:: @@ -148,21 +155,24 @@ For example, to split the data in 10 parts and start a client using the 8th part fedn client start -in client.yaml --secure=True --force-ssl -Start a training session ------------------------- +6. Start a training session +--------------------------- -In Studio click on the "Sessions" link, then the "New session" button in the upper right corner. Click the "Start session" tab and enter your desirable settings (or use default) and hit the "Start run" button. In the terminal where your are running your client you should now see some activity. When the round is completed, you can see the results in the FEDn Studio UI on the "Models" page. +In Studio click on the "Sessions" link, then the "New session" button in the upper right corner. Click the "Start session" tab and enter your desirable settings (the default settings are good for this example) and hit the "Start run" button. +In the terminal where your are running your client you should now see some activity. When a round is completed, you can see the results on the "Models" page. **Watch the training progress** -Once a training session is started, you can monitor the progress of the training by navigating to "Sessions" and click on the "Open" button of the active session. The session page will list the models as soon as they are generated. To get more information about a particular model, navigate to the model page by clicking the model name. From the model page you can download the model weights and get validation metrics. - -To get an overview of how the models have evolved over time, navigate to the "Models" tab in the sidebar. Here you can see a list of all models generated across sessions along with a graph showing some metrics of how the models are performing. +Once a training session is started, you can monitor the progress of the training by navigating to "Sessions" and click on the "Open" button of the active session. The session page will list the models as soon as they are generated. +To get more information about a particular model, navigate to the model page by clicking the model name. From the model page you can download the model weights and get validation metrics. .. image:: img/studio_model_overview.png .. _studio-api: +Congratulations, you have now completed your first federated training session with FEDn! Below you find additional information that can +be useful as you progress in your federated learning journey. + **Control training sessions using the Python APIClient** You can also issue training sessions using the APIClient: @@ -182,8 +192,7 @@ You can also issue training sessions using the APIClient: Please see :py:mod:`fedn.network.api` for more details on how to use the APIClient. -Access model updates --------------------- +**Downloading global model updates** .. note:: In FEDn Studio, you can access global model updates by going to the 'Models' or 'Sessions' tab. Here you can download model updates, metrics (as csv) and view the model trail. @@ -197,8 +206,7 @@ You can also access global model updates via the APIClient: >>> client.download_model("", path="model.npz") -Connecting clients using Docker --------------------------------- +**Connecting clients using Docker** You can also use Docker to containerize the client. For convenience, there is a Docker image hosted on ghrc.io with fedn preinstalled. @@ -214,8 +222,7 @@ To start a client using Docker: ghcr.io/scaleoutsystems/fedn/fedn:0.10.0 run client -in client.yaml --force-ssl --secure=True -Where to go from here? ------------------------- +**Where to go from here?** With you first FEDn federated project set up, we suggest that you take a close look at how a FEDn project is structured and how you develop your own FEDn projects: From 4194bd4458418a27cab559ae26e8bb8808894724 Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Sat, 6 Jul 2024 08:59:34 +0200 Subject: [PATCH 06/14] Remove Studio from the index (keep the file), most info is now in quickstart --- docs/index.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index 7cda27188..79b862c0e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -10,7 +10,6 @@ :maxdepth: 1 :caption: Documentation - studio apiclient architecture aggregators From b3f6344f06ffbe1843e77b8c12f7f66a17074757 Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Sat, 6 Jul 2024 09:51:32 +0200 Subject: [PATCH 07/14] Make the APIClient page Studio-centric --- docs/apiclient.rst | 21 ++++++++++----------- docs/quickstart.rst | 5 ++--- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/apiclient.rst b/docs/apiclient.rst index b4dfd789f..87707dfd2 100644 --- a/docs/apiclient.rst +++ b/docs/apiclient.rst @@ -1,11 +1,7 @@ APIClient ========= -.. note:: - - For access to FEDn Studio API, please see :ref:`studio-api`. - -FEDn comes with an *APIClient* for interacting with the FEDn network. The APIClient is a Python3 library that can be used to interact with the FEDn network programmatically. +FEDn comes with an *APIClient* - a Python3 library that can be used to interact with FEDn programmatically. **Installation** @@ -17,12 +13,15 @@ The APIClient is available as a Python package on PyPI, and can be installed usi **Initialize the APIClient** -To initialize the APIClient, you need to provide the hostname and port of the FEDn API server. The default port is 8092. The following code snippet shows how to initialize the APIClient: +The FEDn REST API is available at /api/v1/. To access this API you need the url to the controller-host, as well as an admin API token. The controller host can be found in the project dashboard (top right corner). +To obtain an admin API token, navigate to the "Settings" tab in your Studio project and click on the "Generate token" button. Copy the 'access' token and use it to access the API using the instructions below. + .. code-block:: python - - from fedn import APIClient - client = APIClient("localhost", 8092) + + >>> from fedn import APIClient + >>> client = APIClient(host="", token="", secure=True, verify=True) + **Set active package and seed model** @@ -38,9 +37,9 @@ To set the initial seed model, you can use the following code snippet: client.set_active_model(path="path/to/seed.npz") -**Start training session** +**Start a training session** -Once the active package and seed model are set, you can connect clients to the network and start training models. The following code snippet initializes a session (training rounds): +Once the active package and seed model are set, you can connect clients to the network and start training models. The following code snippet starts a traing session: .. code-block:: python diff --git a/docs/quickstart.rst b/docs/quickstart.rst index a73434328..7ae3883d5 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -98,9 +98,7 @@ It is also possible to upload a package and seed model using the Python API Clie Do this by going to the 'Settings' tab in FEDn Studio and click 'Generate token'. Copy the access token and use it in the APIClient below. The controller host can be found on the main Dashboard in FEDn Studio. - You can also upload the file via the FEDn Studio UI. Please see :ref:`studio-upload-files` for more details. - -Upload the package and seed model using the APIClient: +To upload the package and seed model using the APIClient: .. code:: python @@ -223,6 +221,7 @@ To start a client using Docker: **Where to go from here?** +-------------------------- With you first FEDn federated project set up, we suggest that you take a close look at how a FEDn project is structured and how you develop your own FEDn projects: From 9861a812ab2fcc525f556403b6935630abac50aa Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Tue, 9 Jul 2024 11:54:15 +0200 Subject: [PATCH 08/14] Less detailed instructions for setting up the Studio project --- docs/quickstart.rst | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 7ae3883d5..b07fa682a 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -16,17 +16,13 @@ Getting started with FEDn Start by creating an account in Studio. Head over to `fedn.scaleoutsystems.com/signup `_ and sign up. -Logged into Studio, do: +Logged into Studio, create a new project by clicking on the "New Project" button in the top right corner of the screen. +You will see a Studio project similar to the image below. The Studio project provides all the necessary server side components of FEDn. +We will use this project in a later stage to run the federated experiments. But first, we will set up the local client. -1. Click on the "New Project" button in the top right corner of the screen. -2. Continue by clicking the "Create button". The FEDn template contains all the services necessary to start a federation. -3. Enter the project name (mandatory). The project description is optional. -4. Click the "Create" button to create the project. .. image:: img/studio_project_overview.png -When these steps are complete, you will see a Studio project similar to the above image. The Studio project provides all server side components of FEDn needed to manage -federated training. We will use this project in a later stage to run the federated experiments. But first, we will set up the local client. 2. Install FEDn on your client ------------------------------- From 005110f514b0a58ad6d3e8e509b3de95716649fd Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Tue, 9 Jul 2024 11:56:48 +0200 Subject: [PATCH 09/14] Add instructions to clone the repo. --- docs/quickstart.rst | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/quickstart.rst b/docs/quickstart.rst index b07fa682a..26334e84b 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -56,8 +56,14 @@ Next, we will prepare the client. For illustrative purposes, we use one of the p In order to train a federated model using FEDn, your Studio project needs to be initialized with a ``compute package`` and a ``seed model``. The compute package is a code bundle containing the code used by the client to execute local training and local validation. The seed model is a first version of the global model. For a detailed explaination of the compute package and seed model, see this guide: :ref:`projects-label` - -Locate into ``examples/mnist-pytorch`` folder in the cloned fedn repository. The compute package is located in the folder ``client``. + +Clone the FEDn repository: + +.. code-block:: bash + + git clone https://github.com/scaleoutsystems/fedn.git + +Locate into ``examples/mnist-pytorch`` folder. The compute package is located in the folder ``client``. Create a compute package: From 5557489f2fa9538966919a7a02480e262d271ed7 Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Tue, 9 Jul 2024 12:00:56 +0200 Subject: [PATCH 10/14] Clarify that we only use the example from the local clone of fedn --- docs/quickstart.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 26334e84b..9b3345be3 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -57,7 +57,8 @@ Next, we will prepare the client. For illustrative purposes, we use one of the p In order to train a federated model using FEDn, your Studio project needs to be initialized with a ``compute package`` and a ``seed model``. The compute package is a code bundle containing the code used by the client to execute local training and local validation. The seed model is a first version of the global model. For a detailed explaination of the compute package and seed model, see this guide: :ref:`projects-label` -Clone the FEDn repository: +To work through this quick start you need a local copy of the ``mnist-pytorch`` example project contained in the main FEDn Git repository. +The following command clones the entire repository but you will only use the example: .. code-block:: bash From f503b080b86c68c7fc7f4ba23ba695afb3c34577 Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Tue, 9 Jul 2024 21:12:54 +0200 Subject: [PATCH 11/14] Fix path --- docs/quickstart.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 9b3345be3..6e870176f 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -64,7 +64,7 @@ The following command clones the entire repository but you will only use the exa git clone https://github.com/scaleoutsystems/fedn.git -Locate into ``examples/mnist-pytorch`` folder. The compute package is located in the folder ``client``. +Locate into the ``fedn/examples/mnist-pytorch`` folder. The compute package is located in the folder ``client``. Create a compute package: From 9ddb96c6f8dcd2441a8079b7be3269f4614c2e7d Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Tue, 9 Jul 2024 21:17:48 +0200 Subject: [PATCH 12/14] Remove Docker client instructions from quickstart --- docs/quickstart.rst | 17 ----------------- 1 file changed, 17 deletions(-) diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 6e870176f..78f36b2ba 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -206,23 +206,6 @@ You can also access global model updates via the APIClient: >>> ... >>> client.download_model("", path="model.npz") - -**Connecting clients using Docker** - -You can also use Docker to containerize the client. -For convenience, there is a Docker image hosted on ghrc.io with fedn preinstalled. -To start a client using Docker: - -.. code-block:: - - docker run \ - -v $PWD/client.yaml:/app/client.yaml \ - -e FEDN_PACKAGE_EXTRACT_DIR=package \ - -e FEDN_NUM_DATA_SPLITS=2 \ - -e FEDN_DATA_PATH=/app/package/data/clients/1/mnist.pt \ - ghcr.io/scaleoutsystems/fedn/fedn:0.10.0 run client -in client.yaml --force-ssl --secure=True - - **Where to go from here?** -------------------------- From 282ec54435f07da67bb21d6e693fa4b56868b208 Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Wed, 10 Jul 2024 10:37:12 +0200 Subject: [PATCH 13/14] Fixed typos and some broken refs --- docs/apiclient.rst | 2 ++ docs/developer.rst | 1 + docs/quickstart.rst | 22 ++++++++++++++++++---- 3 files changed, 21 insertions(+), 4 deletions(-) diff --git a/docs/apiclient.rst b/docs/apiclient.rst index 87707dfd2..2806ebe86 100644 --- a/docs/apiclient.rst +++ b/docs/apiclient.rst @@ -1,3 +1,5 @@ +.. _apiclient-label: + APIClient ========= diff --git a/docs/developer.rst b/docs/developer.rst index d53df2d3c..3e05357b4 100644 --- a/docs/developer.rst +++ b/docs/developer.rst @@ -1,3 +1,4 @@ +.. _developer-label: Local development ================= diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 78f36b2ba..7723e7f28 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -49,13 +49,14 @@ It is recommended to use a virtual environment when installing FEDn. .. _package-creation: -Next, we will prepare the client. For illustrative purposes, we use one of the pre-defined projects in the FEDn repository, ``minst-pytorch``. +Next, we will prepare the client. We will use one of the pre-defined projects in the FEDn repository, ``mnist-pytorch``. 3. Create the compute package and seed model -------------------------------------------- In order to train a federated model using FEDn, your Studio project needs to be initialized with a ``compute package`` and a ``seed model``. The compute package is a code bundle containing the -code used by the client to execute local training and local validation. The seed model is a first version of the global model. For a detailed explaination of the compute package and seed model, see this guide: :ref:`projects-label` +code used by the client to execute local training and local validation. The seed model is a first version of the global model. +For a detailed explaination of the compute package and seed model, see this guide: :ref:`projects-label` To work through this quick start you need a local copy of the ``mnist-pytorch`` example project contained in the main FEDn Git repository. The following command clones the entire repository but you will only use the example: @@ -82,8 +83,12 @@ Next, create the seed model: This will create a file called ``seed.npz`` in the root of the project. -Next will now upload these files to your Studio project: +.. note:: + This example automatically creates the runtime environment for the compute package using Virtualenv. + When you first exectue the above commands, FEDn will build a venv, and this takes + a bit of time. For more information on the various options to manage the environement, see :ref:`projects-label`. +Next will now upload these files to your Studio project: 4. Initialize your FEDn Studio Project -------------------------------------- @@ -99,7 +104,7 @@ It is also possible to upload a package and seed model using the Python API Clie .. note:: You need to create an API admin token and use the token to authenticate the APIClient. Do this by going to the 'Settings' tab in FEDn Studio and click 'Generate token'. Copy the access token and use it in the APIClient below. - The controller host can be found on the main Dashboard in FEDn Studio. + The controller host can be found on the main Dashboard in FEDn Studio. More information on the use of the APIClient can be found here: :ref:`apiclient-label. To upload the package and seed model using the APIClient: @@ -215,6 +220,15 @@ and how you develop your own FEDn projects: - :ref:`projects-label` You can also dive into the architecture overview to learn more about how FEDn is designed and works under the hood: + - :ref:`architecture-label` +For developers looking to cutomize FEDn and develop own aggregators, check out the local development guide. +This page also has instructions for using Docker to run clients: + +- :ref:`developer-label` + + + + From 01d8378ec3ca6b55d2d9acb8c470c117870b767e Mon Sep 17 00:00:00 2001 From: Andreas Hellander Date: Wed, 10 Jul 2024 10:41:16 +0200 Subject: [PATCH 14/14] Updated example readme in the github repo --- examples/mnist-keras/README.rst | 5 ++--- examples/mnist-pytorch/README.rst | 7 +++---- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/examples/mnist-keras/README.rst b/examples/mnist-keras/README.rst index 22bf61c8d..aaf13c21d 100644 --- a/examples/mnist-keras/README.rst +++ b/examples/mnist-keras/README.rst @@ -1,7 +1,7 @@ FEDn Project: Keras/Tensorflow (MNIST) ------------------------------------------- -This is a TF/Keras version of the Quickstart Tutorial (PyTorch) FEDn Project. For a step-by-step guide, refer to that tutorial. +This is a TF/Keras version of the PyTorch Quickstart Tutorial. For a step-by-step guide, refer to that tutorial. **Note: We recommend all new users to start by following the Quickstart Tutorial: https://fedn.readthedocs.io/en/latest/quickstart.html** @@ -45,6 +45,5 @@ This step will take a few minutes, depending on hardware and internet connection Running the project on FEDn ---------------------------- -To set up your FEDn Studio project and connect clients, follow this guide: https://fedn.readthedocs.io/en/latest/studio.html. On the -step "Upload Files", upload 'package.tgz' and 'seed.npz' created above. +To learn how to set up your FEDn Studio project and connect clients, take the quickstart tutorial: https://fedn.readthedocs.io/en/stable/quickstart.html. diff --git a/examples/mnist-pytorch/README.rst b/examples/mnist-pytorch/README.rst index 0d8044327..990b902b2 100644 --- a/examples/mnist-pytorch/README.rst +++ b/examples/mnist-pytorch/README.rst @@ -2,9 +2,9 @@ FEDn Project: MNIST (PyTorch) ----------------------------- This is an example FEDn Project based on the classic hand-written text recognition dataset MNIST. -The example is intented as a minimalistic quickstart. +The example is intented as a minimalistic quickstart to learn how to use FEDn. - **Note: We recommend that all new users start by following the Quickstart Tutorial: https://fedn.readthedocs.io/en/stable/quickstart.html** + **Note: We recommend that all new users start by taking the Quickstart Tutorial: https://fedn.readthedocs.io/en/stable/quickstart.html** Prerequisites ------------- @@ -47,5 +47,4 @@ This will create a model file 'seed.npz' in the root of the project. This step w Running the project on FEDn ---------------------------- -To set up your FEDn Studio project and connect clients, follow this guide: https://fedn.readthedocs.io/en/latest/studio.html. On the -step "Upload Files", upload 'package.tgz' and 'seed.npz' created above. +To learn how to set up your FEDn Studio project and connect clients, take the quickstart tutorial: https://fedn.readthedocs.io/en/stable/quickstart.html.