Merge branch 'master' of https://github.com/scaleoutsystems/fedn into…

… feature/SK-944

merge master # the commit.

.github/workflows/branch-name-check.yaml

@@ -7,7 +7,7 @@ on:
       - master
 
 env:
-  BRANCH_REGEX: '^((feature|github|dependabot|hotfix|bugfix|fix|bug|docs|refactor)\/.+)|(release\/v((([0-9]+)\.([0-9]+)\.([0-9]+)(?:-([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?)(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?))$'
+  BRANCH_REGEX: '(?i)^((feature|github|dependabot|hotfix|bugfix|fix|bug|docs|refactor)\/.+)|(release\/v((([0-9]+)\.([0-9]+)\.([0-9]+)(?:-([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?)(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?))$'
 
 jobs:
   branch-name-check:

README.rst

@@ -9,50 +9,50 @@
 .. |pic3| image:: https://readthedocs.org/projects/fedn/badge/?version=latest&style=flat
    :target: https://fedn.readthedocs.io
 
-FEDn
---------
+FEDn: An enterprise-ready federated learning framework 
+-------------------------------------------------------
 
-FEDn empowers its users to create federated learning applications that seamlessly transition from local proofs-of-concept to secure distributed deployments. 
+Our goal is to provide a federated learning framework that is both secure, scalable and easy-to-use. We believe that that minimal code change should be needed to progress from early proof-of-concepts to production. This is reflected in our core design: 
 
-Leverage a flexible pseudo-local sandbox to rapidly transition your existing ML project to a federated setting. Test and scale in real-world scenarios using FEDn Studio - a fully managed, secure deployment of all server-side components (SaaS). 
+-  **Minimal server-side complexity for the end-user**. Running a proper distributed FL deployment is hard. With FEDn Studio we seek to handle all server-side complexity and provide a UI, REST API and a Python interface to help users manage FL experiments and track metrics in real time.
 
-We develop the FEDn framework following these core design principles:
+-  **Secure by design.** FL clients do not need to open any ingress ports. Industry-standard communication protocols (gRPC) and token-based authentication and RBAC (Jason Web Tokens) provides flexible integration in a range of production environments.  
 
--  **Seamless transition from proof-of-concepts to real-world FL**. FEDn has been designed to make the journey from R&D to real-world deployments as smooth as possibe. Develop your federated learning use case in a pseudo-local environment, then deploy it to FEDn Studio (cloud or on-premise) for real-world scenarios. No code change is required to go from development and testing to production. 
+-  **ML-framework agnostic**. A black-box client-side architecture lets data scientists interface with their framework of choice. 
 
--  **Designed for scalability and resilience.** FEDn enables model aggregation through multiple aggregation servers sharing the workload. A hierarchical architecture makes the framework well suited borh for cross-silo and cross-device use-cases. FEDn seamlessly recover from failures in all critical components, and manages intermittent client-connections, ensuring robust deployment in production environments.
+-  **Cloud native.** By following cloud native design principles, we ensure a wide range of deployment options including private cloud and on-premise infrastructure. 
 
--  **Secure by design.** FL clients do not need to open any ingress ports, facilitating distributed deployments across a wide variety of settings. Additionally, FEDn utilizes secure, industry-standard communication protocols and supports token-based authentication and RBAC for FL clients (JWT), providing flexible integration in production environments.   
+-  **Scalability and resilience.** Multiple aggregation servers (combiners) can share the workload. FEDn seamlessly recover from failures in all critical components and manages intermittent client-connections. 
 
--  **Developer and data scientist friendly.** Extensive event logging and distributed tracing enables developers to monitor experiments in real-time, simplifying troubleshooting and auditing. Machine learning metrics can be accessed via both a Python API and visualized in an intuitive UI that helps the data scientists analyze and communicate ML-model training progress.
+-  **Developer and DevOps friendly.** Extensive event logging and distributed tracing enables developers to monitor the sytem in real-time, simplifying troubleshooting and auditing. Extensions and integrations are facilitated by a flexible plug-in architecture.  
 
+We provide a fully managed deployment for testing, academic, and personal use. Sign up for a `FEDn Studio account <https://fedn.scaleoutsystems.com/signup>`__  and take the `Quickstart tutorial <https://fedn.readthedocs.io/en/stable/quickstart.html>`__ to get started with FEDn. 
 
 Features
 =========
 
-Core FL framework (this repository): 
+Federated learning: 
 
 - Tiered federated learning architecture enabling massive scalability and resilience. 
 - Support for any ML framework (examples for PyTorch, Tensforflow/Keras and Scikit-learn)
 - Extendable via a plug-in architecture (aggregators, load balancers, object storage backends, databases  etc.)
 - Built-in federated algorithms (FedAvg, FedAdam, FedYogi, FedAdaGrad, etc.)
-- CLI and Python API.
+- UI, CLI and Python API.
 - Implement clients in any language (Python, C++, Kotlin etc.)
 - No open ports needed client-side.
-- Flexible deployment of server-side components using Docker / docker compose.   
 
 
-FEDn Studio - From development to FL in production: 
+From development to FL in production: 
 
 -  Secure deployment of server-side / control-plane on Kubernetes.
--  UI with dashboards for orchestrating experiments and visualizing results
+-  UI with dashboards for orchestrating FL experiments and for visualizing results
 -  Team features - collaborate with other users in shared project workspaces. 
 -  Features for the trusted-third party: Manage access to the FL network, FL clients and training progress.
 -  REST API for handling experiments/jobs. 
 -  View and export logging and tracing information. 
 -  Public cloud, dedicated cloud and on-premise deployment options.
 
-Available clients:
+Available client APIs:
 
 - Python client (this repository)
 - C++ client (`FEDn C++ client <https://github.com/scaleoutsystems/fedn-cpp-client>`__)
@@ -64,11 +64,11 @@ Getting started
 
 Get started with FEDn in two steps:  
 
-1. Sign up for a `Free FEDn Studio account <https://fedn.scaleoutsystems.com/signup>`__
+1. Register for a `FEDn Studio account <https://fedn.scaleoutsystems.com/signup>`__
 2. Take the `Quickstart tutorial <https://fedn.readthedocs.io/en/stable/quickstart.html>`__
 
-FEDn Studio (SaaS) is free for academic use and personal development / small-scale testing and exploration. For users and teams requiring
-additional project resources, dedicated support or other hosting options, `explore our plans <https://www.scaleoutsystems.com/start#pricing>`__.  
+Use of our multi-tenant, managed deployment of FEDn Studio (SaaS) is free forever for academic research and personal development/testing purposes.
+For users and teams requiring additional resources, more storage and cpu, dedicated support, and other hosting options (private cloud, on-premise), `explore our plans <https://www.scaleoutsystems.com/start#pricing>`__.  
 
 Documentation
 =============
@@ -77,6 +77,18 @@ More details about the architecture, deployment, and how to develop your own app
 
 -  `Documentation <https://fedn.readthedocs.io>`__
 
+FEDn Project Examples
+=====================
+
+Our example projects demonstrate different use case scenarios of FEDn 
+and its integration with popular machine learning frameworks like PyTorch and TensorFlow.
+
+- `FEDn + PyTorch <https://github.com/scaleoutsystems/fedn/tree/master/examples/mnist-pytorch>`__
+- `FEDn + Tensforflow/Keras <https://github.com/scaleoutsystems/fedn/tree/master/examples/mnist-keras>`__
+- `FEDn + MONAI <https://github.com/scaleoutsystems/fedn/tree/master/examples/monai-2D-mednist>`__
+- `FEDn + Hugging Face <https://github.com/scaleoutsystems/fedn/tree/master/examples/huggingface>`__
+- `FEDn + Flower <https://github.com/scaleoutsystems/fedn/tree/master/examples/flower-client>`__
+- `FEDN + Self-supervised learning <https://github.com/scaleoutsystems/fedn/tree/master/examples/FedSimSiam>`__
 
 FEDn Studio Deployment options
 ==============================

docs/apiclient.rst

@@ -24,6 +24,18 @@ To obtain an admin API token, navigate to the "Settings" tab in your Studio proj
    >>> from fedn import APIClient
    >>> client = APIClient(host="<controller-host>", token="<access-token>", secure=True, verify=True)
 
+Alternatively, the access token can be sourced from an environment variable. 
+
+.. code-block:: bash
+   $ export FEDN_AUTH_TOKEN=<access-token>
+
+Then passing a token as an argument is not required. 
+
+.. code-block:: python
+
+   >>> from fedn import APIClient
+   >>> client = APIClient(host="<controller-host>", secure=True, verify=True)
+
 
 **Set active package and seed model**
 
@@ -78,4 +90,4 @@ And the following code snippet shows how to get a specific session:
    session = client.get_session(id="session_name")
 
 
-For more information on how to use the APIClient, see the :py:mod:`fedn.network.api.client`, and the example `Notebooks <https://github.com/scaleoutsystems/fedn/blob/master/examples/mnist-pytorch/API_Example.ipynb>`_. 
+For more information on how to use the APIClient, see the :py:mod:`fedn.network.api.client`, and the example `Notebooks <https://github.com/scaleoutsystems/fedn/tree/master/examples/notebooks>`_. 

docs/conf.py

@@ -12,7 +12,7 @@
 author = "Scaleout Systems AB"
 
 # The full version, including alpha/beta/rc tags
-release = "0.11.1"
+release = "0.12.0"
 
 # Add any Sphinx extension module names here, as strings
 extensions = [

docs/developer.rst

@@ -1,14 +1,14 @@
 .. _developer-label:
 
-Local development
-=================
+Local development and deployment
+================================
 
 .. note::
    These instructions are for users wanting to set up a local development deployment of FEDn (i.e. without FEDn Studio).
    This requires practical knowledge of Docker and docker-compose. 
 
 Running the FEDn development sandbox (docker-compose)
-=====================================================
+------------------------------------------------------
 
 During development on FEDn, and when working on own aggregators/helpers, it is 
 useful to have a local development setup of the core FEDn services (controller, combiner, database, object store). 
@@ -45,8 +45,7 @@ To connect a native FEDn client, you need to make sure that the combiner service
 One way to achieve this is to edit your '/etc/hosts' and add a line '127.0.0.1  	combiner'. 
 
 Access message logs and validation data from MongoDB  
-====================================================
-
+------------------------------------------------------
 You can access and download event logs and validation data via the API, and you can also as a developer obtain 
 the MongoDB backend data using pymongo or via the MongoExpress interface: 
 
@@ -55,7 +54,7 @@ the MongoDB backend data using pymongo or via the MongoExpress interface:
 Username and password are found in 'docker-compose.yaml'. 
 
 Access global models   
-====================
+------------------------------------------------------
 
 You can obtain global model updates from the 'fedn-models' bucket in Minio: 
 
@@ -64,13 +63,13 @@ You can obtain global model updates from the 'fedn-models' bucket in Minio:
 Username and password are found in 'docker-compose.yaml'. 
 
 Reset the FEDn deployment   
-=========================
+------------------------------------------------------
 
 To purge all data from a deployment incuding all session and round data, access the MongoExpress UI interface and 
 delete the entire ``fedn-network`` collection. Then restart all services. 
 
 Clean up
-========
+------------------------------------------------------
 You can clean up by running 
 
 .. code-block::
@@ -79,7 +78,7 @@ You can clean up by running
 
 
 Connecting clients using Docker:
-================================
+------------------------------------------------------
 
 For convenience, we distribute a Docker image hosted on ghrc.io with FEDn preinstalled. For example, to start a client for the MNIST PyTorch example using Docker
 and FEDN 0.10.0, run this from the example folder:   
@@ -95,7 +94,7 @@ and FEDN 0.10.0, run this from the example folder:
 
 
 Self-managed distributed deployment
-===================================
+------------------------------------------------------
 
 You can use different hosts for the various FEDn services. These instructions shows how to set up FEDn on a **local network** using a single workstation or laptop as 
 the host for the servier-side components, and other hosts or devices as clients. 
@@ -160,9 +159,3 @@ Alternatively updating the `/etc/hosts` file, appending the following lines for
 
    <host local ip>      api-server
    <host local ip>      combiner
-
-
-Start a training session
-------------------------
-
-After connecting with your clients, you are ready to start training sessions from the host machine.