tutorials/Introduction to Federated Learning with CIFAR10 and MLX.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Federated Learning with CIFAR10 and MLX\n",
    "\n",
    "This tutorial demonstrates how to train a DNN with `pfl` using Private Federated Learning.\n",
    "\n",
    "We will learn how to train a simple neural network in simulation to classify images from the CIFAR10 dataset.\n",
    "This dataset contains images of 10 different types of objects.\n",
    "Each image is labelled with one object. The dataset is not inherently split into real users, so we have to create artificial user partitions.\n",
    "\n",
    "This tutorial is using the array framework [MLX](https://ml-explore.github.io/mlx/build/html/index.html) to train models efficiently on Apple silicon. If you don't have an Apple silicon Mac, checkout [the equivalent tutorial in TensorFlow](https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction%20to%20Federated%20Learning%20with%20CIFAR10.ipynb) or go straight to the more [advanced tutorial in PyTorch](https://github.com/apple/pfl-research/tree/main/tutorials/Introduction%20to%20PFL%20research%20with%20FLAIR.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Install environment & `pfl`\n",
    "To use this notebook, follow these prerequisite steps:\n",
    "\n",
    "1. Follow installation instructions at https://github.com/apple/pfl-research/blob/main/benchmarks/README.md#setup-environment. The extras flag `-E mlx` can be used to install `pfl` with supported MLX version.\n",
    "2. Add the kernel created by Poetry to the notebook:\n",
    "```\n",
    "pip install jupyter\n",
    "python -m ipykernel install --name \"notebook-tutorial\"\n",
    "```\n",
    "4. Start the notebook:\n",
    "```\n",
    "python -m jupyter notebook \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "# Additional packages needed to run notebook\n",
    "!{sys.executable} -m pip install matplotlib nest_asyncio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "If you installed the packages above, you need to restart the kernel of the notebook by clicking on \"Kernel -> Restart\"\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preparing data for simulations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the script `pfl-research/dataset/cifar10/download_preprocess.py` to download the CIFAR10 dataset from its original source and preprocess into pickles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!(cd ../benchmarks; {sys.executable} -m dataset.cifar10.download_preprocess --output_dir ./data/cifar10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import sys\n",
    "import mlx\n",
    "import mlx.core as mx\n",
    "import mlx.nn as nn\n",
    "\n",
    "# Both Jupyter and `pfl` uses async, nest_asyncio allows `pfl` to run inside the notebook \n",
    "import nest_asyncio\n",
    "nest_asyncio.apply()\n",
    "\n",
    "# append the root directory to your paths to be able to reach the examples.  \n",
    "sys.path.append('../benchmarks')\n",
    "mx.random.seed(7)\n",
    "np.random.seed(7)\n",
    "\n",
    "# Always import the `pfl` model first to let `pfl` know which Deep Learning framework you will use.\n",
    "from pfl.model.mlx import MLXModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data shapes: features: (10000, 32, 32, 3) targets: (10000, 10)\n",
      "Dev data shapes:   features: (2000, 32, 32, 3) targets: (2000, 10)\n"
     ]
    }
   ],
   "source": [
    "from dataset.cifar10 import load_and_preprocess\n",
    "# Train only on classes 0 and 2 in this example\n",
    "exclude_classes = [1,3,4,5,6,7,8,9]\n",
    "train_images, train_labels, channel_means, channel_stddevs = load_and_preprocess('../benchmarks/data/cifar10/cifar10_train.p', exclude_classes=exclude_classes)\n",
    "val_images, val_labels, _, _ = load_and_preprocess('../benchmarks/data/cifar10/cifar10_test.p', channel_means, channel_stddevs, exclude_classes=exclude_classes)\n",
    "# Convert labels to one-hot vectors.\n",
    "one_hots = np.eye(10)\n",
    "train_labels = one_hots[train_labels].squeeze()\n",
    "val_labels = one_hots[val_labels].squeeze()\n",
    "\n",
    "print('Train data shapes:', 'features:', train_images.shape, 'targets:', train_labels.shape)\n",
    "print('Dev data shapes:  ', 'features:', val_images.shape, 'targets:', val_labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As displayed above, the images are of dimensions `32x32` with `3` color channels and the labels are one-hot vectors `[0-9]`. Also, you can see that we exclude all but 2 classes, Airplane (0) and Bird (2), to make things simple for this tutorial.\n",
    "\n",
    "As the data is not inherently split into real users, we need to use [ArtificialFederatedDataset](https://apple.github.io/pfl-research/reference/data.html#pfl.data.federated_dataset.ArtificialFederatedDataset) to sample datapoints to create artificial user datasets for federated learning, which in turn requires a function for sampling the length of a user dataset and the actual datapoints of this dataset.\n",
    "\n",
    "Firstly, we define a function which samples the length of a dataset from a Poisson distribution with mean 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjkAAAHHCAYAAABdm0mZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABMI0lEQVR4nO3deVhUZf8/8PeAzIDADKICIgi4pJKIiopo7uSoaJn2uGQG5lIKJlIulAHqU26ZWhlWlpRlaX5dShJFEH1SXELJJSH1wS1ZFIUBLBDm/v3hb87jyK4gcny/rmuunHM+5z73uecM8+7MOWcUQggBIiIiIpkxqesOEBEREdUGhhwiIiKSJYYcIiIikiWGHCIiIpIlhhwiIiKSJYYcIiIikiWGHCIiIpIlhhwiIiKSJYYcIiIikiWGHJKtiIgIKBSKR7Kufv36oV+/ftLzhIQEKBQKbNmy5ZGsPyAgAK6uro9kXQ8qPz8fkydPhoODAxQKBYKDg+u6S7UiKioKCoUCFy9erOuuPFKGfT4hIaGuu1IphUKBoKCguu4GPQIMOVQvGD44DA9zc3M4OjpCq9Xio48+Ql5eXo2s59q1a4iIiEBycnKNtFeTHue+VcX777+PqKgoTJs2DRs2bMCECRPqukuPnY0bN2LVqlV13Q0AwO3btxEREVEvQktZDh06hIiICOTk5NR1V6gOMeRQvbJw4UJs2LABkZGRmDFjBgAgODgYHh4eOHnypFHt/Pnz8ffff1er/WvXrmHBggXVDhJ79uzBnj17qrVMdVXUty+++AKpqam1uv6HFR8fjx49eiA8PBwvv/wyvLy86rpLj53HLeQsWLCgXoecBQsWMOQ84RrUdQeIqmPIkCHo2rWr9Dw0NBTx8fEYNmwYnnvuOZw9exYWFhYAgAYNGqBBg9rdxW/fvo2GDRtCqVTW6noqY2ZmVqfrr4qsrCy4u7vXdTeqpaCgAJaWlnXdDSJ6QDySQ/XegAED8O677+LSpUv49ttvpellnZMTGxuLZ555BjY2NrCyskLbtm3x9ttvA7h7TkG3bt0AABMnTpS+GouKigJw97ybDh06ICkpCX369EHDhg2lZe8/J8egpKQEb7/9NhwcHGBpaYnnnnsOV65cMapxdXVFQEBAqWXvbbOyvpV1Tk5BQQHefPNNODs7Q6VSoW3btvjggw8ghDCqM5yfsH37dnTo0AEqlQpPP/00YmJiyh7w+2RlZWHSpEmwt7eHubk5PD098fXXX0vzDedqpKWlITo6Wup7eeesXLx40Wjb7u9rRESE9DwvLw/BwcFwdXWFSqWCnZ0dnn32WRw/ftxouSNHjmDw4MHQaDRo2LAh+vbti4MHDxrVGPaXP/74Ay+99BIaNWqEZ555psJtP3PmDAYMGAALCws4OTnh3//+N/R6fam6HTt2wM/PD46OjlCpVGjVqhUWLVqEkpISqaZfv36Ijo7GpUuXpDEyvKZFRUUICwuDl5cXNBoNLC0t0bt3b+zbt6/Uun744Qd4eXnB2toaarUaHh4eWL16tVFNTk4OgoODpX2jdevWWLp0qdT3ixcvomnTpgCABQsWSP25d+yrqjpjf/78eQQEBMDGxgYajQYTJ07E7du3jWr//vtvvPHGG2jSpAmsra3x3HPP4a+//jLqX0REBGbPng0AcHNzK3efe9B9nuoPHskhWZgwYQLefvtt7NmzB1OmTCmz5syZMxg2bBg6duyIhQsXQqVS4fz589If3Pbt22PhwoUICwvD1KlT0bt3bwBAz549pTays7MxZMgQjB07Fi+//DLs7e0r7Nd7770HhUKBuXPnIisrC6tWrYKvry+Sk5OlI05VUZW+3UsIgeeeew779u3DpEmT0KlTJ+zevRuzZ8/GX3/9hZUrVxrV//rrr9i6dSumT58Oa2trfPTRRxg1ahQuX76Mxo0bl9uvv//+G/369cP58+cRFBQENzc3/PjjjwgICEBOTg5mzpyJ9u3bY8OGDZg1axacnJzw5ptvAoD0IfowXn/9dWzZsgVBQUFwd3dHdnY2fv31V5w9exZdunQBcPdrsiFDhsDLywvh4eEwMTHB+vXrMWDAAPznP/9B9+7djdr817/+hTZt2uD9998vFQjvlZGRgf79+6O4uBjz5s2DpaUlPv/88zJf16ioKFhZWSEkJARWVlaIj49HWFgYdDodli9fDgB45513kJubi6tXr0qvj5WVFQBAp9Nh3bp1GDduHKZMmYK8vDx8+eWX0Gq1OHr0KDp16gTgbogfN24cBg4ciKVLlwIAzp49i4MHD2LmzJkA7h597Nu3L/766y+89tpraNGiBQ4dOoTQ0FCkp6dj1apVaNq0KSIjIzFt2jS88MILGDlyJACgY8eO1Xp9qjv2o0ePhpubGxYvXozjx49j3bp1sLOzk7YFuBvoN2/ejAkTJqBHjx7Yv38//Pz8jNoZOXIk/vzzT3z//fdYuXIlmjRpAsB4n3vQfZ7qGUFUD6xfv14AEMeOHSu3RqPRiM6dO0vPw8PDxb27+MqVKwUAcf369XLbOHbsmAAg1q9fX2pe3759BQCxdu3aMuf17dtXer5v3z4BQDRv3lzodDpp+ubNmwUAsXr1ammai4uL8Pf3r7TNivrm7+8vXFxcpOfbt28XAMS///1vo7oXX3xRKBQKcf78eWkaAKFUKo2m/f777wKA+Pjjj0ut616rVq0SAMS3334rTSsqKhI+Pj7CysrKaNtdXFyEn59fhe0JIURaWlq52wlAhIeHS881Go0IDAwsty29Xi/atGkjtFqt0Ov10vTbt28LNzc38eyzz0rTDPvLuHHjKu2jEEIEBwcLAOLIkSPStKysLKHRaAQAkZaWZrS++7322muiYcOG4p9//pGm+fn5Gb2OBsXFxaKwsNBo2q1bt4S9vb149dVXpWkzZ84UarVaFBcXl9vvRYsWCUtLS/Hnn38aTZ83b54wNTUVly9fFkIIcf369VLjXRHDPr9v3z4hxION/b3bIoQQL7zwgmjcuLH0PCkpSQAQwcHBRnUBAQGl+rp8+fJSr4PBw+zzVL/w6yqSDSsrqwqvsrKxsQFw96uDsr5SqAqVSoWJEydWuf6VV16BtbW19PzFF19Es2bN8MsvvzzQ+qvql19+gampKd544w2j6W+++SaEENi1a5fRdF9fX7Rq1Up63rFjR6jVavz3v/+tdD0ODg4YN26cNM3MzAxvvPEG8vPzsX///hrYmvLZ2NjgyJEjuHbtWpnzk5OTce7cObz00kvIzs7GjRs3cOPGDRQUFGDgwIE4cOBAqX3h9ddfr9K6f/nlF/To0cPoaETTpk0xfvz4UrX3Ht3Jy8vDjRs30Lt3b9y+fRspKSmVrsvU1FQ670uv1+PmzZsoLi5G165djb6as7GxQUFBAWJjY8tt68cff0Tv3r3RqFEjaTxu3LgBX19flJSU4MCBA1Xa/srUxNj37t0b2dnZ0Ol0ACB9nTR9+nSjOsNFCNXxoPs81S/8uopkIz8/H3Z2duXOHzNmDNatW4fJkydj3rx5GDhwIEaOHIkXX3wRJiZVy/vNmzev1knGbdq0MXquUCjQunXrWr+HyqVLl+Do6GgUsIC7X3sZ5t+rRYsWpdpo1KgRbt26Vel62rRpU2r8yltPTVu2bBn8/f3h7OwMLy8vDB06FK+88gpatmwJADh37hwAwN/fv9w2cnNz0ahRI+m5m5tbldZ96dIleHt7l5retm3bUtPOnDmD+fPnIz4+XvrAvnf9VfH1119jxYoVSElJwZ07d8rs7/Tp07F582YMGTIEzZs3x6BBgzB69GgMHjxYqjl37hxOnjxZ7teFWVlZVepPZR5k7O/fDw3zbt26BbVajUuXLsHExKTUa9S6detq9+9B93mqXxhySBauXr2K3NzcCv/YWVhY4MCBA9i3bx+io6MRExODTZs2YcCAAdizZw9MTU0rXU91zqOpqvJuWFhSUlKlPtWE8tYjKjgnpbZUNB73Gz16NHr37o1t27Zhz549WL58OZYuXYqtW7diyJAh0pGC5cuXS+et3M9w3otBTb/GOTk56Nu3L9RqNRYuXIhWrVrB3Nwcx48fx9y5c6t0VPHbb79FQEAARowYgdmzZ8POzg6mpqZYvHgxLly4INXZ2dkhOTkZu3fvxq5du7Br1y6sX78er7zyinQyuF6vx7PPPos5c+aUua6nnnqqRrb7Qcb+Ue6Hj9M+T7WHIYdkYcOGDQAArVZbYZ2JiQkGDhyIgQMH4sMPP8T777+Pd955B/v27YOvr2+N3yHZ8H+zBkIInD9/3ugEzkaNGpV5L49Lly5JRySA8j/8y+Li4oK9e/ciLy/P6GiO4asRFxeXKrdV2XpOnjwJvV5vdDTnYdZj+L/3+8ekvKNCzZo1w/Tp0zF9+nRkZWWhS5cueO+99zBkyBDp6wi1Wg1fX99q96UiLi4upV5fAKXuV5SQkIDs7Gxs3boVffr0kaanpaWVWra813jLli1o2bIltm7dalQTHh5eqlapVGL48OEYPnw49Ho9pk+fjs8++wzvvvsuWrdujVatWiE/P7/S8XjY90JtjL2Liwv0ej3S0tKMjpKeP3++VO2juts5Pd54Tg7Ve/Hx8Vi0aBHc3NzKPB/C4ObNm6WmGf4Ps7CwEACke6LU1A3EvvnmG6PzhLZs2YL09HQMGTJEmtaqVSscPnwYRUVF0rSdO3eWutS8On0bOnQoSkpK8MknnxhNX7lyJRQKhdH6H8bQoUORkZGBTZs2SdOKi4vx8ccfw8rKCn379q12m2q1Gk2aNCl1bsinn35q9LykpKTUVz12dnZwdHSUXk8vLy+0atUKH3zwAfLz80ut6/r169Xun8HQoUNx+PBhHD161Ki97777zqjOcMTg3iMERUVFpbYHuPsal/X1VVltHDlyBImJiUZ12dnZRs9NTEykQG0Yk9GjRyMxMRG7d+8utZ6cnBwUFxcDABo2bChNexC1MfaG/4m5f+w+/vjjUrU1/V6m+olHcqhe2bVrF1JSUlBcXIzMzEzEx8cjNjYWLi4u+Omnn2Bubl7usgsXLsSBAwfg5+cHFxcXZGVl4dNPP4WTk5N0P5RWrVrBxsYGa9euhbW1NSwtLeHt7V3l8zTuZ2tri2eeeQYTJ05EZmYmVq1ahdatWxtd5j558mRs2bIFgwcPxujRo3HhwgV8++23RidFVrdvw4cPR//+/fHOO+/g4sWL8PT0xJ49e7Bjxw4EBweXavtBTZ06FZ999hkCAgKQlJQEV1dXbNmyBQcPHsSqVatKnRNUVZMnT8aSJUswefJkdO3aFQcOHMCff/5pVJOXlwcnJye8+OKL8PT0hJWVFfbu3Ytjx45hxYoVAO5+yK9btw5DhgzB008/jYkTJ6J58+b466+/sG/fPqjVavz8888P1Mc5c+Zgw4YNGDx4MGbOnCldQm44umXQs2dPNGrUCP7+/njjjTegUCiwYcOGMr8W8fLywqZNmxASEoJu3brBysoKw4cPx7Bhw7B161a88MIL8PPzQ1paGtauXQt3d3ejADF58mTcvHkTAwYMgJOTEy5duoSPP/4YnTp1ks6Tmj17Nn766ScMGzYMAQEB8PLyQkFBAU6dOoUtW7bg4sWLaNKkCSwsLODu7o5Nmzbhqaeegq2tLTp06IAOHTpUaXxqY+y9vLwwatQorFq1CtnZ2dIl5IZ9496jN4Y7ar/zzjsYO3YszMzMMHz4cN7c8UlTdxd2EVWd4RJyw0OpVAoHBwfx7LPPitWrVxtdqmxw/yXkcXFx4vnnnxeOjo5CqVQKR0dHMW7cuFKX0u7YsUO4u7uLBg0aGF3K3LdvX/H000+X2b/yLiH//vvvRWhoqLCzsxMWFhbCz89PXLp0qdTyK1asEM2bNxcqlUr06tVL/Pbbb6XarKhv919CLoQQeXl5YtasWcLR0VGYmZmJNm3aiOXLlxtdzivE3ctpy7oMu7xL2++XmZkpJk6cKJo0aSKUSqXw8PAo8/Lvql5CLsTdy4wnTZokNBqNsLa2FqNHjxZZWVlGlwkXFhaK2bNnC09PT2FtbS0sLS2Fp6en+PTTT0u1d+LECTFy5EjRuHFjoVKphIuLixg9erSIi4uTagz7S0W3GLjfyZMnRd++fYW5ublo3ry5WLRokfjyyy9LXbp88OBB0aNHD2FhYSEcHR3FnDlzxO7du40uuRZCiPz8fPHSSy8JGxsbAUB6TfV6vXj//feFi4uLUKlUonPnzmLnzp2lXvctW7aIQYMGCTs7O6FUKkWLFi3Ea6+9JtLT0436nZeXJ0JDQ0Xr1q2FUqkUTZo0ET179hQffPCBKCoqkuoOHTokvLy8hFKprPRy8vsvIa+JsTe87+8dy4KCAhEYGChsbW2FlZWVGDFihEhNTRUAxJIlS4yWX7RokWjevLkwMTExaudh93mqPxRC8CwrIiKqv5KTk9G5c2d8++23FX5lTU8enpNDRET1Rlk/urtq1SqYmJgYndhNBPCcHCIiqkeWLVuGpKQk9O/fHw0aNJAulZ86dSqcnZ3runv0mOHXVUREVG/ExsZiwYIF+OOPP5Cfn48WLVpgwoQJeOedd9CgAf+/nYwx5BAREZEs8ZwcIiIikiWGHCIiIpKlJ/oLTL1ej2vXrsHa2pq3ACciIqonhBDIy8uDo6NjhT+w/ESHnGvXrvFsfCIionrqypUrcHJyKnf+Ex1yDLecv3LlCtRqdR33hoiIiKpCp9PB2dm50p+OeaJDjuErKrVazZBDRERUz1R2qglPPCYiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIllqUNcdIHoYrvOi67oL1XZxiV9dd4GI6InAIzlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLvIScJPXxcmwiIqLy8EgOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREclStUJOZGQkOnbsCLVaDbVaDR8fH+zatUua/88//yAwMBCNGzeGlZUVRo0ahczMTKM2Ll++DD8/PzRs2BB2dnaYPXs2iouLjWoSEhLQpUsXqFQqtG7dGlFRUaX6smbNGri6usLc3Bze3t44evRodTaFiIiIZK5aIcfJyQlLlixBUlISfvvtNwwYMADPP/88zpw5AwCYNWsWfv75Z/z444/Yv38/rl27hpEjR0rLl5SUwM/PD0VFRTh06BC+/vprREVFISwsTKpJS0uDn58f+vfvj+TkZAQHB2Py5MnYvXu3VLNp0yaEhIQgPDwcx48fh6enJ7RaLbKysh52PIiIiEgmFEII8TAN2NraYvny5XjxxRfRtGlTbNy4ES+++CIAICUlBe3bt0diYiJ69OiBXbt2YdiwYbh27Rrs7e0BAGvXrsXcuXNx/fp1KJVKzJ07F9HR0Th9+rS0jrFjxyInJwcxMTEAAG9vb3Tr1g2ffPIJAECv18PZ2RkzZszAvHnzqtx3nU4HjUaD3NxcqNXqhxkGWXCdF13XXXgiXFziV9ddICKq16r6+f3A5+SUlJTghx9+QEFBAXx8fJCUlIQ7d+7A19dXqmnXrh1atGiBxMREAEBiYiI8PDykgAMAWq0WOp1OOhqUmJho1IahxtBGUVERkpKSjGpMTEzg6+sr1ZSnsLAQOp3O6EFERETyVO2Qc+rUKVhZWUGlUuH111/Htm3b4O7ujoyMDCiVStjY2BjV29vbIyMjAwCQkZFhFHAM8w3zKqrR6XT4+++/cePGDZSUlJRZY2ijPIsXL4ZGo5Eezs7O1d18IiIiqieqHXLatm2L5ORkHDlyBNOmTYO/vz/++OOP2uhbjQsNDUVubq70uHLlSl13iYiIiGpJg+ouoFQq0bp1awCAl5cXjh07htWrV2PMmDEoKipCTk6O0dGczMxMODg4AAAcHBxKXQVluPrq3pr7r8jKzMyEWq2GhYUFTE1NYWpqWmaNoY3yqFQqqFSq6m4yERER1UMPfZ8cvV6PwsJCeHl5wczMDHFxcdK81NRUXL58GT4+PgAAHx8fnDp1yugqqNjYWKjVari7u0s197ZhqDG0oVQq4eXlZVSj1+sRFxcn1RARERFV60hOaGgohgwZghYtWiAvLw8bN25EQkICdu/eDY1Gg0mTJiEkJAS2trZQq9WYMWMGfHx80KNHDwDAoEGD4O7ujgkTJmDZsmXIyMjA/PnzERgYKB1hef311/HJJ59gzpw5ePXVVxEfH4/NmzcjOvp/V/6EhITA398fXbt2Rffu3bFq1SoUFBRg4sSJNTg0REREVJ9VK+RkZWXhlVdeQXp6OjQaDTp27Ijdu3fj2WefBQCsXLkSJiYmGDVqFAoLC6HVavHpp59Ky5uammLnzp2YNm0afHx8YGlpCX9/fyxcuFCqcXNzQ3R0NGbNmoXVq1fDyckJ69atg1arlWrGjBmD69evIywsDBkZGejUqRNiYmJKnYxMRERET66Hvk9Ofcb75BjjfXIeDd4nh4jo4dT6fXKIiIiIHmcMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEvVCjmLFy9Gt27dYG1tDTs7O4wYMQKpqalGNf369YNCoTB6vP7660Y1ly9fhp+fHxo2bAg7OzvMnj0bxcXFRjUJCQno0qULVCoVWrdujaioqFL9WbNmDVxdXWFubg5vb28cPXq0OptDREREMlatkLN//34EBgbi8OHDiI2NxZ07dzBo0CAUFBQY1U2ZMgXp6enSY9myZdK8kpIS+Pn5oaioCIcOHcLXX3+NqKgohIWFSTVpaWnw8/ND//79kZycjODgYEyePBm7d++WajZt2oSQkBCEh4fj+PHj8PT0hFarRVZW1oOOBREREcmIQgghHnTh69evw87ODvv370efPn0A3D2S06lTJ6xatarMZXbt2oVhw4bh2rVrsLe3BwCsXbsWc+fOxfXr16FUKjF37lxER0fj9OnT0nJjx45FTk4OYmJiAADe3t7o1q0bPvnkEwCAXq+Hs7MzZsyYgXnz5lWp/zqdDhqNBrm5uVCr1Q86DLLhOi+6rrvwRLi4xK+uu0BEVK9V9fP7oc7Jyc3NBQDY2toaTf/uu+/QpEkTdOjQAaGhobh9+7Y0LzExER4eHlLAAQCtVgudToczZ85INb6+vkZtarVaJCYmAgCKioqQlJRkVGNiYgJfX1+ppiyFhYXQ6XRGDyIiIpKnBg+6oF6vR3BwMHr16oUOHTpI01966SW4uLjA0dERJ0+exNy5c5GamoqtW7cCADIyMowCDgDpeUZGRoU1Op0Of//9N27duoWSkpIya1JSUsrt8+LFi7FgwYIH3WQiIiKqRx445AQGBuL06dP49ddfjaZPnTpV+reHhweaNWuGgQMH4sKFC2jVqtWD97QGhIaGIiQkRHqu0+ng7Oxchz0iIiKi2vJAIScoKAg7d+7EgQMH4OTkVGGtt7c3AOD8+fNo1aoVHBwcSl0FlZmZCQBwcHCQ/muYdm+NWq2GhYUFTE1NYWpqWmaNoY2yqFQqqFSqqm0kERER1WvVOidHCIGgoCBs27YN8fHxcHNzq3SZ5ORkAECzZs0AAD4+Pjh16pTRVVCxsbFQq9Vwd3eXauLi4ozaiY2NhY+PDwBAqVTCy8vLqEav1yMuLk6qISIioidbtY7kBAYGYuPGjdixYwesra2lc2g0Gg0sLCxw4cIFbNy4EUOHDkXjxo1x8uRJzJo1C3369EHHjh0BAIMGDYK7uzsmTJiAZcuWISMjA/Pnz0dgYKB0lOX111/HJ598gjlz5uDVV19FfHw8Nm/ejOjo/139ExISAn9/f3Tt2hXdu3fHqlWrUFBQgIkTJ9bU2BAREVE9Vq2QExkZCeDuZeL3Wr9+PQICAqBUKrF3714pcDg7O2PUqFGYP3++VGtqaoqdO3di2rRp8PHxgaWlJfz9/bFw4UKpxs3NDdHR0Zg1axZWr14NJycnrFu3DlqtVqoZM2YMrl+/jrCwMGRkZKBTp06IiYkpdTIyERERPZke6j459R3vk2OM98l5NHifHCKih/NI7pNDRERE9LhiyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlmqVshZvHgxunXrBmtra9jZ2WHEiBFITU01qvnnn38QGBiIxo0bw8rKCqNGjUJmZqZRzeXLl+Hn54eGDRvCzs4Os2fPRnFxsVFNQkICunTpApVKhdatWyMqKqpUf9asWQNXV1eYm5vD29sbR48erc7mEBERkYxVK+Ts378fgYGBOHz4MGJjY3Hnzh0MGjQIBQUFUs2sWbPw888/48cff8T+/ftx7do1jBw5UppfUlICPz8/FBUV4dChQ/j6668RFRWFsLAwqSYtLQ1+fn7o378/kpOTERwcjMmTJ2P37t1SzaZNmxASEoLw8HAcP34cnp6e0Gq1yMrKepjxICIiIplQCCHEgy58/fp12NnZYf/+/ejTpw9yc3PRtGlTbNy4ES+++CIAICUlBe3bt0diYiJ69OiBXbt2YdiwYbh27Rrs7e0BAGvXrsXcuXNx/fp1KJVKzJ07F9HR0Th9+rS0rrFjxyInJwcxMTEAAG9vb3Tr1g2ffPIJAECv18PZ2RkzZszAvHnzqtR/nU4HjUaD3NxcqNXqBx0G2XCdF13XXXgiXFziV9ddICKq16r6+f1Q5+Tk5uYCAGxtbQEASUlJuHPnDnx9faWadu3aoUWLFkhMTAQAJCYmwsPDQwo4AKDVaqHT6XDmzBmp5t42DDWGNoqKipCUlGRUY2JiAl9fX6mGiIiInmwNHnRBvV6P4OBg9OrVCx06dAAAZGRkQKlUwsbGxqjW3t4eGRkZUs29Accw3zCvohqdToe///4bt27dQklJSZk1KSkp5fa5sLAQhYWF0nOdTleNLSYiIqL65IGP5AQGBuL06dP44YcfarI/tWrx4sXQaDTSw9nZua67RERERLXkgUJOUFAQdu7ciX379sHJyUma7uDggKKiIuTk5BjVZ2ZmwsHBQaq5/2orw/PKatRqNSwsLNCkSROYmpqWWWNooyyhoaHIzc2VHleuXKnehhMREVG9Ua2QI4RAUFAQtm3bhvj4eLi5uRnN9/LygpmZGeLi4qRpqampuHz5Mnx8fAAAPj4+OHXqlNFVULGxsVCr1XB3d5dq7m3DUGNoQ6lUwsvLy6hGr9cjLi5OqimLSqWCWq02ehAREZE8VeucnMDAQGzcuBE7duyAtbW1dA6NRqOBhYUFNBoNJk2ahJCQENja2kKtVmPGjBnw8fFBjx49AACDBg2Cu7s7JkyYgGXLliEjIwPz589HYGAgVCoVAOD111/HJ598gjlz5uDVV19FfHw8Nm/ejOjo/139ExISAn9/f3Tt2hXdu3fHqlWrUFBQgIkTJ9bU2BAREVE9Vq2QExkZCQDo16+f0fT169cjICAAALBy5UqYmJhg1KhRKCwshFarxaeffirVmpqaYufOnZg2bRp8fHxgaWkJf39/LFy4UKpxc3NDdHQ0Zs2ahdWrV8PJyQnr1q2DVquVasaMGYPr168jLCwMGRkZ6NSpE2JiYkqdjExERERPpoe6T059x/vkGON9ch4N3ieHiOjhPJL75BARERE9rhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJYYcoiIiEiWGHKIiIhIlhhyiIiISJaqHXIOHDiA4cOHw9HREQqFAtu3bzeaHxAQAIVCYfQYPHiwUc3Nmzcxfvx4qNVq2NjYYNKkScjPzzeqOXnyJHr37g1zc3M4Oztj2bJlpfry448/ol27djA3N4eHhwd++eWX6m4OERERyVS1Q05BQQE8PT2xZs2acmsGDx6M9PR06fH9998bzR8/fjzOnDmD2NhY7Ny5EwcOHMDUqVOl+TqdDoMGDYKLiwuSkpKwfPlyRERE4PPPP5dqDh06hHHjxmHSpEk4ceIERowYgREjRuD06dPV3SQiIiKSIYUQQjzwwgoFtm3bhhEjRkjTAgICkJOTU+oIj8HZs2fh7u6OY8eOoWvXrgCAmJgYDB06FFevXoWjoyMiIyPxzjvvICMjA0qlEgAwb948bN++HSkpKQCAMWPGoKCgADt37pTa7tGjBzp16oS1a9dWqf86nQ4ajQa5ublQq9UPMALy4jovuq678ES4uMSvrrtARFSvVfXzu1bOyUlISICdnR3atm2LadOmITs7W5qXmJgIGxsbKeAAgK+vL0xMTHDkyBGppk+fPlLAAQCtVovU1FTcunVLqvH19TVar1arRWJiYrn9KiwshE6nM3oQERGRPNV4yBk8eDC++eYbxMXFYenSpdi/fz+GDBmCkpISAEBGRgbs7OyMlmnQoAFsbW2RkZEh1djb2xvVGJ5XVmOYX5bFixdDo9FID2dn54fbWCIiInpsNajpBseOHSv928PDAx07dkSrVq2QkJCAgQMH1vTqqiU0NBQhISHSc51Ox6BDREQkU7V+CXnLli3RpEkTnD9/HgDg4OCArKwso5ri4mLcvHkTDg4OUk1mZqZRjeF5ZTWG+WVRqVRQq9VGDyIiIpKnWg85V69eRXZ2Npo1awYA8PHxQU5ODpKSkqSa+Ph46PV6eHt7SzUHDhzAnTt3pJrY2Fi0bdsWjRo1kmri4uKM1hUbGwsfH5/a3iQiIiKqB6odcvLz85GcnIzk5GQAQFpaGpKTk3H58mXk5+dj9uzZOHz4MC5evIi4uDg8//zzaN26NbRaLQCgffv2GDx4MKZMmYKjR4/i4MGDCAoKwtixY+Ho6AgAeOmll6BUKjFp0iScOXMGmzZtwurVq42+apo5cyZiYmKwYsUKpKSkICIiAr/99huCgoJqYFiIiIiovqt2yPntt9/QuXNndO7cGQAQEhKCzp07IywsDKampjh58iSee+45PPXUU5g0aRK8vLzwn//8ByqVSmrju+++Q7t27TBw4EAMHToUzzzzjNE9cDQaDfbs2YO0tDR4eXnhzTffRFhYmNG9dHr27ImNGzfi888/h6enJ7Zs2YLt27ejQ4cODzMeREREJBMPdZ+c+o73yTHG++Q8GrxPDhHRw6nT++QQERER1TWGHCIiIpIlhhwiIiKSJYYcIiIikiWGHCIiIpIlhhwiIiKSJYYcIiIikiWGHCIiIpIlhhwiIiKSJYYcIiIikiWGHCIiIpIlhhwiIiKSJYYcIiIikiWGHCIiIpIlhhwiIiKSJYYcIiIikiWGHCIiIpIlhhwiIiKSJYYcIiIikiWGHCIiIpIlhhwiIiKSJYYcIiIikiWGHCIiIpIlhhwiIiKSJYYcIiIikiWGHCIiIpIlhhwiIiKSJYYcIiIikqUGdd0BuXKdF13XXSAiInqi8UgOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJ/hZyIqsR1XnRdd6HaLi7xq+suEFEdqnbIOXDgAJYvX46kpCSkp6dj27ZtGDFihDRfCIHw8HB88cUXyMnJQa9evRAZGYk2bdpINTdv3sSMGTPw888/w8TEBKNGjcLq1athZWUl1Zw8eRKBgYE4duwYmjZtihkzZmDOnDlGffnxxx/x7rvv4uLFi2jTpg2WLl2KoUOHPsAwED069TEsEBHVR9X+uqqgoACenp5Ys2ZNmfOXLVuGjz76CGvXrsWRI0dgaWkJrVaLf/75R6oZP348zpw5g9jYWOzcuRMHDhzA1KlTpfk6nQ6DBg2Ci4sLkpKSsHz5ckRERODzzz+Xag4dOoRx48Zh0qRJOHHiBEaMGIERI0bg9OnT1d0kIiIikiGFEEI88MIKhdGRHCEEHB0d8eabb+Ktt94CAOTm5sLe3h5RUVEYO3Yszp49C3d3dxw7dgxdu3YFAMTExGDo0KG4evUqHB0dERkZiXfeeQcZGRlQKpUAgHnz5mH79u1ISUkBAIwZMwYFBQXYuXOn1J8ePXqgU6dOWLt2bZX6r9PpoNFokJubC7Va/aDDUCb+3zpR3ePXVUTyVNXP7xo98TgtLQ0ZGRnw9fWVpmk0Gnh7eyMxMREAkJiYCBsbGyngAICvry9MTExw5MgRqaZPnz5SwAEArVaL1NRU3Lp1S6q5dz2GGsN6ylJYWAidTmf0ICIiInmq0ZCTkZEBALC3tzeabm9vL83LyMiAnZ2d0fwGDRrA1tbWqKasNu5dR3k1hvllWbx4MTQajfRwdnau7iYSERFRPfFEXUIeGhqK3Nxc6XHlypW67hIRERHVkhoNOQ4ODgCAzMxMo+mZmZnSPAcHB2RlZRnNLy4uxs2bN41qymrj3nWUV2OYXxaVSgW1Wm30ICIiInmq0ZDj5uYGBwcHxMXFSdN0Oh2OHDkCHx8fAICPjw9ycnKQlJQk1cTHx0Ov18Pb21uqOXDgAO7cuSPVxMbGom3btmjUqJFUc+96DDWG9RAREdGTrdohJz8/H8nJyUhOTgZw92Tj5ORkXL58GQqFAsHBwfj3v/+Nn376CadOncIrr7wCR0dH6Qqs9u3bY/DgwZgyZQqOHj2KgwcPIigoCGPHjoWjoyMA4KWXXoJSqcSkSZNw5swZbNq0CatXr0ZISIjUj5kzZyImJgYrVqxASkoKIiIi8NtvvyEoKOjhR4WIiIjqvWrfDPC3335D//79peeG4OHv74+oqCjMmTMHBQUFmDp1KnJycvDMM88gJiYG5ubm0jLfffcdgoKCMHDgQOlmgB999JE0X6PRYM+ePQgMDISXlxeaNGmCsLAwo3vp9OzZExs3bsT8+fPx9ttvo02bNti+fTs6dOjwQANBRERE8vJQ98mp73ifHCJ5431yiOSpTu6TQ0RERPS4YMghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZYsghIiIiWWLIISIiIlliyCEiIiJZqvGQExERAYVCYfRo166dNP+ff/5BYGAgGjduDCsrK4waNQqZmZlGbVy+fBl+fn5o2LAh7OzsMHv2bBQXFxvVJCQkoEuXLlCpVGjdujWioqJqelOIiIioHquVIzlPP/000tPTpcevv/4qzZs1axZ+/vln/Pjjj9i/fz+uXbuGkSNHSvNLSkrg5+eHoqIiHDp0CF9//TWioqIQFhYm1aSlpcHPzw/9+/dHcnIygoODMXnyZOzevbs2NoeIiIjqoQa10miDBnBwcCg1PTc3F19++SU2btyIAQMGAADWr1+P9u3b4/Dhw+jRowf27NmDP/74A3v37oW9vT06deqERYsWYe7cuYiIiIBSqcTatWvh5uaGFStWAADat2+PX3/9FStXroRWq62NTSIiIqJ6plaO5Jw7dw6Ojo5o2bIlxo8fj8uXLwMAkpKScOfOHfj6+kq17dq1Q4sWLZCYmAgASExMhIeHB+zt7aUarVYLnU6HM2fOSDX3tmGoMbRRnsLCQuh0OqMHERERyVONhxxvb29ERUUhJiYGkZGRSEtLQ+/evZGXl4eMjAwolUrY2NgYLWNvb4+MjAwAQEZGhlHAMcw3zKuoRqfT4e+//y63b4sXL4ZGo5Eezs7OD7u5RERE9Jiq8a+rhgwZIv27Y8eO8Pb2houLCzZv3gwLC4uaXl21hIaGIiQkRHqu0+kYdIiIiGSq1i8ht7GxwVNPPYXz58/DwcEBRUVFyMnJMarJzMyUzuFxcHAodbWV4XllNWq1usIgpVKpoFarjR5EREQkT7UecvLz83HhwgU0a9YMXl5eMDMzQ1xcnDQ/NTUVly9fho+PDwDAx8cHp06dQlZWllQTGxsLtVoNd3d3qebeNgw1hjaIiIiIajzkvPXWW9i/fz8uXryIQ4cO4YUXXoCpqSnGjRsHjUaDSZMmISQkBPv27UNSUhImTpwIHx8f9OjRAwAwaNAguLu7Y8KECfj999+xe/duzJ8/H4GBgVCpVACA119/Hf/9738xZ84cpKSk4NNPP8XmzZsxa9asmt4cIiIiqqdq/Jycq1evYty4ccjOzkbTpk3xzDPP4PDhw2jatCkAYOXKlTAxMcGoUaNQWFgIrVaLTz/9VFre1NQUO3fuxLRp0+Dj4wNLS0v4+/tj4cKFUo2bmxuio6Mxa9YsrF69Gk5OTli3bh0vHyciIiKJQggh6roTdUWn00Gj0SA3N7fGz89xnRddo+0RUfVdXOJX110golpQ1c9v/nYVERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyVKDuu4AEVFtcZ0XXdddqLaLS/zqugtEssEjOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkSww5REREJEsMOURERCRLDDlEREQkS/U+5KxZswaurq4wNzeHt7c3jh49WtddIiIiosdAg7ruwMPYtGkTQkJCsHbtWnh7e2PVqlXQarVITU2FnZ1dXXePiKjaXOdF13UXqu3iEr+67gJRmer1kZwPP/wQU6ZMwcSJE+Hu7o61a9eiYcOG+Oqrr+q6a0RERFTH6u2RnKKiIiQlJSE0NFSaZmJiAl9fXyQmJtZhz4iIniw8+kSPq3obcm7cuIGSkhLY29sbTbe3t0dKSkqZyxQWFqKwsFB6npubCwDQ6XQ13j994e0ab5OIiGpGbfzdp0fH8PoJISqsq7ch50EsXrwYCxYsKDXd2dm5DnpDRER1RbOqrntANSEvLw8ajabc+fU25DRp0gSmpqbIzMw0mp6ZmQkHB4cylwkNDUVISIj0XK/X4+bNm2jcuDEUCkWt9vdxotPp4OzsjCtXrkCtVtd1d+oMx+F/OBZ3cRzu4jjcxXH4n8dtLIQQyMvLg6OjY4V19TbkKJVKeHl5IS4uDiNGjABwN7TExcUhKCiozGVUKhVUKpXRNBsbm1ru6eNLrVY/FjtrXeM4/A/H4i6Ow10ch7s4Dv/zOI1FRUdwDOptyAGAkJAQ+Pv7o2vXrujevTtWrVqFgoICTJw4sa67RkRERHWsXoecMWPG4Pr16wgLC0NGRgY6deqEmJiYUicjExER0ZOnXoccAAgKCir36ykqm0qlQnh4eKmv7p40HIf/4VjcxXG4i+NwF8fhf+rrWChEZddfEREREdVD9fqOx0RERETlYcghIiIiWWLIISIiIlliyCEiIiJZYsiRmcWLF6Nbt26wtraGnZ0dRowYgdTU1AqXiYqKgkKhMHqYm5s/oh7XjoiIiFLb1K5duwqX+fHHH9GuXTuYm5vDw8MDv/zyyyPqbe1ydXUtNRYKhQKBgYFl1stlfzhw4ACGDx8OR0dHKBQKbN++3Wi+EAJhYWFo1qwZLCws4Ovri3PnzlXa7po1a+Dq6gpzc3N4e3vj6NGjtbQFNaOicbhz5w7mzp0LDw8PWFpawtHREa+88gquXbtWYZsP8v56HFS2TwQEBJTarsGDB1farpz2CQBl/r1QKBRYvnx5uW0+rvsEQ47M7N+/H4GBgTh8+DBiY2Nx584dDBo0CAUFBRUup1arkZ6eLj0uXbr0iHpce55++mmjbfr111/LrT106BDGjRuHSZMm4cSJExgxYgRGjBiB06dPP8Ie145jx44ZjUNsbCwA4F//+le5y8hhfygoKICnpyfWrFlT5vxly5bho48+wtq1a3HkyBFYWlpCq9Xin3/+KbfNTZs2ISQkBOHh4Th+/Dg8PT2h1WqRlZVVW5vx0Coah9u3b+P48eN49913cfz4cWzduhWpqal47rnnKm23Ou+vx0Vl+wQADB482Gi7vv/++wrblNs+AcBo+9PT0/HVV19BoVBg1KhRFbb7WO4TgmQtKytLABD79+8vt2b9+vVCo9E8uk49AuHh4cLT07PK9aNHjxZ+fn5G07y9vcVrr71Wwz2rezNnzhStWrUSer2+zPly3B8AiG3btknP9Xq9cHBwEMuXL5em5eTkCJVKJb7//vty2+nevbsIDAyUnpeUlAhHR0exePHiWul3Tbt/HMpy9OhRAUBcunSp3Jrqvr8eR2WNhb+/v3j++eer1c6TsE88//zzYsCAARXWPK77BI/kyFxubi4AwNbWtsK6/Px8uLi4wNnZGc8//zzOnDnzKLpXq86dOwdHR0e0bNkS48ePx+XLl8utTUxMhK+vr9E0rVaLxMTE2u7mI1VUVIRvv/0Wr776aoU/SivH/eFeaWlpyMjIMHrNNRoNvL29y33Ni4qKkJSUZLSMiYkJfH19ZbWf5ObmQqFQVPq7ftV5f9UnCQkJsLOzQ9u2bTFt2jRkZ2eXW/sk7BOZmZmIjo7GpEmTKq19HPcJhhwZ0+v1CA4ORq9evdChQ4dy69q2bYuvvvoKO3bswLfffgu9Xo+ePXvi6tWrj7C3Ncvb2xtRUVGIiYlBZGQk0tLS0Lt3b+Tl5ZVZn5GRUernQOzt7ZGRkfEouvvIbN++HTk5OQgICCi3Ro77w/0Mr2t1XvMbN26gpKRE1vvJP//8g7lz52LcuHEV/ghjdd9f9cXgwYPxzTffIC4uDkuXLsX+/fsxZMgQlJSUlFn/JOwTX3/9NaytrTFy5MgK6x7XfaLe/6wDlS8wMBCnT5+u9HtRHx8f+Pj4SM979uyJ9u3b47PPPsOiRYtqu5u1YsiQIdK/O3bsCG9vb7i4uGDz5s1V+j8Sufryyy8xZMgQODo6llsjx/2BKnfnzh2MHj0aQghERkZWWCvX99fYsWOlf3t4eKBjx45o1aoVEhISMHDgwDrsWd356quvMH78+EovPnhc9wkeyZGpoKAg7Ny5E/v27YOTk1O1ljUzM0Pnzp1x/vz5Wurdo2djY4Onnnqq3G1ycHBAZmam0bTMzEw4ODg8iu49EpcuXcLevXsxefLkai0nx/3B8LpW5zVv0qQJTE1NZbmfGALOpUuXEBsbW+FRnLJU9v6qr1q2bIkmTZqUu11y3icA4D//+Q9SU1Or/TcDeHz2CYYcmRFCICgoCNu2bUN8fDzc3Nyq3UZJSQlOnTqFZs2a1UIP60Z+fj4uXLhQ7jb5+PggLi7OaFpsbKzREY36bv369bCzs4Ofn1+1lpPj/uDm5gYHBwej11yn0+HIkSPlvuZKpRJeXl5Gy+j1esTFxdXr/cQQcM6dO4e9e/eicePG1W6jsvdXfXX16lVkZ2eXu11y3ScMvvzyS3h5ecHT07Payz42+0Rdn/lMNWvatGlCo9GIhIQEkZ6eLj1u374t1UyYMEHMmzdPer5gwQKxe/duceHCBZGUlCTGjh0rzM3NxZkzZ+piE2rEm2++KRISEkRaWpo4ePCg8PX1FU2aNBFZWVlCiNJjcPDgQdGgQQPxwQcfiLNnz4rw8HBhZmYmTp06VVebUKNKSkpEixYtxNy5c0vNk+v+kJeXJ06cOCFOnDghAIgPP/xQnDhxQrpqaMmSJcLGxkbs2LFDnDx5Ujz//PPCzc1N/P3331IbAwYMEB9//LH0/IcffhAqlUpERUWJP/74Q0ydOlXY2NiIjIyMR759VVXROBQVFYnnnntOODk5ieTkZKO/GYWFhVIb949DZe+vx1VFY5GXlyfeeustkZiYKNLS0sTevXtFly5dRJs2bcQ///wjtSH3fcIgNzdXNGzYUERGRpbZRn3ZJxhyZAZAmY/169dLNX379hX+/v7S8+DgYNGiRQuhVCqFvb29GDp0qDh+/Pij73wNGjNmjGjWrJlQKpWiefPmYsyYMeL8+fPS/PvHQAghNm/eLJ566imhVCrF008/LaKjox9xr2vP7t27BQCRmppaap5c94d9+/aV+V4wbKterxfvvvuusLe3FyqVSgwcOLDU+Li4uIjw8HCjaR9//LE0Pt27dxeHDx9+RFv0YCoah7S0tHL/Zuzbt09q4/5xqOz99biqaCxu374tBg0aJJo2bSrMzMyEi4uLmDJlSqmwIvd9wuCzzz4TFhYWIicnp8w26ss+oRBCiFo9VERERERUB3hODhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw5RHbp48SIUCgWSk5PruiuSlJQU9OjRA+bm5ujUqVOVl+vXrx+Cg4NrrV81ra76+6DjS0TVx5BDT7SAgAAoFAosWbLEaPr27duhUCjqqFd1Kzw8HJaWlkhNTS31e141KSEhAQqFAjk5ObW2jops3bq1Wr+qXlOB9FGNLxEx5BDB3NwcS5cuxa1bt+q6KzWmqKjogZe9cOECnnnmGbi4uDzQjzXWF7a2trC2tn7k662J8X2Y11cu7ty5U9ddoHqAIYeeeL6+vnBwcMDixYvLrYmIiCj11cKqVavg6uoqPQ8ICMCIESPw/vvvw97eHjY2Nli4cCGKi4sxe/Zs2NrawsnJCevXry/VfkpKCnr27Alzc3N06NAB+/fvN5p/+vRpDBkyBFZWVrC3t8eECRNw48YNaX6/fv0QFBSE4OBgNGnSBFqttszt0Ov1WLhwIZycnKBSqdCpUyfExMRI8xUKBZKSkrBw4UIoFApERESU2U5BQQFeeeUVWFlZoVmzZlixYkWpmg0bNqBr166wtraGg4MDXnrpJWRlZQG4e1Skf//+AIBGjRpBoVAgICAAABATE4NnnnkGNjY2aNy4MYYNG4YLFy5I7RqOqPzwww8Vjtn+/fvRvXt3qFQqNGvWDPPmzUNxcbHRmN37dZWrqyvef/99vPrqq7C2tkaLFi3w+eefS/Pd3NwAAJ07d4ZCoUC/fv0A3D0i1b17d1haWsLGxga9evXCpUuXyhy38sb31KlTGDBgACwsLNC4cWNMnToV+fn50nKGfeu9996Do6Mj2rZtW2b7ABAZGYlWrVpBqVSibdu22LBhQ6k+rFu3Di+88AIaNmyINm3a4KeffjKqqWx/u19V3h+VjdOOHTvQpUsXmJubo2XLlliwYIHR66VQKBAZGYnnnnsOlpaWeO+998rtD5Gkrn88i6gu+fv7i+eff15s3bpVmJubiytXrgghhNi2bZu49+0RHh4uPD09jZZduXKlcHFxMWrL2tpaBAYGipSUFPHll18KAEKr1Yr33ntP/Pnnn2LRokXCzMxMWo/hBxKdnJzEli1bxB9//CEmT54srK2txY0bN4QQQty6dUs0bdpUhIaGirNnz4rjx4+LZ599VvTv319ad9++fYWVlZWYPXu2SElJESkpKWVu74cffijUarX4/vvvRUpKipgzZ44wMzMTf/75pxBCiPT0dPH000+LN998U6Snp4u8vLwy25k2bZpo0aKF2Lt3rzh58qQYNmyYsLa2FjNnzpRqvvzyS/HLL7+ICxcuiMTEROHj4yOGDBkihBCiuLhY/N///Z/0o6Hp6enSDwFu2bJF/N///Z84d+6cOHHihBg+fLjw8PAQJSUlVR6zq1evioYNG4rp06eLs2fPim3btokmTZoY/aBg3759jfrr4uIibG1txZo1a8S5c+fE4sWLhYmJiTSWR48eFQDE3r17RXp6usjOzhZ37twRGo1GvPXWW+L8+fPijz/+EFFRUUa/5nyvssY3Pz9fNGvWTIwcOVKcOnVKxMXFCTc3N6MfS/T39xdWVlZiwoQJ4vTp0+L06dNltr9161ZhZmYm1qxZI1JTU8WKFSuEqampiI+Pl2oMY7dx40Zx7tw58cYbbwgrKyuRnZ0thKja/na/yt4flY3TgQMHhFqtFlFRUeLChQtiz549wtXVVURERBj1287OTnz11VfiwoUL5Y4x0b0YcuiJZgg5QgjRo0cP8eqrrwohHjzkuLi4SB/GQgjRtm1b0bt3b+l5cXGxsLS0FN9//70Q4n8f2EuWLJFq7ty5I5ycnMTSpUuFEEIsWrRIDBo0yGjdV65cMfpV8b59+4rOnTtXur2Ojo7ivffeM5rWrVs3MX36dOm5p6dnqV9ZvldeXp5QKpVi8+bN0rTs7GxhYWFhFBrud+zYMQFACk6GX0K+detWhX2+fv26ACBOnTolhKjamL399tuibdu2Qq/XSzVr1qwRVlZW0utTVsh5+eWXped6vV7Y2dmJyMhIo/WeOHHCaLsBiISEhAq34V73j+/nn38uGjVqJPLz86Vp0dHRwsTERPoFbH9/f2Fvby8KCwsrbLtnz55iypQpRtP+9a9/iaFDh0rPAYj58+dLz/Pz8wUAsWvXLiFE1fa3+1X2/qhsnAYOHCjef/99o2kbNmwQzZo1M+p3cHBwmcsTlYdfVxH9f0uXLsXXX3+Ns2fPPnAbTz/9NExM/ve2sre3h4eHh/Tc1NQUjRs3lr62MfDx8ZH+3aBBA3Tt2lXqx++//459+/bByspKerRr1w4AjL7G8fLyqrBvOp0O165dQ69evYym9+rVq1rbfOHCBRQVFcHb21uaZmtrW+orlKSkJAwfPhwtWrSAtbU1+vbtCwC4fPlyhe2fO3cO48aNQ8uWLaFWq6WvPO5frqIxO3v2LHx8fIxOHu/Vqxfy8/Nx9erVctfdsWNH6d8KhQIODg6lXqt72draIiAgAFqtFsOHD8fq1auRnp5e4fbd7+zZs/D09ISlpaVRX/V6PVJTU6VpHh4eUCqVlbZVldf33u20tLSEWq2WtrOq+1t1VDZOv//+OxYuXGi0zilTpiA9PR23b9+W6rp27fpA66cnF0MO0f/Xp08faLVahIaGlppnYmICIYTRtLJOfDQzMzN6rlAoypym1+ur3K/8/HwMHz4cycnJRo9z586hT58+Ut29H5J1raCgAFqtFmq1Gt999x2OHTuGbdu2Aaj8pNnhw4fj5s2b+OKLL3DkyBEcOXKkSsvVhAd5rdavX4/ExET07NkTmzZtwlNPPYXDhw/XeN9q8vWtaDurur/dqyrvj4rGKT8/HwsWLDBa36lTp3Du3DmYm5tLbTxO+zjVDww5RPdYsmQJfv75ZyQmJhpNb9q0KTIyMoz+kNfkvW3u/VAsLi5GUlIS2rdvDwDo0qULzpw5A1dXV7Ru3droUZ0/+mq1Go6Ojjh48KDR9IMHD8Ld3b3K7bRq1QpmZmZS+ACAW7du4c8//5Sep6SkIDs7G0uWLEHv3r3Rrl27UkdEDEclSkpKpGnZ2dlITU3F/PnzMXDgQLRv377cq94qGrP27dsjMTHR6PU6ePAgrK2t4eTkVOVtray/Bp07d0ZoaCgOHTqEDh06YOPGjVVut3379vj9999RUFBg1FcTE5MKTzAur62HfX0fZH+r6vujvHHq0qULUlNTS62vdevWRkdGiaqLew/RPTw8PDB+/Hh89NFHRtP79euH69evY9myZbhw4QLWrFmDXbt21dh616xZg23btiElJQWBgYG4desWXn31VQBAYGAgbt68iXHjxuHYsWO4cOECdu/ejYkTJ5b5gVuR2bNnY+nSpdi0aRNSU1Mxb948JCcnY+bMmVVuw8rKCpMmTcLs2bMRHx+P06dPIyAgwOjDqEWLFlAqlfj444/x3//+Fz/99FOpe9K4uLhAoVBg586duH79OvLz89GoUSM0btwYn3/+Oc6fP4/4+HiEhIRUe8ymT5+OK1euYMaMGUhJScGOHTsQHh6OkJCQB/7QtLOzg4WFBWJiYpCZmYnc3FykpaUhNDQUiYmJuHTpEvbs2YNz585JYasqxo8fD3Nzc/j7++P06dPYt28fZsyYgQkTJsDe3r5afZw9ezaioqIQGRmJc+fO4cMPP8TWrVvx1ltvVbmNB9nfKnt/VDZOYWFh+Oabb7BgwQKcOXMGZ8+exQ8//ID58+dXa/uJSqnTM4KI6ti9Jx4bpKWlCaVSKe5/e0RGRgpnZ2dhaWkpXnnlFfHee++VOvH4/rbuP7lViLsnuK5cuVJaFwCxceNG0b17d6FUKoW7u7vR1TBCCPHnn3+KF154QdjY2AgLCwvRrl07ERwcLJ1YW9Z6ylJSUiIiIiJE8+bNhZmZmfD09JROODWo7MRjIe6efPzyyy+Lhg0bCnt7e7Fs2bJSfdi4caNwdXUVKpVK+Pj4iJ9++qnUibsLFy4UDg4OQqFQSFcTxcbGivbt2wuVSiU6duwoEhISBACxbdu2ao1ZQkKC6Natm1AqlcLBwUHMnTtX3LlzR5pf1onHhtelvLH44osvhLOzszAxMRF9+/YVGRkZYsSIEaJZs2ZCqVQKFxcXERYWZnTy+f3KGt+TJ0+K/v37C3Nzc2FrayumTJlidGVbWftWeT799FPRsmVLYWZmJp566inxzTffGM2/dywNNBqNWL9+vfS8sv2tLBW9P6oyTjExMaJnz57CwsJCqNVq0b17d/H5559X2G+iyiiEuO+LVCKix9jFixfh5uaGEydO8GcRiKhC/LqKiIiIZIkhh4iIiGSJX1cRERGRLPFIDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERyRJDDhEREckSQw4RERHJEkMOERERydL/A4cVUkMppsJBAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def user_dataset_len_sampler():\n",
    "    while True:\n",
    "        length = np.random.poisson(5)\n",
    "        if length > 0:\n",
    "            return length\n",
    "\n",
    "# Visualize sampling from this function.\n",
    "plt.xlabel('Number of datapoints for one user')\n",
    "plt.title('Distribution of user dataset length')\n",
    "plt.hist([user_dataset_len_sampler() for _ in range(100000)])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Secondly, we define a function `Callable[[length: int], List[int]]` which samples a set of datapoint identifiers of a given length. We use the helper function `get_data_sampler` to build a sampler which draws a set of datapoints uniformly. This will result in IID user datasets. When doing this for your use case you might want to consider sampling in a way which creates more realistic and heterogeneous user datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sampled a user dataset of length 5 with datapoint ids  [7192 2501 4752 3850 5314]\n"
     ]
    }
   ],
   "source": [
    "from pfl.data.sampling import get_data_sampler\n",
    "\n",
    "train_data_sampler = get_data_sampler('random', len(train_images))\n",
    "val_data_sampler = get_data_sampler('random', len(val_images))\n",
    "\n",
    "print('Sampled a user dataset of length 5 with datapoint ids ', train_data_sampler(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thirdly, the federated datasets are initialized using the above sampling functions. We also create a central dataset for central evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pfl.data.federated_dataset import ArtificialFederatedDataset\n",
    "from pfl.data.dataset import Dataset\n",
    "\n",
    "train_federated_dataset = ArtificialFederatedDataset.from_slices([train_images, train_labels], train_data_sampler, user_dataset_len_sampler)\n",
    "val_federated_dataset = ArtificialFederatedDataset.from_slices([val_images, val_labels], val_data_sampler, user_dataset_len_sampler)\n",
    "central_data = Dataset(raw_data=[val_images, val_labels])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When iterating through an artificial federated dataset, a user dataset is sampled every iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User: <pfl.data.dataset.Dataset object at 0x174537cd0>, unique user seed: 385465319\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA9EAAAFTCAYAAAA+14+JAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABeTUlEQVR4nO3deZRmVX3v/8855xlr7rm66G66mZHxpgVsNYpIQPRHNHLVmEFMXEFNY5Z2XCq/ZTSa5NcOiZpcEcwNFxJzCQ436NUYDDI0agClhTAEmsFuuqG7eq6qruEZz/n9gV2xpZ7vd1d3FdVVvF9r1VpQe/f37Geffb7n7Hqqnm+UZVkmAAAAAADgimd6AAAAAAAAzBZsogEAAAAACMQmGgAAAACAQGyiAQAAAAAIxCYaAAAAAIBAbKIBAAAAAAjEJhoAAAAAgEBsogEAAAAACMQmGgAAAACAQGyicdhuuOEGRVGkLVu2TOrfnX/++Tr99NOndCwrV67UO9/5zimNCQBHghwJABMjP2K2YxMN/IKf/OQnuvLKK3Xaaaepvb1dK1as0Fvf+lY9/vjjMz00AJhx5EgAaK1arerDH/6w+vr6VC6Xdd555+nWW2+d6WFhGuRmegDA0eTTn/60fvSjH+ktb3mLzjzzTPX39+uLX/yifuVXfkX33HPPlP/0EwBmE3IkALT2zne+U9/4xjf0/ve/XyeeeKJuuOEGvf71r9cdd9yhV77ylTM9PEwhNtHAL1i3bp1uvPFGFQqF8e+97W1v0xlnnKFPfepT+sd//McZHB0AzCxyJABM7Mc//rFuuukmffazn9UHP/hBSdI73vEOnX766frQhz6kf//3f5/hEWIq8evcmDLf+ta39IY3vEF9fX0qFos6/vjj9Wd/9mdqNpsT9t+4caNe/vKXq1wua9WqVbr22muf16darerjH/+4TjjhBBWLRS1fvlwf+tCHVK1Wp+U1vPzlLz/k4VCSTjzxRJ122ml69NFHp+WYAF4cyJEAMLG5kB+/8Y1vKEkSXXHFFePfK5VKete73qW7775b27Ztm5bjYmbwTjSmzA033KCOjg6tW7dOHR0duv322/Wxj31MQ0ND+uxnP3tI3/379+v1r3+93vrWt+rtb3+7vva1r+m9732vCoWCfv/3f1+SlKapfv3Xf10//OEPdcUVV+jUU0/VQw89pM9//vN6/PHH9c1vfrPlWNI01b59+4LG3d3drXw+37I9yzLt3LlTp512WlA8AJgIORIAJjYX8uP999+vk046SV1dXYf0OffccyVJDzzwgJYvXx46JTjaZcBhuv766zNJ2ebNm7Msy7LR0dHn9Xn3u9+dtbW1ZZVKZfx7r371qzNJ2V/91V+Nf69arWZnn312tnjx4qxWq2VZlmVf+cpXsjiOsx/84AeHxLz22mszSdmPfvSj8e8de+yx2eWXXz7+/5s3b84kBX3dcccd5uv8yle+kknKrrvuutCpAQByJAC0MBfz42mnnZZdcMEFz3sdjzzySCYpu/baayc1Rzi68U40pky5XB7/7wMHDqharepXf/VX9eUvf1mPPfaYzjrrrPH2XC6nd7/73eP/XygU9O53v1vvfe97tXHjRr3sZS/T17/+dZ166qk65ZRTtGfPnvG+F1xwgSTpjjvu0Mtf/vIJx9Lb2xv8aYi/OK5f9thjj2nt2rVas2aNLr/88qB4ADARciQATGwu5MexsTEVi8Xn9SmVSuPtmDvYRGPKPPLII/roRz+q22+/XUNDQ4e0DQ4OHvL/fX19am9vP+R7J510kiRpy5YtetnLXqYnnnhCjz76qBYtWjTh8Xbt2tVyLKVSSRdeeOHhvIxx/f39esMb3qDu7u7xv3MBgMNFjgSAic2F/Fgulyf8e+tKpTLejrmDTTSmxMDAgF796lerq6tLn/zkJ3X88cerVCrppz/9qT784Q8rTdNJx0zTVGeccYY+97nPTdhu/V1Js9nU7t27g44zf/78531QzuDgoC655BINDAzoBz/4gfr6+sIHDgC/hBwJABObK/lx6dKlevbZZ5/XZ8eOHZJEnpxj2ERjStx5553au3ev/vmf/1mvetWrxr+/efPmCftv375dIyMjh/wk8fHHH5ckrVy5UpJ0/PHH6z/+4z/02te+VlEUTWo827Zt06pVq4L63nHHHTr//PPH/79SqejSSy/V448/ru9///t6yUteMqljA8AvI0cCwMTmSn48++yzdccdd2hoaOiQDxe79957x9sxd7CJxpQ4+Gt8WZaNf69Wq+lLX/rShP0bjYa+/OUva926deN9v/zlL2vRokVavXq1JOmtb32rvvvd7+p//s//eUi5AOm5vytJ0/R5v85z0OH+PUuz2dTb3vY23X333frWt76lNWvWBMUAAAs5EgAmNlfy43//7/9df/mXf6m//du/Ha8TXa1Wdf311+u8887jk7nnGDbRmBIvf/nLNW/ePF1++eX6oz/6I0VRpK985SuHJMRf1NfXp09/+tPasmWLTjrpJH31q1/VAw88oL/9278dLxXwu7/7u/ra176m97znPbrjjjv0ile8Qs1mU4899pi+9rWv6Xvf+55e+tKXThj/cP+e5Y//+I/1f//v/9Wll16qffv26R//8R8Paf+d3/mdSccEAHIkAExsruTH8847T295y1t01VVXadeuXTrhhBP093//99qyZYuuu+66ScfDUW4mPxocs9svlyf40Y9+lL3sZS/LyuVy1tfXl33oQx/Kvve97z2vBMCrX/3q7LTTTsvuu+++bM2aNVmpVMqOPfbY7Itf/OLzjlGr1bJPf/rT2WmnnZYVi8Vs3rx52erVq7NPfOIT2eDg4Hi/Xy5PcLgOlk5o9QUAociRADCxuZgfsyzLxsbGsg9+8INZb29vViwWs3POOSe75ZZbpiQ2ji5RlrX4MQ8AAAAAADhEPNMDAAAAAABgtmATDQAAAABAIDbRAAAAAAAEYhMNAAAAAEAgNtEAAAAAAARiEw0AAAAAQKDcTA/gl6Vpqu3bt6uzs1NRFM30cADMQlmW6cCBA+rr61Mcz62fFZIjARwJ8iMATGxS+XG6ClB/8YtfzI499tisWCxm5557bnbvvfcG/btt27Zlkvjiiy++jvhr27Zt05Xijsjh5scsI0fyxRdfU/NFfuSLL774mvgrJD9OyzvRX/3qV7Vu3Tpde+21Ou+88/SFL3xBF198sTZt2qTFixeb/7azs1OS9Na3/pYKhULLfj/b/IQZZ+GC+e44TzrpZLP9uFWr3BgLFiwy25OcP8X+z0oDesQBP3HNMrM5ddpDZGnq9qnX62Z7rV51YzQbjYDR2HOSy/vnppDP2zFyrdfoQXGS2B0C5t07N2nTn48soE+S2HOSyxfdGN4P/6OAvyLx1nPkrLPR0VG97XfePp5PjiZHkh+l/8qRb3jHOcoXWp+vocEDZpzBvYPusQYH7BiSv6Y62tvN9lKp7MYoluzrMO+sW0kqF/zjeJditTbqxhgas/PbUNXPb7VRu0+95ufZSr3p9omc15sG3FZS2UFC3unMJXafrrY2N0ZH2c5NUezkYYXcaaUss+e1XvfPzdCIvY6GDwy7MeoN+zhNI0dmaabhrdU5nR9f+Xv/r3KFUst+zdTOXdWxmnuspnOqM+fakKTIWXVJzl+33vNByO8ahFwfmewX7OUTyX8eCnkKzZzbThSQc7LoyPNjyPl1+/jpQpmTmLKQZ3e3j5/9Qp7v/a1IyAv2zl/AM7PzbJI1WufgZr2m/7j5uqD8OC2b6M997nP6gz/4A/3e7/2eJOnaa6/Vv/zLv+h//a//pY985CPmvz346zeFQsHcROeczWne2fxIUrFo33TLZf+hq825uecCxnHUbKLTgAvROUza9BOTt4nO1f1kPps20e4PUgISYNM5N2nTnlNpajbReeOh5KCjYRP9X2M5+n6d70jyo/RfrylfyJmb6Fzevo6SnH8eksSbP39+vePk8gGbLOe15L0fVEnKOzEk/1JMMz9GztncJM2AeXfmLCDNKg64RtyH3im4fOKAe1PsbKKD1qqz2YhDNtEht1FnWlPviVdS7Lye2L3upMg5jrc5k+Z2fswVSuYmOnI20c1m0E+QTCGbrNjdRPvPKS+6TbTzgthET9jJ6eCv9zTgnpK4i+CF2UQ35ayzOOD8B+THKf9jmFqtpo0bN+rCCy/8r4PEsS688ELdfffdz+tfrVY1NDR0yBcAzEWTzY8SORLAiwP5EcBsMuWb6D179qjZbGrJkiWHfH/JkiXq7+9/Xv/169eru7t7/Gv58uVTPSQAOCpMNj9K5EgALw7kRwCzyYx/LONVV12lwcHB8a9t27bN9JAA4KhBjgSAiZEfAcyUKf+b6IULFypJEu3cufOQ7+/cuVO9vb3P618sFt2/TQaAuWCy+VEiRwJ4cSA/AphNpvyd6EKhoNWrV+u2224b/16aprrtttu0Zs2aqT4cAMwa5EcAmBj5EcBsMi2fzr1u3TpdfvnleulLX6pzzz1XX/jCFzQyMjL+aYshent7zZ8uNpt2+QHvUwQl/+PaQz4l0Pvk0TgK+DmF8wFwccAnxEUBx/E+8TgL+HTupvOxsFkc8GmUzjiSgE+JTIoBfZxPaAz5tFbvE19zAaV13OMEfAio90m7acj5DyhPlUuO/NPI3aEEfDij9+makfPptfWAT8idKVORHyUpF8fKGfOUOfVX6g3/E90rTjmmWi3g0/gbdrmeSs0fR37MXrt559OdJamUr7h9EuPTzqWwT8VuVJzSONUxP4Zz7oI+ATrgvuGFSQI+WTtxgoRciqnzkddj3pzKz7MlP3UFfaK190m4mffx3fI/TT4J+MT6yFkj5ifLHr3pccryY2Nkv1RvnTe8DyvOBXzUdCQ7IVRr/rqtOSXT2qOOgHHY6ylN/cTV0e4fp+ZURckC5qyUsxefV4lEkrKCHaNR83N9yDWQRU6lhZASZs6zTC2gskrsvN+ZBXwaedM5Ti7gGbLc7j9DDg/tNdtHB+12SYqdKhiF7m43RlYfMdv37Wj9Zx9pUPWf50zLJvptb3ubdu/erY997GPq7+/X2WefrVtuueV5HxYBAC825EcAmBj5EcBsMS2baEm68sordeWVV05XeACYtciPADAx8iOA2WDGP50bAAAAAIDZgk00AAAAAACB2EQDAAAAABCITTQAAAAAAIHYRAMAAAAAEGjaPp37SC1c2KNSqdSyvae73fz3ITWeC3n75ZeLZT+IU4szpD6fV+M5pPayW/RQUhQ5pzukbp5T01ABNTK92ssBJe+CTnDk1Y51atFJUj625yznrCHJPzUBp85dZyFzlji1SSUpydmvJ6AMpFtLPKQWq5waqHFqv+C0HlDQd7aLInNduOW6nTmWpJrTp1YLKfpt16dMAy7meMyuV90ION2dZX/9H6jZxYRHC51ujKhpv55y6ueMJLPrnAbMelCdaDn5LSTROiXbg6TOWJsB99Gxij1nASWvlQTMWerUsfVqXkuSV5K1WPCLWtfr9jiazfBap3PR4L5nleRbz6OXH0vlgOc/p5533qkzLEmdznEaDbverSRFTjH2OOCemzb8az2t2nnYe5aVpNGqfS3nk7wbo9xm17RuFPy1X3CeQyWp5tT5bjT8etT1qn3/a1T9WuJd83rM9pzz3CZJe3fvtMeR+nOWa/jH2b9js9k+tN8ehyTFqX1uuqrL3BiNpr1Wx/Y/27ItdZ5hfxHvRAMAAAAAEIhNNAAAAAAAgdhEAwAAAAAQiE00AAAAAACB2EQDAAAAABCITTQAAAAAAIHYRAMAAAAAEIhNNAAAAAAAgfzK2TNk4cKFamtra9keR3bx+JCi7562UsntU8gVzPbMr3GvxOuT+TGaWer2aTTsou+SP9hmahchz1J/HEliL7ukmHdjRM44JKnZsIvHR/LHms/ZY0kCitxnznGafo17pU17EUR2bfrn+gRcE5mz1hqpP9hadcTpEXBtOtd3ktoDrY55Y5j90maktNl6nmoNe935+UBKG/Y8N/1LSPW6fa0GpAwt6Flstncvmu/G+M1zfub2eWr7AbP9737gX2i5hfPM9nrVz289Uc1sz5p+/osy/zpLnJtLlPhjjSMnRuzfV7zbV8j9remss4rsOZWkXD4kR9qvNw7Ib43MHmvOuUdKUrlsz2uajbVuc+4nc0F7OTbPp3c7LNqPdpKkOOflA38tFHL2eRyuVfyBOPfLXBzwnFLzr4/Euw4DknnkPGcmif+MUR+1x1oql90Yjfqo36cybHcIeB4qOPkvzvn34ag+aLaPDDnjlLRr65Nme6PSOl8cFHA7kNKq3V532iU1Uvu62b/jKTdGLm+v+VzUOgenUcBDyc/xTjQAAAAAAIHYRAMAAAAAEIhNNAAAAAAAgdhEAwAAAAAQiE00AAAAAACB2EQDAAAAABCITTQAAAAAAIGO2jrRXV3ddp1op/ZaQHlmpU4d1Tjxf8bwyH8+bLZv2vS4G6Orq8tsX7TQr4G6YvlKt0/3fLt+qVskWFIS23URs9ifs1xix0ic2tuSFAWMNavstzsEFGhOSna9Qa/mtSQ1nONEATXpYm9enTqRgV3cNdBs+PVpq3W7hmMaUJ/Yu4JzzvVfrfk1IGe7Rr2uyKjR23DqpNcDzqVX9z3L/EXl1atOnHFK0vCBIbP9Ta860Y3xmtV+fcrff/0TZntn2ckpkr72YNFs31HtcGN09trXe5v8a+jZrX4urjl1otsCckbOyU1ZQI5U4hWo99dqPbPX0XDNr4Oaa/pjzTlzVnBrB4fdvzylgj3WSK3vXWkz1X75tWVnswXzO5UvtH6W8GplFwt+Udy2sn0t1wNy21jVrgM9r8vPF0nOWbcBb5flnGc7SUqcQsFRwPOft/bTkGc7pz0JeNip1vzjFGL79dZG/Rre3uuJE//5b2jPNrP9wOCAG6M6vNtsjxp+nfC43npPNt7HqM0uSQGHUcPJ94W8P2e1pn2/zyfGPiPgueYg3okGAAAAACAQm2gAAAAAAAKxiQYAAAAAIBCbaAAAAAAAArGJBgAAAAAgEJtoAAAAAAACsYkGAAAAACAQm2gAAAAAAAI5FdpnTiFJVLQKyGd2se00oFZ2mjoFvds63RjP9u81279/551ujFplzGzv6Znvxjj7v53l9rnkojeY7X3H9LkxGnW7UnpIjfIksX92k8T+sswCDpQmw3aMesWNocxujqLEDRHLWav+KBQn9uuNIv/nYU3nmpGkLLOviTQNGG3mzEnsTKokpV4fb96P2tQ2ZWr1urKo9TzVGva1Wq/757LpnQbj+Ad512pIrt43NGK2P/DwD9wYbztvmdtnJCua7aedUnBj/PUl9gv6yRN+ztixyz43o5W6GyNXsOdMknbu7DDbKxV/rMrlzeYkYI3EsX2cfBSQ753c1HAXs1Sp2vdi6blnkyMZhyTlnHwex/acSlI+Z6/FfK71Wm42Ukl73GPMZrlCQbli6zl4dtPPzH8/sH23e4wVLznZbD/+xBPdGF3ddnua+Guh4DwzRbmA69hZk5KUT7z858cYGx01273nckkaGxsy2717nySVS/4zwo6tz5jtg0P+NZQ1nH2Gcx1L0uD+fWZ7vVZ1Y0ROWoq85zZJ1RH73ElSsdxuH8d9dpOSxMnVIc//TXvem8aTd9oMeSp/zpS/E/2nf/qniqLokK9TTjllqg8DALMO+REAJkZ+BDCbTMvbNaeddpq+//3v/9dBrHeUAeBFhPwIABMjPwKYLaYlO+VyOfX29k5HaACY1ciPADAx8iOA2WJaPljsiSeeUF9fn4477jj99m//trZu3dqyb7Va1dDQ0CFfADBXTSY/SuRIAC8e5EcAs8WUb6LPO+883XDDDbrlllt0zTXXaPPmzfrVX/1VHThwYML+69evV3d39/jX8uXLp3pIAHBUmGx+lMiRAF4cyI8AZpMp30Rfcsklestb3qIzzzxTF198sb773e9qYGBAX/va1ybsf9VVV2lwcHD8a9u2bVM9JAA4Kkw2P0rkSAAvDuRHALPJtH9iQ09Pj0466SQ9+eSTE7YXi0UVjTIEADBXeflRIkcCeHEiPwI4mk37Jnp4eFhPPfWUfvd3f3dy/7BelWqt64llTs3bkBrPHT1tZntb10I3xjnnvtxsr1T9WsRbttr1Ch97dJMb46mn7L8bkqR7f/wTs/2C11zgxliydLHZXq/59Uuj2PkFiIC6sVlAcdmsbNdAjXL+L2Lk8k4Nv4D6zIlTR1Xy6yJ6FU7jOKBuXsOvfVerN+wOAfVai0W7TmCWBdSJdl5xPrFTV5rNjk90Pez8KKlRbygy5qnqXIv1hnOuJfmlxY/8XDYCLvjUqRt5oOHXUlVpl9ul4qz/v/+uX9PzI39s18Fc9zv+vP/0IXtOvn9fQB3LgFrD3V12PdXtO/xzMzhYMttLef9ajBPvnhCSd+yNVLXib7SqNb++bOTUvS6U7PmQpKJTt9e/Z0jtzv2tVC63bGvUG3pMT7vHmGlHkh+P6VumonEuhvvt2ruP3v2Ae4xqw84HK1b55bk6euz7Zb3uP1PFzvXh1SqWpDTgmaresGvPh/xqazFv9/JqxkvS9q3bzfb9g4NujAXd9rxLUv+2zWb7gQH/OF7qitKAe6hX4z6gDHixvXU+kKRGFnD2nNwnSanzXBEHvF7vcbaZ+vdQT6HYOsemAc/kB035r3N/8IMf1IYNG7Rlyxb9+7//u37jN35DSZLo7W9/+1QfCgBmFfIjAEyM/AhgNpnyt2ueeeYZvf3tb9fevXu1aNEivfKVr9Q999yjRYsWTfWhAGBWIT8CwMTIjwBmkynfRN90001THRIA5gTyIwBMjPwIYDaZljrRAAAAAADMRWyiAQAAAAAIxCYaAAAAAIBAbKIBAAAAAAjEJhoAAAAAgEBT/uncU6VeqagWt97j50t24fB9ewfcY2zZus0eQ9MvuH1gZNQex769boxzz1lttp904olujPqYX8D8mWeeNdu//S/fdmNc9Gu/ZravOHaFGyNziq0309SN4RWwl6Sk0Ga2p/miHyRvXyKR/HnPZA82ySVujLRpz4l3DEkK6ZJl9nEKAXOWd15P6pz/5/rY40iM3CBJ+XrePcZs18gaUtZ6Lr3rLA24iJLE7hM750GS6sYYJf9cSlIusvvsOuCGUCWuu31OXGSP9a8/MOLG2DNaM9s3bPDX5lDTvs662vxz12WnP0nSrsy+Vk860b/ed+2xc2Qht9KNUcqXzPZ80X9UiWJ7TuoN//xn8s+NlyOjgPcmik7KLzrPN5KUzxfsGEaurlVrulP/7h5jNtvdv135Yus5yOXs81Qq+Wu/tsd+vtt4991ujN7jjzHbo9qwG6Ojs9Pu4Fwbksx7yXiY2Mnlzv1CklLnubrpPOtIUuWAPe9xo+LGGNzj3zRqo/bzfVzz9whZ056zRsA+I/HOTd5/hqzl7PzXbPjjCMlt7rNqzc/D3sNqQwFz5jyH1iut79Pea/hFvBMNAAAAAEAgNtEAAAAAAARiEw0AAAAAQCA20QAAAAAABGITDQAAAABAIDbRAAAAAAAEYhMNAAAAAECgo7ZOdK6tU/m29pbtSWLXc6zUxtxj7B+y68Q99eTjboxiufUYJWnJkl43RnubXePvjDPsOtKS9NP7HnD7/GzzU2Z7badfv+3W7/+b2X7J617vxujt6zPbGwH16kKKHnu1k9OQnyGlTo1np26iJEVOFz9CQD3qgBqPIX1yOTsl5J262SGHSZ06q5LcQuBO6UW3fS5ob+tWvmCcj4V2jiyVFrjHqDWOrF63JCl2aq0H1Bb1rpIk83PXlh12/pOkk5c3zPaODn/tjtSqZntFdrskDTftPvmiXwQ6Ls53++TLg2b7rn3+hbS0165ju2j+aW6Mjq55ZnuzaZ8XSWpU7TVQbQTM+4hfO3asYtegjZz6w5LUu2CJ2d7V7tT9ldRhPB9JUj5uff1Xxvw6urPdfzzyUyXG/awrb19Dne12HW5J6nFq3o5t3+zG2J7Y56JU9M/V8Ig91pBnqpD7cmdbh9keR3694t17d9vjqPvXuldXPi761+DgoJ37JKlRtceSBjzKxJnzDJmF1F62x+HV75akuGkfp5kGrJGAPJw5rzcKelZ1ak2HlD33ak0bryUNObE/xzvRAAAAAAAEYhMNAAAAAEAgNtEAAAAAAARiEw0AAAAAQCA20QAAAAAABGITDQAAAABAIDbRAAAAAAAEYhMNAAAAAECg1pXoZ1ixvVvF9vaW7c2GXQx76bIV7jGWLl9mtmfyC27v2LHTbA+p2f3oo0/YMZp+Aftntz/r9ml4BcxTv5D6M8/Yx7n1+//mxrjo4teb7T3zetwYad0vDJ+5Rd39ou9RbF8iQXXjI2fem3U3RqNeNdvjxL+UA+rTK46cAvWpP+/1mv16qhX/9eYLebM9ie1rotHwxznbLetdqWKx0LK9vshed8004Geo+aLZHIcsKudntU0nL0lS1rQvtLEhOw9L0sDwz9w+g1X7OGnYVWQfY8SPcKBuz8lQtfV5P6jW6HL75JwcWK3vcmPs32tfz3H0TTdGtXq22b6jf8iNEaf2xPYd0+HGmNfm3wPnd9txtu/c78bY/DP7PnrqiS91Yyx2nnEK+VLLttHRUTf+bNdeTpTkW98ncrF9jfX0+NfYiaN2PhiO/BgP79hnthdW2DlYkuqpd88dc2Mo4JY57Dx3RwH5sek8H1TG/LHm2+0cm5Rar/2DDoz6idibkihnP6dI/nNXWvefhzLnXh3wGCo17NwWO/ccSWoGbGgy5xkyyvn7GS9GMe8/u6Spcy83njvSKGhGJfFONAAAAAAAwdhEAwAAAAAQiE00AAAAAACB2EQDAAAAABCITTQAAAAAAIHYRAMAAAAAEIhNNAAAAAAAgdhEAwAAAAAQKDfTA2il0WioYRQHbzTtot+xU2hbkoZHDpjtjz22yY2xc/dus73pjFOS2ts77GPs2unGaAQUbK9Wamb7WGXUjdFWtIvLP/74E26MKPqu2X7+qy9wYyxevMTtU63YxeWdeu6SpDRp2jEC1lkzs/s06lU3xtiYvVaLxXY3Rhb7l3vTW0cBNehrToxGzT4vIQfK8vZraab+9TDbLVnYp1K52LI9iu1rNcn56yGfK5ntcez/HLaZ2TmwbuT5g7x8v39PxY0xPOyvidE0MdsL8sc6VrFzxoaH7GNI0lix02wvlcpuDDW3u13i/JjZ3tXhr5H9u+0++XiHG6OzYM/ZScfb8yFJ5dyw2b58xUo3Rnf3r7l9sniZ2b5z+6AbY2zMzvkvOfU0fxxNO0emUetrJgtJ5LNcV2eHcgUjBzrpYDjgHlJL7XWbi/xrvT5q566xYTuPS1LvvHlme6ngPx/IeS3PsfN9yPNuV75gtmep/2DmHSWt+3k6CjhOzhlrruSfX3nPiIk/jrzz3K2AZ9nMeeBNAh6Ic84zhSRlzvNu5DwPSFJSdp47Cna7FPZ830rabGqP/PuWdBjvRN9111269NJL1dfXpyiK9M1vfvOQ9izL9LGPfUxLly5VuVzWhRdeqCee8DdWADDbkR8BYGLkRwBzyaQ30SMjIzrrrLN09dVXT9j+mc98Rn/zN3+ja6+9Vvfee6/a29t18cUXq1Lx3ykAgNmM/AgAEyM/AphLJv3r3JdccokuueSSCduyLNMXvvAFffSjH9Ub3/hGSdI//MM/aMmSJfrmN7+p3/zN3zyy0QLAUYz8CAATIz8CmEum9IPFNm/erP7+fl144YXj3+vu7tZ5552nu+++e8J/U61WNTQ0dMgXAMw1h5MfJXIkgLmP/AhgtpnSTXR/f78kacmSQz/0acmSJeNtv2z9+vXq7u4e/1q+fPlUDgkAjgqHkx8lciSAuY/8CGC2mfESV1dddZUGBwfHv7Zt2zbTQwKAowY5EgAmRn4EMFOmdBPd29srSdq589CSTDt37hxv+2XFYlFdXV2HfAHAXHM4+VEiRwKY+8iPAGabKa0TvWrVKvX29uq2227T2WefLUkaGhrSvffeq/e+972TC1YdlpLW9cRS2fXK4qJfR9P7xMctT29xY6w89ji7Q0Ad1eEDdl3Jgf373Bj1hl/TsF61+3j13SQpbrPrsyWJ/3qfeNyuv502/dfy384+z+2z9Jg+u4Nfrk4jo3Yd1XrTr/EcuQfy58wre1ip2eOUpCj2axp6tfWi1K+/mDovN0r8tJM6tRWbTklLr30mTGl+lLSwq0dlo55i5NS4TPKta0wfVMg5NT3dCFLT6RVSN7xSt2vc14f9OqjOpSxJGhi2x9rT5hefbO+wL4CTjvXn/YmhVWb7zp1+8soZdYIPasuNmu31fEBN9+aA2Tyw36/pedGrDpjtJ62016EkldtWm+37hl/pxqjUT3b7RE6t8GNXLHZjxM59st6w17vkP7xlRiK22mbKVOfHymhFOevG2bCv9VLAZnzzgT12h6Zdu1ySYieJ7tvlx2gr2/mvs8fPj9XMf+5SZA825H5QqdrPTHHOfx4aHbOf3RsN/xkjH1DDu627zWxPCn5eqjv7jGrFr72cd2pJ5wP2GYWik4cD6iqnAW+7emHimr/O4jb7HtnM/HNXqzvrzHgtzUb4A+SkN9HDw8N68sknx/9/8+bNeuCBBzR//nytWLFC73//+/Xnf/7nOvHEE7Vq1Sr9yZ/8ifr6+vSmN71psocCgFmF/AgAEyM/AphLJr2Jvu+++/Sa17xm/P/XrVsnSbr88st1ww036EMf+pBGRkZ0xRVXaGBgQK985St1yy23qFTyfxoNALMZ+REAJkZ+BDCXTHoTff7555u/9htFkT75yU/qk5/85BENDABmG/IjAEyM/AhgLpnxT+cGAAAAAGC2YBMNAAAAAEAgNtEAAAAAAARiEw0AAAAAQCA20QAAAAAABJr0p3O/UNI4UZq0Hl7sDT3zK4c30tRs7+ye58bo6ekx2wf2D7gxPNWxMbfPgeFht0+m1p+KKUlt5bIbIzI+WVOSUrtZklRu7zDbR0drboy7frjB7XPG6WeY7cedcJIbo1KxC7bv37/HjdHWbs9rqejPe61mr9VqzR6nJMWRf00kVgV6SZJfhD5zFkFkXNfjR0nt4+SccY6NjbrHmO1KxYLKpWLL9ijJm/8+zhXcY+TydowQjaa9drNcQNKo2eshSv2fB9er/vqvNO21ud9PxVKubjYvWeznt6HMnrP2fLsbY2TYH2y13m22J7Gf39Jeeyzbt7ohdNzyE8z2Vcee7Ma4++FzzfbdQ/75j7In3T7FxL5ucgU/vxXzdoxS3i/p1MzZaz4y8n0zbbjxZ7tSkimXGLnFWQ75hV3uMbbtHTHbh/YdcGNkBftcZwG5bd8eexztbZ1ujI42/3m3nlbM9jjvX2PtBft5J3Xu/ZLUbNr3jHzAM4Y/UqlUaH1/laQ0IMrIqP0sEvBYpnanxFtHm58v8rE9J3EucWM0mv65GXaeRWuj/rPq6Ji9n2nW7PUuSbmK8wxpPNs0GuH5kXeiAQAAAAAIxCYaAAAAAIBAbKIBAAAAAAjEJhoAAAAAgEBsogEAAAAACMQmGgAAAACAQGyiAQAAAAAIdNTWiW4mbWombS3bY6fmcTOgzld7m13f8ld+5VfcGMNDQ2b70NCgGyNzai8v6e11YyyN/J+HZJndJ2SsB4bt17u0d4Ubo+8Yu08usmukStLIqF8X+8CwXaNx/16/BmpHl11fsbu7x41RLNo1/CK3NrOUZnZtvUbDr/GXOrVnJanetK+bLKD+uls6seHXcB4ZtusAxon9eisBtdVnu3qtplrSeu1ksX0uk8TPkc26c4sIWLteDfNqza+bPDpm1yetNux2SYoa/ljHRuzakgv6/Nrag8P2vHZ0+NfQ0uoms33H/ovdGE9tecTt01m0r8W04V9HixbPN9vbu+xaq5L0tzfZ1/uvrjnLjTE00m+27x8ccGPUZNf4lqSOsv3c0N1h196WpAU99pxFAfVY80594cSog9p0njnmgqWLl6tQbH29lop2veKTTn6Je4zokg6zvV7x11O9bueLRsNfC+U2+xrr7rbHGdoncncM/roqFu2xxgH3lEz2/T8khgKugVzAc7VnrGHf33IhNa29fOAcIyRGzblPS1ISt84pB6Vd9joaHPBrPO/q32G279j5tBtjZMjeIyRG3exatap7vn+3ewyJd6IBAAAAAAjGJhoAAAAAgEBsogEAAAAACMQmGgAAAACAQGyiAQAAAAAIxCYaAAAAAIBAbKIBAAAAAAjEJhoAAAAAgEB+le8ZkjYqatZbF1SvNlLz32cBhdTjODLbzzrjLDfGE49vMtu3Pr3VjVGpjNodspIbI87Zr0WSalX7OMPD+90YubxdbH3JksVujJ55djH2zT971I3RVrJjSFIztYvLb3rsETfGvJ4eu33hEjdGudRutudyBTdGsWCvgfa2hhuj4VwzkpQ5yyj1OkjKUvvay8X+tTmvy349UWT//G901Lmm5oBGo65Go3WOlOz1nxStf/uctFEz25tNf03Vm/a5rNYrbgxlVbO5UO50Q1SH57t9RgZ3mu3NRf6cdXfYOTIp+rlr6XJ7fS941r+v7Np+kttn05Y9ZntSOODGOGG+Pff5xP9ZfbO2z2z/wY//2Y3R07HMbN+1b5cboxaQ33q6F5rtxyz2c3HivX/R3e3G6IztGEliPN45OXouOOHkc1Qql1u2N5v2HCxZfrJ7jLhg37sH9w24MebPs8/1K19+nhujVLDXbcjzsDL7fiFJdWfOQo5TcOYsZGV6l2nAI0YQZ1q17dl+N8aDj9nPs6Vi0Y2x5XE7Rlvs35de84o1ZvvI2LAboxHwvmvb0l67Q9L6mjyoHtmvx5tTSdr0n/9ptueNOauMhT8/8k40AAAAAACB2EQDAAAAABCITTQAAAAAAIHYRAMAAAAAEIhNNAAAAAAAgdhEAwAAAAAQiE00AAAAAACBjto60Yp+/tVCs+HVkfXrpim26+K1l+16n5LU1WnXAO7osNslKZe3i9HVan4d1Wef3e32UWzXdO3q6HJDJHn79Wx5xq9fev8D95jtzaZdm1aSuuctcPvs3mvXfK1X626MoQNDZvu8Dr8+7Stfc7HZvmzFSjeGt5obTk1sSUr9kr5u3d9mQK1p7ydzSc6/NgsF+9ord7SZ7WlIXcxZLo1jpUat2CS203sUUkzTqUWbBSyqKLP75K16tuMx7Gu1ozTPjdEoHu/22b1zh9k+v8e/zo45zs6Rxyz061V399i1Noslf84W9Dzp9rnnPrvO/Q/v3u7GiGr2WLrKfs3jtuIqs/2xzfZ5kaSntm8x28cqAfWbc3YNW0lqK9n346pTF12SRupjZnth1B+HVedUkhS1Pi/1mn/vm+2a+TY1863vE3Xn+e+xzc+4xxgZsWvrbn76KTfGymXHmO3HnXyCG2Nep/3sVh3znyELOf89tULermncCLiljNardgznGUTy60R79xwprJZ05PR5cov/vPvwI3a94q7Iv9a3b7OP07fEfx5+9hk7l+fK/vmvjPj1k/cP2tdErs2/H5SWLjPbk7jDjdHZYc/Jzmc3t2yrVuz8/Ism/U70XXfdpUsvvVR9fX2Kokjf/OY3D2l/5zvfqSiKDvl63eteN9nDAMCsQ34EgImRHwHMJZPeRI+MjOiss87S1Vdf3bLP6173Ou3YsWP865/+6Z+OaJAAMBuQHwFgYuRHAHPJpH+d+5JLLtEll1xi9ikWi+rttX8dDQDmGvIjAEyM/AhgLpmWDxa78847tXjxYp188sl673vfq71797bsW61WNTQ0dMgXAMxVk8mPEjkSwIsH+RHAbDHlm+jXve51+od/+Afddttt+vSnP60NGzbokksuUbM58Yc4rF+/Xt3d3eNfy5cvn+ohAcBRYbL5USJHAnhxID8CmE2m/NO5f/M3f3P8v8844wydeeaZOv7443XnnXfqta997fP6X3XVVVq3bt34/w8NDZEEAcxJk82PEjkSwIsD+RHAbDLtdaKPO+44LVy4UE8+OXG5jWKxqK6urkO+AODFwMuPEjkSwIsT+RHA0WzaN9HPPPOM9u7dq6VLl073oQBgViE/AsDEyI8AjmaT/nXu4eHhQ34quHnzZj3wwAOaP3++5s+fr0984hO67LLL1Nvbq6eeekof+tCHdMIJJ+jiiy+e1HGKxbJKpXLLdq/ou1uNXVIzdYq+p3419u4eu+j30t4lbgxPpeoX/i6X290+oyMVs72Rtv67o4PqtbrZXnPaJUlx6/MqSbUxf94ffeRxt0+lZr/ectEehyTNn7/QbG82/Nd7+23/YrYX2zrdGAvmzTPb+/qOcWPMX2gXn5ekUrFktmf+ElGcJGZ7e94+hiTV6zW7vWYfo17zr5np8ELlR0mKolhR1PrnoHHOnqMo9nNklqVme5r516oXIx/7P8vNldrM9ma+4cZIlpzj9tny9ANme7xlxI2Rlew+i5fucmOkkX2N9HT1uDG6ygW3T0/Ps2Z7Z9kf68NPzjfbo9TPb5Kdq4vy1+rm7fZr6Vno34vbO/yxljvtPjnnupOkpvN6ak07/0nS6Nio2V5PW193Y6NzPz/uOzCsQr31DSvzclfk57bIec5ceoz/a+W5sn2t/+jen7oxliy213ac+tdPd8DaX+w8z/ozJlUadq6uBzxTJbF9T4kD7ilJQJ9m3R5r0mY/l0nSqhNOM9tL9kuRJPUutD+tvi0gX+zt32225wN+gyPXZu93JKlRted1YL/9QYGSNN+Z1+55i90YO7Y+bbb/9Mf3tmxrOM+fv2jSm+j77rtPr3nNa8b//+Dfolx++eW65ppr9OCDD+rv//7vNTAwoL6+Pl100UX6sz/7MxWLzqYXAGY58iMATIz8CGAumfQm+vzzzzd/gve9733viAYEALMV+REAJkZ+BDCXTPvfRAMAAAAAMFewiQYAAAAAIBCbaAAAAAAAArGJBgAAAAAgEJtoAAAAAAACsYkGAAAAACDQpEtcvVDyuZzyudbDq8uuUJ42m+4xjEoLz7WniRujXLaLjy9xitNLUpRFZvvwyJgbI0v9n4fk80Nm++DgoBujmTlF7mW/Fknq6rILqfcds9yNkeT843g9TjntTDfGiuUrzfbtz9oF3SWpf/tWs33n9n43xpaf2TEeeeRRN8bC+fPdPquOO95sX7lqlRvjmL5jzPbueV1ujNGREbN95/ZnzPaxMf+ame3iOFactL7uvfWfGPn1oDS182hWt/OBJDWdXC0nD0tSnNm5OIpLboxie6/bZ3HfW8z2p5/9P26MOGfn2e5u/95UyNvrN6vvcmMo8c/N4J4DZnukHjfG0lUXmu279rkhtPXpTWb74MiwG2PxIjvvdPX0uDG6exb5fbrt+1epVHZjRKm96McqFTdGWq+a7YVa6zVUGfPjz3ZjY2NqGg95Wb1h/nsjtY6LY/s8xpEfpDpij+NnT9rXhiQ9s3WL2X7M0mVujN1x3u3z1Fb7vltt+LmtVrf7jFXtdS1Jzcao2X7gwH43xuA+v49y9pws6V3hhsg797cD+/xcvqin02xfkPP3KtXMnteejgVujKjqH6fmXFejNf++tOWhx8z2A6P2+ZekPc88ZbZ3dLae03qt5sY/iHeiAQAAAAAIxCYaAAAAAIBAbKIBAAAAAAjEJhoAAAAAgEBsogEAAAAACMQmGgAAAACAQGyiAQAAAAAIdNTWia43U9WbreuJjY3ZdcIqFb/OVy5v1zxrNvwCpnFkx2hr82tGDh+wa3XGAT/qKJWLfqfYrmmdpnZ9N0mq7tlttjcDir52d9njWLFsqRtj2TF+/e0uow6cJC1ZatcVlaTImfyVx650Y+QSe04yr46upH17B8z2PXv3uDEG9tnnTpL6++2a1VudOpGSNG++XUf1mL6Aer1L7PPr1d9sZn69ytmuqcysgxql9rqq1f0c6V3N9UbdjdFw+qSRX/M9lxTM9lghr8XPTbnOk832zuXvcGM8+J9fN9srY35d+GXHDprtUeTnjHkB9dh37rXvG1W91o1RKtq1UtvK290Yi5b0me3tzj1DkvKR/ThTLPi1xMslv09Hmz2WxKktK0mZcd1KUiOgTmnqrOdm1Lq92QivgzpbLVu0UKVyW8v2plNnOwnJS5Gd2zInBz/XyW5OA2pNN50+1RG7dr0kDTn1fSVptOrUAK74MbzXMxxQA3jswIDZvn+Pn2OfffoJt08uZ993Xvma/8eNMbBzp9n+2P0/dWN0LJxvtsc5/962cPFCs717j79GlPnXROrUCk9jf9vZdIq0tznnRZJKHXaePuGMs1u2VStjbvyDeCcaAAAAAIBAbKIBAAAAAAjEJhoAAAAAgEBsogEAAAAACMQmGgAAAACAQGyiAQAAAAAIxCYaAAAAAIBAbKIBAAAAAAjkV72eIaOjo5JaF/au1Wvmv6/Vq+4xoqRkd4j9wuKFol30e+GCBW6MujPWkTG/CHouoNh6OXJeb0+XG0NqOO3+z2UWLVxstp9w3HFujAWLlrp9Cnn73ESxP2dpeuSvN8l5l5k/jvb2TrN95cpj3RghxxkeHjbb9w/6a3FH/w6zvX+n3S5JW57eYrYvW2a/3mrVv/5nu2p1zFzDcZSY/z6qpO4xMmfNVJw8LEmNtGmPw8jzB+WSutn+8I9/4MZ4eru/dpcf12O2t6cjboz6gbLZ/szopW6MoS32cfKRP++lvQE5Ml5kH6ew0I2Rc/JoT3uHGyN21kitw4+RS+z1XnLzsJTL2TEkqVQomu2x/BhN5/XWE//aVGbPe2LMR5qzjz8XLOjoVLmtrWV7I3WehzL/HKSp3cc7z5LUbNjPGPWAU9VI7Ry6a8DPfQNDB9w+iXdPifxcPjrmPO9W/Ht3ozZmtvcu9J+7e3L2PUWSnnaeQ3pif6zPbN9sto/0b3Vj5Jx8PxawVmt1e86Us5+XJSlfbH09HeQ9ZUYB+bHc3e308J7LpVrdXotN435Qa/rr+CDeiQYAAAAAIBCbaAAAAAAAArGJBgAAAAAgEJtoAAAAAAACsYkGAAAAACAQm2gAAAAAAAKxiQYAAAAAINBRWye6Wq3adQ6d+nxefVNJqozZddOSxJ+eJLHriXV2+vUt8wW75m3eqXcsSf39290+Q0N2rcB6w689mndqZB6ztM+Nccqpp5ntCxf4tUlLZX9eaw27DmDW8Gv8xc4aqATMWeRcZl67JOUL9hpo1v26eXHAcTo77VrhXd3tboxjltp1wJvNs9wYu3ftdGLYr7dSqbjHmO0qtarMkoupnQOzgPrMWWbPc6MRUAfVScWxk0MlafOTT5vtd/3wfjdGvqvH7aP6XrO54eRQSerssevPbt+zzY3x0gvfYLYXOvx6nVX/Fqjc2G67Q2LXvJakSPYayAfcRzvLdl5JnZrIklRwcmQhYBxRQL3VXOK89xAw1kbDqfFcyLsxvGecOG6dHJrNuV8nenhsRA1jjurOPSSVvxaamZ27AspEu7WkK1X/OWV41O4zVvWfU6p1v25yktr31UbdP07FOU614sfwjrNrm/38IEnDu591+wzt22O23/Iv/8eNMerU6B4d9e8p6T475xRK/v1gbMiuz7zrWb9edaHkP3c3nRrLtZq/zjp75pvtUdHPj0nezveFfOv7dL0a/vw4qXei169fr3POOUednZ1avHix3vSmN2nTpk2H9KlUKlq7dq0WLFigjo4OXXbZZdq501/QADCbkR8BoDVyJIC5ZFKb6A0bNmjt2rW65557dOutt6per+uiiy7SyMjIeJ8PfOAD+va3v62vf/3r2rBhg7Zv3643v/nNUz5wADiakB8BoDVyJIC5ZFK/zn3LLbcc8v833HCDFi9erI0bN+pVr3qVBgcHdd111+nGG2/UBRdcIEm6/vrrdeqpp+qee+7Ry172sqkbOQAcRciPANAaORLAXHJEHyw2ODgoSZo//7nfX9+4caPq9bouvPDC8T6nnHKKVqxYobvvvnvCGNVqVUNDQ4d8AcBsNxX5USJHApibeIYEMJsd9iY6TVO9//3v1yte8QqdfvrpkqT+/n4VCgX19PQc0nfJkiXq7++fMM769evV3d09/rV8+fLDHRIAHBWmKj9K5EgAcw/PkABmu8PeRK9du1YPP/ywbrrppiMawFVXXaXBwcHxr23b/E8uBYCj2VTlR4kcCWDu4RkSwGx3WCWurrzySn3nO9/RXXfdpWXLlo1/v7e3V7VaTQMDA4f8JHHnzp3q7e2dMFaxWFSxaJdNAoDZYirzo0SOBDC38AwJYC6Y1DvRWZbpyiuv1M0336zbb79dq1atOqR99erVyufzuu2228a/t2nTJm3dulVr1qyZmhEDwFGI/AgArZEjAcwlk3oneu3atbrxxhv1rW99S52dneN/o9Ld3a1yuazu7m69613v0rp16zR//nx1dXXpfe97n9asWTPpT1WMokxRlLVuV+s2ScondmFxSWo07BiFnB9DSs3WKLILj0tST7ddWLyjw26XpKV9x7p9vALiP/rRXW4MZfacnHPOuW6IefPmme179+13Ywz/QkmMVrq6usz2OLHPvyTFkf1zplJSdmNkkX2ZNaoNfxyZvY6ivF983n+1UiFnx2nURt0Yaa1mthdLHW6MVStXme1R3DTbR0f9cU61FzI/SlKj0VSj0Xoe6nX7PGSpvyKixF7/ceznt8g5TubkYUkqt9trJivZ17ok7dy11z9OeanZnqY9bowTSva7Yv/68JNujGer/8dsf9c7fseNEcX+/Su2b1/KEv8Rodm011kakDMiJ8+25wtujILTJw54LVkzIBdHzqQF3POz2H69aeocQ1Lm3BMKxuttBszFdHghc+RoraFmUm/ZXk+ddZv55yBxrrGApSDvzpzP+9dxV6edc3o6290YsfzBNlP7+qjWW8/3QTXjniVJVef5QZLGhobN9uF9u/0Ydf/8ds5fZLZXnedhSSrMW2i2l+bZx5CkqFiyY7T5z1RF53m4WPbXSC5gLUbO/bzNyVuSVO7sNNuLZf/1xs4esFhsfb+oVcbc+AdNKpNec801kqTzzz//kO9ff/31euc73ylJ+vznP684jnXZZZepWq3q4osv1pe+9KXJHAYAZh3yIwC0Ro4EMJdMahOdZf47BqVSSVdffbWuvvrqwx4UAMw25EcAaI0cCWAuOaI60QAAAAAAvJiwiQYAAAAAIBCbaAAAAAAAArGJBgAAAAAgEJtoAAAAAAACzUyxwABjIyOSVV/UKTXW3WXXGZOkUsGuiZs5NaAlKXbrBPo/p8icPs3Ur71XKrW5fYoFu5ZgpVZ1Y6z+lZea7YsWL3FjVCt2jb+Odr9eXRJS89P5JNBGw695l+TtOUu8mqGSUmccSeLXzas5NX/rzYCalgE1/qpV+zi1MX+NFEp2TcMk8Wuxbt++zWz3ShyPjYXX+ZutcklOOeM6qDv1ODOn1rYkJU6N84KTU55jr800YO0uXXaM2f7bv/MWN8Y/3/wvbp+nnt5utofcMAdG7PtKveHXdN/xzB6z/c4N/+7G+PU3vsnt00ydGs5OfVJJajh1ohupn3fkLMU478985NQsj7yDSEry/v06iZzXE1AcOMnbayDX8O/5XhKMjWM0A2qzz3ZxkleStK4FG3nPEJE/R96Zzpwa0JLUjOz7YZL5azKfs9dkrerft6PUX7e5gv1sFhf9ayyu2ms7yvk14eVcgyedcbYb4tSz/D45Z17zAfc/L0YUss6c2sqRUxNZkrKcvd69uvOSlDj17SUpcfJfFPDerZcfZVzXBzXq9lpMs9btVWdv+It4JxoAAAAAgEBsogEAAAAACMQmGgAAAACAQGyiAQAAAAAIxCYaAAAAAIBAbKIBAAAAAAjEJhoAAAAAgEBsogEAAAAACORUm585tVpNiVFAPE3tAuVpmrrHaDbsou/tbd1+jNQu6F0uBxSOr9vjqNTsdknKBRRbf2brZrO9Wh1zY6xatdJsrweMtemcu2bTP3eVSs3tEzmF4ePI/xlSozFix0j8sdYrzpzE/mXYaDqvN7PnVJJSp/i8JOVy9joKeb3Vyqh9jI52N4aiyG53rjtl/jhnu2KpTcVSqWV7FHs5wV//uZydvwqFgPzmnMrMO5cBMeZ1L3RDXPGud7l9brzxJrP9/ocedGMsOfFks71U96/VheWi2X7Pjze6MZpVPxdfetE5Znubsb4Oqtftk+OvQylzcn4UcD1HOXs9J7GziCTlkrx/nMh+Pc7tTZLUbNrnpuHlP0mZ7DmJ09bjrKdz//2TKMoURa1PRpzac+w9p0hSM7LPQdrwY2TOuY7lx2g07PZ604+RJP6ac4fiXBuSlHdSSpz3r8Fiuc3p4b/eOGCssTMnIecmdp7NrDV6UOL1yfzr2VnuagQ8d6cBx/HWc0Bqk3vDD5n3xOlj5MDY2Tsc0je4JwAAAAAAL3JsogEAAAAACMQmGgAAAACAQGyiAQAAAAAIxCYaAAAAAIBAbKIBAAAAAAjEJhoAAAAAgEBHbZ3oZpapadRXq9ar5r/PAurm1pwY1apfi1ixXSe1GXn17KRCYv8sIx9QuzKgFKf+46H7zfacU2dTkn729NNme1d72Y0xr9MbbEAhuUbF7RI5daBzBX/59+/ea7Yv7vbr5JZzdmHEes1/vUnerhtbKPrrLEn9OuBe+cV8vsONsfmZfWZ7s+kv1r5lK832sQNDZvvoqF2rei7o6OhWua319VavO/kroGBjzlt3iX8NZU6N3zSopqfdnnfqWUtST4+fR9de+Ydm+5euucaNsW3foNm+eOXxbowD+7eb7aeffoYb42ebHnf7bFth5+tTX32SG8Ob+STnz7uX89OAWuINp65vM+C+0sz89dx0aqV69VifG4sdI8r7NVtzBTtG2arh7jy3zAW5JKeckZ+acoorN/0T6dUazvJ+bou897IC1lPq5NhS2X8uCzmO1ymX+UFSZ84asXNeJDWdLl4N9Z9HcXvETj31kBrPsdMnDpj4xHmWzQLG4SWmxLvJSqo3/DlLneNEAceJVbfbA2pae2vAei7JmgF7v4NjCe4JAAAAAMCLHJtoAAAAAAACsYkGAAAAACAQm2gAAAAAAAKxiQYAAAAAIBCbaAAAAAAAArGJBgAAAAAgEJtoAAAAAAACta5EP8OqlYqiqHVR7kKpaP77ro5O9xiNWslszxXzbozIKYJeyPkxqpURs/3AyAE3Rn9/v9vn/p8+aLYvWrzQjXH7bbea7aecfIIbY3lvm9meZn7h+HmdHW6fzpJ9frPEX/5Jzi4M30j9ou+jo4Nme0fZfy1Z5MxJvenGiDTq9hkbdg5TXODGGBmxj1Nv+j+7K9Ts6+ahh+4326vVqnuM2a5YbFepWG7ZnsvbOdJdU5JyOTtGbF8ez2k6a9PJoZIU5+w+uYBrOU78XNzTZt83Vq483o2x6Wk7R6rS48ZIY3vOtu/Y68Y4/ZRVbp8H/vNndodFm9wYLzn9DLM9rY65Mbyc3wy4jzbrdi6OAt4zyGL/OInTp8O570hSseCs17TuxmiMVcx267Jq5v371myXKFZinHMvH0TOvV+SMuMZVZIy+TlWzjNEMySEjvx8ZmnAgTwh9wPnMLmA+0GUNMz2tBHwWgLGmkvsOIWAe0qS2AdKAtZI7KyzZsAzc5p5z7J+jKbseX+ON7H+xDsvV3EWsN6d1xsb95S44eff8b7BPSWtX79e55xzjjo7O7V48WK96U1v0qZNh95kzz//fEVRdMjXe97znskcBgBmHfIjALRGjgQwl0xqE71hwwatXbtW99xzj2699VbV63VddNFFGhk59J3UP/iDP9COHTvGvz7zmc9M6aAB4GhDfgSA1siRAOaSSf069y233HLI/99www1avHixNm7cqFe96lXj329ra1Nvb+/UjBAAZgHyIwC0Ro4EMJcc0QeLDQ4+97ee8+fPP+T7//t//28tXLhQp59+uq666iqNjrb+G8lqtaqhoaFDvgBgtpuK/CiRIwHMTTxDApjNDvuDxdI01fvf/3694hWv0Omnnz7+/d/6rd/Sscceq76+Pj344IP68Ic/rE2bNumf//mfJ4yzfv16feITnzjcYQDAUWeq8qNEjgQw9/AMCWC2O+xN9Nq1a/Xwww/rhz/84SHfv+KKK8b/+4wzztDSpUv12te+Vk899ZSOP/75n2x61VVXad26deP/PzQ0pOXLlx/usABgxk1VfpTIkQDmHp4hAcx2h7WJvvLKK/Wd73xHd911l5YtW2b2Pe+88yRJTz755IQJsFgsqli0y6gAwGwxlflRIkcCmFt4hgQwF0xqE51lmd73vvfp5ptv1p133qlVqwLqUD7wgCRp6dKlhzVAAJgNyI8A0Bo5EsBcMqlN9Nq1a3XjjTfqW9/6ljo7O9Xf3y9J6u7uVrlc1lNPPaUbb7xRr3/967VgwQI9+OCD+sAHPqBXvepVOvPMMyc1sHJbm8rlcst2ryx4tVZzjzE0uM9sbzT9wuJZahf9TjK/aLdXN3yk1nRj3H3v3W6fRrNito+ODLsxcrE9llxA4fj+HfvN9iTvL8s9e8fcPgsXLTbbC4l/fucvWGK2b+vf7sY4sG+n2b58xUo3xr6h3WZ7ZWzAjVEZq/p9KvZ6XTBvvtkuSfsHRsz2pOCf3zS119meXbvM9lrA9T/VXsj8KEn5fKK8ca3klLcDJJF/kMzuU6v6a8oJoXzsr4dmZn8GZpol/jgaTqKVFDXtdbf6nJe5Mbb3P2O2P/H4JrNdkvJtre99ktTTM8+N0VX0c+SB6kKz/dZbb3djrDzlLLO9XvfXWcOZ92bkfwZqoWC/G9ldKrkxnNu5JCmXK5jtpYD8ljjXXq3uv95Kw75/jVRa58CxUX9tTIcXMkfmkli5xJpHZ10GpMfUWTAh60mRfaDMuRdKUuQMNgt4LvNihEgj/ziZ88CbNgMmzZmTOGAcUeTfM/KJ3Scf8tHMzj4iChlrbB8oF/BaUmedNZ0cLEnFgDwsZ6z+7k2KnIeGgJer2Hm9uVzr56M49fcG43GCe0q65pprJEnnn3/+Id+//vrr9c53vlOFQkHf//739YUvfEEjIyNavny5LrvsMn30ox+dzGEAYNYhPwJAa+RIAHPJpH+d27J8+XJt2LDhiAYEALMR+REAWiNHAphLjqhONAAAAAAALyZsogEAAAAACMQmGgAAAACAQGyiAQAAAAAIxCYaAAAAAIBAk/p07hdSsVBUsdi6ruPAkF1r+Nltdm1eSerp7jLb5/XY7ZLUv9M+zuYtT7kx2sptZvvWbf1ujL177ZrXkrS0b5HdIaA8X2eHPyeetrJdZ7MeUFzRq7MpSXFm170b2u/PWWXU7tP0S96pp8s+vwcG9/rjcGp412t2DXBJigOK65Vy9rwODdr1qiXpZ1vt2tltJbsGriQVC3ZN1575dp3cavWFrxP9Qlu8sE/tHe0t271a2fWAxVtv2DFCKotmzrorFO3r47kDOUdy6khLUpbzb3dlp9Zw33nL3BjnrD7PbL/rrjvdGF+5/jqzPRsecGPkin5N9+HUXgOPPbXZjfHgQ4+Y7ceddKobQ7FdlzMJqCVeLra+FiSpq8uvE62AWuJVZ87GnLqwktQwajhLUq3m12xVYt9H29pbt0ch9V5nuyg2a9amTp3tzCtwLymO7T6J0y755yJk7cfOp56HZOpmSD1qZ9mElMWuNZ0awJF//eQye+2nzrOfJOUTf17zRi1hSUrceffnLORKjJx61SFrte7U3w6Zj1zAeo5je6xRQAx58xp0XTmHMPYZSUAt6/GhBPcEAAAAAOBFjk00AAAAAACB2EQDAAAAABCITTQAAAAAAIHYRAMAAAAAEIhNNAAAAAAAgdhEAwAAAAAQiE00AAAAAACB/OraM2Tfvu0qjZZatmexXQR9dGzIPUZXR+v4kjQ46McoFOxxdHfNc2M8u6Pfad/uxpjf0+326V2yyGzftWeXG6One4HZPnBgzI0xUrGLvh8YGnZjDAwOun26u7rM9vZSwY3R7qyR4044xY0xMjxitg8MHHBj7N6132yPk6YbY+WqE90+23c8Y7Y/9NAjboxarWG2L1u23I2Ra9rF7rPUXkNe+1yQRUVlUev1WU/t8zBWq7jHSJv2uiq3d7gxcomdIwulzoAYdnvkL3+liX+7K+btPllqr0tJypXsXPwbb36LG+O4lavM9i/85Z+6MZ7YYecMSVp0/Llm+zFOrpak2/7t22b72f/NPoYk5UttZnsh9n/eX0gisz2N7OtBkuTEkKSml3sye71LUrlk9+ks+6+3WCya7YV864vGuyfNBWmaKm22Pldeyohjfy1EkZ0voiTkPmSf68Q5xnPjsF9N2ghJkP7rTZ3rMMv8/FjI2es2Nu5p48dx7m1R5L+WOPGvsZwTx2uXJG8FhNxTGk6fNPPPrzcnuSjgPdXIuRFLip3jRAExUmfWGmndjdF05iwy1nIaMhc/xzvRAAAAAAAEYhMNAAAAAEAgNtEAAAAAAARiEw0AAAAAQCA20QAAAAAABGITDQAAAABAIDbRAAAAAAAEOmrrRO/e96xZBzHL7FpylYpfR+w/HnrA6eHXb+vututAjwz7NYDLebuu2rHHLHZjVGs1t8+wM5bIf7mqNe3jDO32a5MuWGDXmlZA7b2REb/OpVfne+WqlW6MzoJd8/XAsF8X+9lnnrWP0eHXyVXinN/EP3kDgwNun3yu3Ww/dsVxboxI9vkLqfFt1feUpEbdnvdazb/+Z7sd/TvV1tb6mm7vsGvvRl7xZUnlvF3PttxWdmPknXq2uZxfF7TZsM93HPvrv1zwb3c1py62V3tbkkqJ/XrqlaobY/mxK8z233vXe9wYn/urv3H75Id+YrYvWXWKG+Op/9hhtm+89w43xuve8Gazvd7w12qzac9rpeLfI5X69VbjpGC2lwt+nei2ov3+hVevXJJip5ZpatTsTQLusbNdtVFX1Gh9H/BKGie5gJrhzpqLmwG1yT2Zv/Yz554bBdW89ofiljQOqK8bOROfhZTWdgabBNSVjwLeQ/RerleL+LkD2eevGTDvXl3kOPZzTuw9I4a8linYMmYB+6rUX2huDO/sRmp9XmKjbbLHAQAAAAAAP8cmGgAAAACAQGyiAQAAAAAIxCYaAAAAAIBAbKIBAAAAAAjEJhoAAAAAgEBsogEAAAAACMQmGgAAAACAQEdeOXuaDO4fVaHQulB9Lt80//22bT9zj+GV616wYIEbozJywGwfGtjvxsgXC2Z7tTLsxkhi/1TGTgH6fN7/mUq1MmK2j4za8yFJSWLPfHdXjxtj+fJlbp/du3fb4wj4EdKOZ5822zc9+oAbo6dnkdleLhfdGF1dbWZ7krPXkCQNDg66fUqFDrN93jz7tUj++V24aKEbI47sYvdxlJrt1WrVPcZst/nxJ1QqlVq2x7E9R6MjY+4xikV7bfbM73Zj5HN2jFJbuxujXq+Y7fv373NjRG7Gl7q77deTc3K1JJ1x1hlm+/CQfx0O7N9rtr/yFa92YxTz/rn5/z79F2b7EwM/dmMsXrrKbP/Wd7/rxhgdsa/X33zr290YNef+1lbscmMkWeb2iZxllAbEGBuz7+m1SsDNyb2BtR7H6Jh9H58LGmmmemqci6z186Uk1St1/xjNmtmeNfxnt6hhHyeKy24M5VvfByQpl+T9EAW/j7/i/BzbdK6POOD6iZ2LsJ7a+4OfjySgj8cfa71h32cz2c86khQ713qU2vd6SfKm1X8lkrKQObPPTRZwpGbTvjbjzF9nUWz3iaPW40jr9vEPiRPcU9I111yjM888U11dXerq6tKaNWv0r//6r+PtlUpFa9eu1YIFC9TR0aHLLrtMO3funMwhAGBWIj8CQGvkSABzyaQ20cuWLdOnPvUpbdy4Uffdd58uuOACvfGNb9QjjzwiSfrABz6gb3/72/r617+uDRs2aPv27Xrzm988LQMHgKMJ+REAWiNHAphLJvXr3Jdeeukh//8Xf/EXuuaaa3TPPfdo2bJluu6663TjjTfqggsukCRdf/31OvXUU3XPPffoZS972YQxq9XqIb96OTQ0NNnXAAAzbjryo0SOBDA38AwJYC457A8WazabuummmzQyMqI1a9Zo48aNqtfruvDCC8f7nHLKKVqxYoXuvvvulnHWr1+v7u7u8a/ly5cf7pAA4KgwVflRIkcCmHt4hgQw2016E/3QQw+po6NDxWJR73nPe3TzzTfrJS95ifr7+1UoFNTT03NI/yVLlqi/v79lvKuuukqDg4PjX9u2bZv0iwCAo8FU50eJHAlg7uAZEsBcMelP5z755JP1wAMPaHBwUN/4xjd0+eWXa8OGDYc9gGKx6H4CLADMBlOdHyVyJIC5g2dIAHPFpDfRhUJBJ5xwgiRp9erV+slPfqK//uu/1tve9jbVajUNDAwc8pPEnTt3qre3d8oGDABHK/IjALRGjgQwVxxxneg0TVWtVrV69Wrl83nddtttuuyyyyRJmzZt0tatW7VmzZpJx+3f8azy+dZ16+LYrmkXUq84Su36bFFA/bZGfdRsHxv16zHmy3Zt3mbq1ytc2O3XWpVTe3cgYKxeDeC2gJrHXp3NwUG/tnbkBZFUKNprZGzUr+E4Nmaf330Du9wYFadmcZL466xrnl0nurPc6cZIm3YtSUmqVOyxpk2/xl+tbscoFv2/Iununm93MGr8SVKc+uvjhTBd+VGScoqUM2oyenWR4zigxmXNroNaq9rtkrSn3y5R095u5z9JUs6+VTXrfo7cs9OuGy9J+/bY9ZkXL/FrnO9x6tMPjfoffBQ7dU6ffnarG2PZCcvcPlf+0fvN9s9+5s/dGHu2P222Lz/5TDfGTzc9YbZn/3CdG+M9V7zbbG/r6HFjeHV/Janh1BCtNvwao5FTozaJ/XrkOeP5SJLSrHXt2Mh5FnghTVeOHKnU1Iha542C8/TrlB3/eSc7x0bOOZKkojOQOLbv/ZIUFf0+boyAZ6rMmxSrLvfB43j3pYDnIf/UBNz/A7qkTh4OqQnfdOYsikJqwjvrKOTlOrWkvfMiSVnAGpFznCxgjSR5O/9FTb8utpx5jXOt11kckMMPmtQm+qqrrtIll1yiFStW6MCBA7rxxht155136nvf+566u7v1rne9S+vWrdP8+fPV1dWl973vfVqzZo35ybMAMBeQHwGgNXIkgLlkUpvoXbt26R3veId27Nih7u5unXnmmfre976nX/u1X5Mkff7zn1ccx7rssstUrVZ18cUX60tf+tK0DBwAjibkRwBojRwJYC6Z1Cb6uuvsX6kqlUq6+uqrdfXVVx/RoABgtiE/AkBr5EgAc8lh14kGAAAAAODFhk00AAAAAACB2EQDAAAAABCITTQAAAAAAIGOuE70VMt+Xnet7tRi9MrVNUJqNTr1yuoBtUe9mpEh45ATI039GCFj9Qo0h4zVO05QzTvnZzchdeRCaho2nHkLGan3ehsNu47gczHscdScWrySVK3al6pX3/m5GP4aqTo1rUPqRNcb3uvxZ94bh2SP4+C/zwLqOM42B19TtVIx+0WxXUuxGrBmlNnnqjJmjyHkOEkScBty+jTdNReypqTMqfsa8npHR0bs9lG79rzk14luBuTqRkAerYyNme2pU/NT8mt2NgLuTc0pyJHDw8Nme5r56yyoTrQz97WAczM6Zq+BJPbnLJe3X49VJ/rgGp3L+bFWsde2txxC6kS710fm54ssss9B7OQkSYoCyua6MUJqAMdODeeQZzenSxIw8UdPnWh/4utefeaAmu2JUyc65NR59ZtDJmQq6kSnAWvEG8pU1IlOjDrRlZ/n55D8GGVHWRZ95plntHz58pkeBoA5YNu2bVq2bNlMD2NKkSMBTAXyIwBMLCQ/HnWb6DRNtX37dnV2do7/ZGxoaEjLly/Xtm3b1NXVNcMjnBuY0+nBvE69w5nTLMt04MAB9fX1KQ55S2EW+eUcyZqbeszp9GBepx758VDkxxcG8zr1mNPpMdl5nUx+POp+nTuO45Y7/66uLhbWFGNOpwfzOvUmO6fd3d3TOJqZ0ypHsuamHnM6PZjXqUd+fA758YXFvE495nR6TGZeQ/Pj3PoRJAAAAAAA04hNNAAAAAAAgWbFJrpYLOrjH/+4isXiTA9lzmBOpwfzOvWYUxvzM/WY0+nBvE495tTG/EwP5nXqMafTYzrn9aj7YDEAAAAAAI5Ws+KdaAAAAAAAjgZsogEAAAAACMQmGgAAAACAQGyiAQAAAAAIxCYaAAAAAIBAR/0m+uqrr9bKlStVKpV03nnn6cc//vFMD2lWueuuu3TppZeqr69PURTpm9/85iHtWZbpYx/7mJYuXapyuawLL7xQTzzxxMwMdpZYv369zjnnHHV2dmrx4sV605vepE2bNh3Sp1KpaO3atVqwYIE6Ojp02WWXaefOnTM04qPfNddcozPPPFNdXV3q6urSmjVr9K//+q/j7cznxMiPR4b8OPXIj9ODHHl4yJGHj/w49ciP02Om8uNRvYn+6le/qnXr1unjH/+4fvrTn+qss87SxRdfrF27ds300GaNkZERnXXWWbr66qsnbP/MZz6jv/mbv9G1116re++9V+3t7br44otVqVRe4JHOHhs2bNDatWt1zz336NZbb1W9XtdFF12kkZGR8T4f+MAH9O1vf1tf//rXtWHDBm3fvl1vfvObZ3DUR7dly5bpU5/6lDZu3Kj77rtPF1xwgd74xjfqkUcekcR8ToT8eOTIj1OP/Dg9yJGTR448MuTHqUd+nB4zlh+zo9i5556brV27dvz/m81m1tfXl61fv34GRzV7Scpuvvnm8f9P0zTr7e3NPvvZz45/b2BgICsWi9k//dM/zcAIZ6ddu3ZlkrINGzZkWfbcHObz+ezrX//6eJ9HH300k5TdfffdMzXMWWfevHnZ3/3d3zGfLZAfpxb5cXqQH6cPOdJGjpw65MfpQX6cPi9Efjxq34mu1WrauHGjLrzwwvHvxXGsCy+8UHffffcMjmzu2Lx5s/r7+w+Z4+7ubp133nnM8SQMDg5KkubPny9J2rhxo+r1+iHzesopp2jFihXMa4Bms6mbbrpJIyMjWrNmDfM5AfLj9CM/Tg3y49QjR/rIkdOL/Dg1yI9T74XMj7kjHex02bNnj5rNppYsWXLI95csWaLHHntshkY1t/T390vShHN8sA22NE31/ve/X694xSt0+umnS3puXguFgnp6eg7py7zaHnroIa1Zs0aVSkUdHR26+eab9ZKXvEQPPPAA8/lLyI/Tj/x45MiPU4scGY4cOb3Ij0eO/Di1ZiI/HrWbaGA2WLt2rR5++GH98Ic/nOmhzHonn3yyHnjgAQ0ODuob3/iGLr/8cm3YsGGmhwXgMJEfpxY5Epg7yI9Tayby41H769wLFy5UkiTP+/S0nTt3qre3d4ZGNbccnEfm+PBceeWV+s53vqM77rhDy5YtG/9+b2+varWaBgYGDunPvNoKhYJOOOEErV69WuvXr9dZZ52lv/7rv2Y+J0B+nH7kxyNDfpx65Mhw5MjpRX48MuTHqTcT+fGo3UQXCgWtXr1at9122/j30jTVbbfdpjVr1szgyOaOVatWqbe395A5Hhoa0r333sscG7Is05VXXqmbb75Zt99+u1atWnVI++rVq5XP5w+Z102bNmnr1q3M6ySkaapqtcp8ToD8OP3Ij4eH/PjCIUe2Ro6cXuTHw0N+fOG8IPnxiD6WbJrddNNNWbFYzG644YbsP//zP7Mrrrgi6+npyfr7+2d6aLPGgQMHsvvvvz+7//77M0nZ5z73uez+++/Pnn766SzLsuxTn/pU1tPTk33rW9/KHnzwweyNb3xjtmrVqmxsbGyGR370eu9735t1d3dnd955Z7Zjx47xr9HR0fE+73nPe7IVK1Zkt99+e3bfffdla9asydasWTODoz66feQjH8k2bNiQbd68OXvwwQezj3zkI1kURdm//du/ZVnGfE6E/HjkyI9Tj/w4PciRk0eOPDLkx6lHfpweM5Ufj+pNdJZl2f/4H/8jW7FiRVYoFLJzzz03u+eee2Z6SLPKHXfckUl63tfll1+eZdlzZQr+5E/+JFuyZElWLBaz1772tdmmTZtmdtBHuYnmU1J2/fXXj/cZGxvL/vAP/zCbN29e1tbWlv3Gb/xGtmPHjpkb9FHu93//97Njjz02KxQK2aJFi7LXvva148kvy5jPVsiPR4b8OPXIj9ODHHl4yJGHj/w49ciP02Om8mOUZVl2ZO9lAwAAAADw4nDU/k00AAAAAABHGzbRAAAAAAAEYhMNAAAAAEAgNtEAAAAAAARiEw0AAAAAQCA20QAAAAAABGITDQAAAABAIDbRAAAAAAAEYhMNAAAAAEAgNtEAAAAAAARiEw0AAAAAQKD/H7HypwMALxqKAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1200x600 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "user, seed = next(train_federated_dataset)\n",
    "print('User: {}, unique user seed: {}'.format(user, seed))\n",
    "fig, axes = plt.subplots(1,len(user),figsize=(12,6))\n",
    "for ax, image, label in zip(axes, *user.raw_data):\n",
    "    ax.set_title('label={}'.format(np.argmax(label)))\n",
    "    ax.imshow((image.reshape(3,32,32).transpose(1,2,0)*channel_stddevs + channel_means).astype(np.uint8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Defining the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Defining a model in `pfl` is exactly the same as you would normally define a model for training. One of `pfl`’s strengths is that it is agnostic to any Deep Learning framework, which enables `pfl` to support multiple Deep Learning frameworks. \n",
    "\n",
    "At this moment, frameworks supported by `pfl` are TensorFlow, PyTorch and MLX.\n",
    "`pfl` also support ML models like GBDTs which use numpy.\n",
    "\n",
    "We will define an MLX model in the exact same way as we would for regular centralized training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MLP(\n",
      "  (linear1): Linear(input_dims=3072, output_dims=50, bias=True)\n",
      "  (linear2): Linear(input_dims=50, output_dims=50, bias=True)\n",
      "  (linear3): Linear(input_dims=50, output_dims=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "from pfl.metrics import Weighted\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear1 = nn.Linear(np.prod((32, 32, 3)), 50)\n",
    "        self.linear2 = nn.Linear(50, 50)\n",
    "        self.linear3 = nn.Linear(50, 10)\n",
    "\n",
    "    def __call__(self, x):\n",
    "        x = x.reshape(x.shape[0], -1)  # Flatten\n",
    "        x = nn.relu(self.linear1(x))\n",
    "        x = nn.relu(self.linear2(x))\n",
    "        x = nn.softmax(self.linear3(x))\n",
    "        return x\n",
    "\n",
    "    def loss(self, x, y, is_eval=False):\n",
    "        self.eval() if is_eval else self.train()\n",
    "        return nn.losses.cross_entropy(self(x), y, reduction=\"mean\")\n",
    "\n",
    "    def metrics(self, x, y):\n",
    "        self.eval()\n",
    "        pred = self(x)\n",
    "        num_samples = len(y)\n",
    "        correct = (pred.argmax(axis=1) == y.argmax(axis=1)).sum().item()\n",
    "        loss = mx.sum(nn.losses.cross_entropy(pred, y)).item()\n",
    "        return {\n",
    "            \"loss\": Weighted(loss, num_samples),\n",
    "            \"accuracy\": Weighted(correct, num_samples)\n",
    "        }\n",
    "\n",
    "mlx_model = MLP()\n",
    "print(mlx_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a model using `pfl`, a [Model](https://apple.github.io/pfl-research/reference/model.html#models) object must be instantiated for the Deep Learning framework in which the model was defined. A `pfl` Model acts as an adapter between the model of the Deep Learning framework and `pfl`. We use MLX for this example.\n",
    "\n",
    "The central optimizer is an input parameter to the model object. This optimizer is used when applying the aggregated model update to the central model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pfl.model.mlx import MLXModel\n",
    "\n",
    "model = MLXModel(model=mlx_model, \n",
    "                local_optimizer=mlx.optimizers.SGD(0.0),\n",
    "                central_optimizer=mlx.optimizers.SGD(1.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Training a model using Private Federated Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Backend` is a component in `pfl` used to collect and average statistics from user devices. [SimulatedBackend](https://apple.github.io/pfl-research/reference/aggregate.html#pfl.aggregate.SimulatedBackend) provides simulation. For larger datasets or models, it also [supports distributed simulations](https://apple.github.io/pfl-research/tutorials/simulation_distributed.html) with multiple processes, GPUs and machines.\n",
    "\n",
    "`pfl` provides various privacy mechanisms in [pfl.privacy](https://apple.github.io/pfl-research/reference/privacy.html#module-privacy).\n",
    "These mechanisms are used to guarantee local and/or central differential privacy and are parameterized by `SimulatedBackend` in simulations.\n",
    "\n",
    "Below, we initialize `SimulatedBackend` with Gaussian Moments Accountant as the central differential privacy mechanism, which provides the minimum privacy guarantee required by most use-cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pfl.aggregate.simulate import SimulatedBackend\n",
    "from pfl.privacy import CentrallyAppliedPrivacyMechanism, GaussianMechanism, PLDPrivacyAccountant, PRVPrivacyAccountant\n",
    "\n",
    "cohort_size = 200\n",
    "central_num_iterations = 5\n",
    "population = 1e7\n",
    "\n",
    "# Use Gaussian Moments Accountant, and transform it to a central privacy mechanism.\n",
    "# This one takes a bit longer to initialize.\n",
    "# accountant = PLDPrivacyAccountant(\n",
    "accountant = PRVPrivacyAccountant(\n",
    "    num_compositions=central_num_iterations,\n",
    "    sampling_probability=cohort_size / population,\n",
    "    mechanism='gaussian',\n",
    "    epsilon=2,\n",
    "    delta=1e-5)\n",
    "central_privacy = CentrallyAppliedPrivacyMechanism(\n",
    "    GaussianMechanism.from_privacy_accountant(accountant=accountant, clipping_bound=1.0))\n",
    "\n",
    "# Instantiate simulated federated averaging\n",
    "simulated_backend = SimulatedBackend(\n",
    "    training_data=train_federated_dataset,\n",
    "    val_data=val_federated_dataset,\n",
    "    postprocessors=[\n",
    "        central_privacy\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main component of a `pfl` modeling setup is the algorithm. In this tutorial, we will be using the [Federated Averaging](https://arxiv.org/pdf/1602.05629.pdf) algorithm. `pfl` implements Federated Averaging using the class [FederatedAveraging](https://apple.github.io/pfl-research/reference/algorithm.html#pfl.algorithm.federated_averaging.FederatedAveraging).\n",
    "\n",
    "Everything is tied together in the `run` method of an algorithm. This method requires the model, backend, optional callbacks, as well as hyperparameters, which depends on the model and algorithm combination. In this case, training a neural network with federated averaging requires a `NNTrainHyperParams` and `NNAlgorithmParams`.\n",
    "\n",
    "Important hyperparameters for federated learning include `local_learning_rate` and `local_num_epochs` which respectively specify the learning rate and the number of training epochs to be used for local training on devices during federated averaging. These hyperparameters differ to `central_learning_rate` and `central_num_iterations` which apply to training the central model using aggregated updates from devices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics at iteration 0 ():\n",
      "    Central val | loss                                : 2.318231689453125\n",
      "    Central val | accuracy                            : 0.0\n",
      "    Central val | number of data points               : 2000\n",
      "    Train population | number of devices              : 200\n",
      "    Train population | number of data points          : 5.135\n",
      "    Train population | loss before local training     : 2.318489120672833\n",
      "    Train population | accuracy before local training : 0.0\n",
      "    Train population | loss after local training      : 2.2984093108451122\n",
      "    Train population | accuracy after local training  : 0.11392405063291139\n",
      "    Central DP | l2 norm bound                        : 1.0\n",
      "    Central DP | fraction of clipped norms            : 0.0\n",
      "    Central DP | norm before clipping                 : 0.05452930993400514\n",
      "    Train population | total weight                   : 1.0\n",
      "    Central DP | DP noise std. dev. on summed stats   : 0.2452850341796875\n",
      "    Central DP | signal-to-DP-noise ratio on summed stats: 0.048254152574532014\n",
      "    Number of parameters                              : 156710\n",
      "    Val population | number of devices                : 10\n",
      "    Val population | number of data points            : 5.0\n",
      "    Val population | loss before local training       : 2.3187771892547606\n",
      "    Val population | accuracy before local training   : 0.0\n",
      "    Learning rate                                     : 1.0\n",
      "Metrics at iteration 1 ():\n",
      "    Train population | number of devices              : 200\n",
      "    Train population | number of data points          : 4.88\n",
      "    Central DP | l2 norm bound                        : 1.0\n",
      "    Central DP | fraction of clipped norms            : 0.0\n",
      "    Central DP | norm before clipping                 : 0.05644856893457472\n",
      "    Train population | total weight                   : 1.0\n",
      "    Central DP | DP noise std. dev. on summed stats   : 0.2452850341796875\n",
      "    Central DP | signal-to-DP-noise ratio on summed stats: 0.05316144334151408\n",
      "    Number of parameters                              : 156710\n",
      "    Learning rate                                     : 1.0\n",
      "Metrics at iteration 2 ():\n",
      "    Train population | number of devices              : 200\n",
      "    Train population | number of data points          : 4.89\n",
      "    Central DP | l2 norm bound                        : 1.0\n",
      "    Central DP | fraction of clipped norms            : 0.0\n",
      "    Central DP | norm before clipping                 : 0.06364037001505495\n",
      "    Train population | total weight                   : 1.0\n",
      "    Central DP | DP noise std. dev. on summed stats   : 0.2452850341796875\n",
      "    Central DP | signal-to-DP-noise ratio on summed stats: 0.06528176977204846\n",
      "    Number of parameters                              : 156710\n",
      "    Learning rate                                     : 1.0\n",
      "Metrics at iteration 3 ():\n",
      "    Train population | number of devices              : 200\n",
      "    Train population | number of data points          : 5.04\n",
      "    Central DP | l2 norm bound                        : 1.0\n",
      "    Central DP | fraction of clipped norms            : 0.0\n",
      "    Central DP | norm before clipping                 : 0.067404122678563\n",
      "    Train population | total weight                   : 1.0\n",
      "    Central DP | DP noise std. dev. on summed stats   : 0.2452850341796875\n",
      "    Central DP | signal-to-DP-noise ratio on summed stats: 0.07312495702459967\n",
      "    Number of parameters                              : 156710\n",
      "    Learning rate                                     : 1.0\n",
      "Metrics at iteration 4 ():\n",
      "    Train population | number of devices              : 200\n",
      "    Train population | number of data points          : 4.915\n",
      "    Train population | loss before local training     : 2.29211255007576\n",
      "    Train population | accuracy before local training : 0.2807731434384537\n",
      "    Train population | loss after local training      : 2.2138647083590985\n",
      "    Train population | accuracy after local training  : 0.5391658189216684\n",
      "    Central DP | l2 norm bound                        : 1.0\n",
      "    Central DP | fraction of clipped norms            : 0.0\n",
      "    Central DP | norm before clipping                 : 0.09131947554647922\n",
      "    Train population | total weight                   : 1.0\n",
      "    Central DP | DP noise std. dev. on summed stats   : 0.2452850341796875\n",
      "    Central DP | signal-to-DP-noise ratio on summed stats: 0.11285091731891947\n",
      "    Number of parameters                              : 156710\n",
      "    Val population | number of devices                : 10\n",
      "    Val population | number of data points            : 5.3\n",
      "    Val population | loss before local training       : 2.2854642058318517\n",
      "    Val population | accuracy before local training   : 0.3584905660377358\n",
      "    Learning rate                                     : 1.0\n",
      "    Central val | accuracy                            : 0.351\n",
      "    Central val | loss                                : 2.27258203125\n",
      "    Central val | number of data points               : 2000.0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pfl.model.mlx.MLXModel at 0x17462a890>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pfl.algorithm import FederatedAveraging, NNAlgorithmParams\n",
    "from pfl.callback import CentralEvaluationCallback\n",
    "from pfl.hyperparam import NNTrainHyperParams, NNEvalHyperParams\n",
    "\n",
    "model_train_params = NNTrainHyperParams(\n",
    "    local_learning_rate=0.05,\n",
    "    local_num_epochs=2,\n",
    "    local_batch_size=5)\n",
    "\n",
    "# Do full-batch evaluation to run faster.\n",
    "model_eval_params = NNEvalHyperParams(local_batch_size=None)\n",
    "\n",
    "algorithm_params = NNAlgorithmParams(\n",
    "    central_num_iterations=central_num_iterations,\n",
    "    evaluation_frequency=4,\n",
    "    train_cohort_size=cohort_size,\n",
    "    val_cohort_size=10)\n",
    "\n",
    "callbacks = [CentralEvaluationCallback(central_data, model_eval_params, 4)]\n",
    "\n",
    "algorithm = FederatedAveraging()\n",
    "algorithm.run(\n",
    "    backend=simulated_backend,\n",
    "    model=model,\n",
    "    algorithm_params=algorithm_params,\n",
    "    model_train_params=model_train_params,\n",
    "    model_eval_params=model_eval_params,\n",
    "    callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "This concludes the basics for simulating PFL using a simple dataset with MLX.\n",
    "\n",
    "* If you want to learn more about how to do research with `pfl`, check out [this tutorial](https://github.com/apple/pfl-research/tree/main/tutorials/Introduction%20to%20PFL%20research%20with%20FLAIR.ipynb) in PyTorch.\n",
    "* If you want to try the benchmark suite and learn setups for different datasets and models, see [our benchmark setups](https://github.com/apple/pfl-research/tree/main/benchmarks)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mlx",
   "language": "python",
   "name": "mlx"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}