{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9f6ceddd",
   "metadata": {},
   "source": [
    "# Federated Learning with Pytorch Backend"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c2fb4f8",
   "metadata": {},
   "source": [
    ">The following codes are demos only. It's **NOT for production** due to system security concerns, please **DO NOT** use it directly in production."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13008203",
   "metadata": {},
   "source": [
    "In this tutorial, We will walk you through how to use pytorch backend on SecretFlow for federated learning.\n",
    "+ We will use the image clasification task as example\n",
    "+ Use pytorch as backend\n",
    "+ We will show how to use multi fl strategy\n",
    "  \n",
    "If you want to learn more about federated learning, datasets, etc., you can move to [Federated Learning for Image Classification](Federate_Learning_for_Image_Classification.ipynb)\n",
    "  \n",
    "**Here we go!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4c69265d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b00d46f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-08-31 23:43:03.362818: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/devtoolset-10/root/usr/lib64:/opt/rh/devtoolset-10/root/usr/lib:/opt/rh/devtoolset-10/root/usr/lib64/dyninst:/opt/rh/devtoolset-10/root/usr/lib/dyninst:/opt/rh/devtoolset-10/root/usr/lib64:/opt/rh/devtoolset-10/root/usr/lib\n"
     ]
    }
   ],
   "source": [
    "import secretflow as sf\n",
    "\n",
    "# In case you have a running secretflow runtime already.\n",
    "sf.shutdown()\n",
    "\n",
    "sf.init(['alice', 'bob', 'charlie'], address='local')\n",
    "alice, bob, charlie = sf.PYU('alice'), sf.PYU('bob'), sf.PYU('charlie')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f58fe11d",
   "metadata": {},
   "source": [
    "BaseModule: Similar to the `torch.nn.module`    \n",
    "TorchModel: A wrap class include `loss_fn`,`optim_fn`,`model_def`,`metrics`  \n",
    "metric_wrapper: Wrap metrics to workers  \n",
    "optim_wrapper: Wrap optim_fn to workers  \n",
    "FLModel: Federated model, use `backend` to specify which bachend will be use, use `strategy` to spcify which federated strategy will be use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "de99cbf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from secretflow.ml.nn.fl.backend.torch.utils import BaseModule, TorchModel\n",
    "from secretflow.ml.nn.fl.utils import metric_wrapper, optim_wrapper\n",
    "from secretflow.ml.nn import FLModel\n",
    "from torchmetrics import Accuracy, Precision\n",
    "from secretflow.security.aggregation import SecureAggregator\n",
    "from secretflow.utils.simulation.datasets import load_mnist\n",
    "from torch import nn, optim\n",
    "from torch.nn import functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9c3ea64",
   "metadata": {},
   "source": [
    "When we define the model, we only need to inherit `BaseModule` instead of `nn.Module`, and the others are consistent with pytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "85d2028a",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "class ConvNet(BaseModule):\n",
    "    \"\"\"Small ConvNet for MNIST.\"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        super(ConvNet, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 3, kernel_size=3)\n",
    "        self.fc_in_dim = 192\n",
    "        self.fc = nn.Linear(self.fc_in_dim, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(F.max_pool2d(self.conv1(x), 3))\n",
    "        x = x.view(-1, self.fc_in_dim)\n",
    "        x = self.fc(x)\n",
    "        return F.softmax(x, dim=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e62ce093",
   "metadata": {},
   "source": [
    "We can continue to use the loss function and optimizer defined in pytorch, the only difference is that we need to wrap it with the wrapper provided in secretflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "645e3fbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "(train_data, train_label), (test_data, test_label) = load_mnist(\n",
    "    parts={alice: 0.4, bob: 0.6},\n",
    "    normalized_x=True,\n",
    "    categorical_y=True,\n",
    "    is_torch=True,\n",
    ")\n",
    "\n",
    "loss_fn = nn.CrossEntropyLoss\n",
    "optim_fn = optim_wrapper(optim.Adam, lr=1e-2)\n",
    "model_def = TorchModel(\n",
    "    model_fn=ConvNet,\n",
    "    loss_fn=loss_fn,\n",
    "    optim_fn=optim_fn,\n",
    "    metrics=[\n",
    "        metric_wrapper(Accuracy, task=\"multiclass\", num_classes=10, average='micro'),\n",
    "        metric_wrapper(Precision, task=\"multiclass\", num_classes=10, average='micro'),\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "95fcf7b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "device_list = [alice, bob]\n",
    "server = charlie\n",
    "aggregator = SecureAggregator(server,[alice,bob])\n",
    "\n",
    "# spcify params\n",
    "fl_model = FLModel(\n",
    "    server=server,\n",
    "    device_list=device_list,\n",
    "    model=model_def,\n",
    "    aggregator=aggregator,\n",
    "    strategy='fed_avg_w', # fl strategy\n",
    "    backend=\"torch\", # backend support ['tensorflow', 'torch']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c595099d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████▉| 749/750 [00:15<00:00, 47.41it/s]2022-08-31 23:43:34.759168: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/devtoolset-10/root/usr/lib64:/opt/rh/devtoolset-10/root/usr/lib:/opt/rh/devtoolset-10/root/usr/lib64/dyninst:/opt/rh/devtoolset-10/root/usr/lib/dyninst:/opt/rh/devtoolset-10/root/usr/lib64:/opt/rh/devtoolset-10/root/usr/lib\n",
      "2022-08-31 23:43:34.759205: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "100%|██████████| 750/750 [00:16<00:00, 46.78it/s, epoch: 1/20 -  accuracy:0.9709533452987671  precision:0.8571249842643738  val_accuracy:0.9840199947357178  val_precision:0.8955000042915344 ]\n",
      "100%|██████████| 125/125 [00:03<00:00, 31.28it/s, epoch: 2/20 -  accuracy:0.9825800061225891  precision:0.9190000295639038  val_accuracy:0.9850000143051147  val_precision:0.903249979019165 ]\n",
      "100%|██████████| 125/125 [00:02<00:00, 41.70it/s, epoch: 3/20 -  accuracy:0.9850000143051147  precision:0.9302499890327454  val_accuracy:0.9856399893760681  val_precision:0.906499981880188 ]\n",
      "100%|██████████| 125/125 [00:03<00:00, 41.66it/s, epoch: 4/20 -  accuracy:0.9859799742698669  precision:0.9334999918937683  val_accuracy:0.9861800074577332  val_precision:0.9085000157356262 ]\n",
      "100%|██████████| 125/125 [00:03<00:00, 41.46it/s, epoch: 5/20 -  accuracy:0.9870200157165527  precision:0.940500020980835  val_accuracy:0.9864799976348877  val_precision:0.9097499847412109 ]\n",
      "100%|██████████| 125/125 [00:02<00:00, 41.67it/s, epoch: 6/20 -  accuracy:0.987779974937439  precision:0.9422500133514404  val_accuracy:0.9869400262832642  val_precision:0.9137499928474426 ]\n",
      "100%|██████████| 125/125 [00:02<00:00, 41.92it/s, epoch: 7/20 -  accuracy:0.988099992275238  precision:0.9447500109672546  val_accuracy:0.9870200157165527  val_precision:0.9139999747276306 ]\n",
      "100%|██████████| 125/125 [00:03<00:00, 41.55it/s, epoch: 8/20 -  accuracy:0.9887800216674805  precision:0.9477499723434448  val_accuracy:0.986739993095398  val_precision:0.9135000109672546 ]\n",
      "100%|██████████| 125/125 [00:03<00:00, 41.49it/s, epoch: 9/20 -  accuracy:0.9892399907112122  precision:0.9502500295639038  val_accuracy:0.9868199825286865  val_precision:0.9132500290870667 ]\n",
      "100%|██████████| 125/125 [00:02<00:00, 41.81it/s, epoch: 10/20 -  accuracy:0.989359974861145  precision:0.9522500038146973  val_accuracy:0.9873600006103516  val_precision:0.9175000190734863 ]\n",
      "100%|██████████| 125/125 [00:03<00:00, 41.46it/s, epoch: 11/20 -  accuracy:0.9898999929428101  precision:0.953249990940094  val_accuracy:0.9874200224876404  val_precision:0.9194999933242798 ]\n",
      "100%|██████████| 125/125 [00:02<00:00, 41.77it/s, epoch: 12/20 -  accuracy:0.990119993686676  precision:0.953499972820282  val_accuracy:0.9871600270271301  val_precision:0.9154999852180481 ]\n",
      "100%|██████████| 125/125 [00:02<00:00, 41.87it/s, epoch: 13/20 -  accuracy:0.9906600117683411  precision:0.9570000171661377  val_accuracy:0.9876800179481506  val_precision:0.9202499985694885 ]\n",
      "100%|██████████| 125/125 [00:03<00:00, 40.91it/s, epoch: 14/20 -  accuracy:0.9910399913787842  precision:0.9572499990463257  val_accuracy:0.9880200028419495  val_precision:0.9227499961853027 ]\n",
      "100%|██████████| 125/125 [00:03<00:00, 41.49it/s, epoch: 15/20 -  accuracy:0.9903600215911865  precision:0.9542499780654907  val_accuracy:0.9878000020980835  val_precision:0.9194999933242798 ]\n",
      "100%|██████████| 125/125 [00:02<00:00, 41.68it/s, epoch: 16/20 -  accuracy:0.9914000034332275  precision:0.9585000276565552  val_accuracy:0.9878799915313721  val_precision:0.921750009059906 ]\n",
      "100%|██████████| 125/125 [00:02<00:00, 42.21it/s, epoch: 17/20 -  accuracy:0.9915599822998047  precision:0.9597499966621399  val_accuracy:0.988099992275238  val_precision:0.921750009059906 ]\n",
      "100%|██████████| 125/125 [00:03<00:00, 41.41it/s, epoch: 18/20 -  accuracy:0.9915800094604492  precision:0.9595000147819519  val_accuracy:0.9880399703979492  val_precision:0.921500027179718 ]\n",
      "100%|██████████| 125/125 [00:02<00:00, 41.83it/s, epoch: 19/20 -  accuracy:0.9916200041770935  precision:0.9605000019073486  val_accuracy:0.9887400269508362  val_precision:0.9244999885559082 ]\n",
      "100%|██████████| 125/125 [00:03<00:00, 41.34it/s, epoch: 20/20 -  accuracy:0.9922599792480469  precision:0.9637500047683716  val_accuracy:0.9883599877357483  val_precision:0.922249972820282 ]\n"
     ]
    }
   ],
   "source": [
    "history = fl_model.fit(\n",
    "            train_data,\n",
    "            train_label,\n",
    "            validation_data=(test_data, test_label),\n",
    "            epochs=20,\n",
    "            batch_size=32,\n",
    "            aggregate_freq=1,\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "55c13406",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3v0lEQVR4nO3deXxU9bn48c+TPSGBrKxJCAqyKTu4K4sbboiiQlurtb9q7abtr7dV21q0em293mvrbW/7s1WrvVW0bnWBggso7uzIviSRJEBIQhayJzPP749zAkPIMiGZTJJ53q/Xec3Z55nJZJ453+2IqmKMMcb4KyzYARhjjOldLHEYY4zpEEscxhhjOsQShzHGmA6xxGGMMaZDLHEYY4zpEEscxpwkEVklIv/Hz31VREYGOiZjuoMlDtOriEiuiNSISKXPNFREstwv54gWjlnsbruz2fo73fWLu+0FGNMHWOIwvdFVqhrvM+3345hdwNebrbvZXR/SRCQ82DGY3sUShwkVa4A4ERkP4D7GuOuPEpFvicgeETksIq+LyFCfbReLyA4RKReR3wPS7NhbRWS7iJSKyHIRGe5PYCLyDfe4IyKSLSK3N9s+T0Q2ikiFiOwVkcvc9cki8rSI7Hef8zV3/S0i8mGzcxwtKhORv4rIH0VkqYhUAbNE5AoR2eA+R17zqzAROU9EPhaRMnf7LSIyXUQKfROPiFwrIpv8ed2m97LEYULJ3zh21XGzu3yUiMwGHgZuAIYAXwJL3G2pwCvAz4FUYC9wrs+x84B7gWuBNGA18LyfcR0CrgT6A98AHhORKe55ZwDPAv8GJAIXALk+rycOGA8MBB7z8/kAvgI8BCQAHwJVOO9NInAFcIeIXOPGMBxYBvy3+9omARtVdQ1QAlzic96b3HhNH2aJw/RGr7m/fMuafmX76X+BRSISCSx0l319FXhKVderah1wD3C2iGQBlwNbVfUlVW0Afgsc9Dn228DDqrpdVRuBfwcm+XPVoapvqepedbwPrADOdzd/043pbVX1qmqBqu4QkSHAXODbqlqqqg3usf76p6p+5J6zVlVXqeoX7vJmnKR3obvvV4B3VPV593lKVHWju+0Z4GvgXAEBlwLPdSAO0wtZ4jC90TWqmuhO1/h7kKruA/bgfKnvVtW8ZrsMxbnKaNq/EucX9TB3W57PNvVdBoYDv2tKaMBhnKKsYe3FJSJzReRTt3isDCdJpbqbM3CubprLAA6raml752/Fca9dRM4UkZUiUiQi5TiJsL0YwEm+V4lIP5wrtdWqeuAkYzK9hCUOE2qeBf4vLRen7MdJAAC4X4YpQAFwAOcLtGmb+C7jfBHf7pPQElU1VlU/bisYEYkGXgYeBQapaiKwlGP1J3nAqS0cmgcki0hiC9uqcIqwmp5jcAv7NB8W+zngdSBDVQcAf/IjBlS1APgEp4juJpoV/5m+yRKH6WuiRSTGZ2r+GX8Bp0z+xRaOfR74hohMcr/Q/x34TFVzgbeA8W7lbwTwA8D3C/lPwD0+le8DROR6P+KNAqKBIqBRROZyfJ3Bk25Mc0QkTESGicgY91f9MuB/RCRJRCJF5AL3mE1urJNEJAZY7EccCThXMLVuvcpXfLb9HbhIRG4QkQgRSRGRST7bnwV+ApyBUw9k+jhLHKavqQRqfKbZvhtVtUZV31HVmuYHquo7wC9wrgAO4PzKXuhuKwauB36NU3w1CvjI59hXgd8AS0SkAtiCUwfRJlU9gpOEXgRKcb6wX/fZ/jluhTlQDrzPsauim4AGYAdOBftd7jG7gAeAd4DdOJXf7fkO8ICIHAHuwyexukV8l+NcqR0GNgITfY591Y3pVVWt9uO5TC8ndiMnY0xnichenKK6d4Idiwk8u+IwxnSKiFyHU2fyXrBjMd3jhOEZjDHGXyKyChgH3KSq3iCHY7qJFVUZY4zpECuqMsYY0yEhUVSVmpqqWVlZwQ7DGGN6lXXr1hWralrz9SGROLKysli7dm2wwzDGmF5FRL5sab0VVRljjOkQSxzGGGM6xBKHMcaYDgmJOo6WNDQ0kJ+fT21tbbBDCbiYmBjS09OJjIwMdijGmD4gZBNHfn4+CQkJZGVl4Qx02jepKiUlJeTn5zNixIhgh2OM6QNCtqiqtraWlJSUPp00AESElJSUkLiyMsZ0j5BNHECfTxpNQuV1GmO6R0gnDmOM6YtUlXVflvLAG9to8HT9EGIhW8cRbCUlJcyZMweAgwcPEh4eTlqa00Hz888/JyoqqtVj165dy7PPPsvjjz/eLbEaY3qH/NJqXl1fwCsbCsgpriImMoxrpwzj9GEDuvR5LHEESUpKChs3bgRg8eLFxMfH8+Mf//jo9sbGRiIiWv7zTJs2jWnTpnVHmMaYHq6qrpFlWw7y8rp8PskuAeDMEcncMfNU5p4+mISYrm9NaYmjB7nllluIiYlhw4YNnHvuuSxcuJA777yT2tpaYmNjefrppxk9ejSrVq3i0Ucf5c0332Tx4sXs27eP7Oxs9u3bx1133cUPfvCDYL8UY0wAeb3Kp9klvLQ+n39tOUh1vYfhKXH88KLTuHbKMDKS49o/SSdY4gDuf2Mr2/ZXdOk5xw3tzy+vGt/h4/Lz8/n4448JDw+noqKC1atXExERwTvvvMO9997Lyy+/fMIxO3bsYOXKlRw5coTRo0dzxx13WJ8NY/qg7KJKXllfwKsbCigoqyEhOoJ5k4Zy7ZR0pg1P6raGMJY4epjrr7+e8PBwAMrLy7n55pvZvXs3IkJDQ0OLx1xxxRVER0cTHR3NwIEDKSwsJD09vTvDNsYESHl1A29s3s/L6/PZsK+MMIHzR6Xx07ljuGTcIGIiw7s9JksccFJXBoHSr1+/o/O/+MUvmDVrFq+++iq5ubnMnDmzxWOio6OPzoeHh9PY2BjoMI0JCY0eL41exauKx6t4veBpmtfW13u8x6ZGr5f6RqXB4z061XuUhsbjlxt9t3m8NDR6yS+t4b2dh6hv9HLaoHjumTuGayYPY1D/mKC+L5Y4erDy8nKGDRsGwF//+tfgBmNMH9Hg8VJSWU/RkTqKK+soOlJHUbPHYnf+SG1wfoRFhYcRGS70j43kKzMyWTA1nfFD+/eYPlmWOHqwn/zkJ9x88808+OCDXHHFFcEOx5heZcfBCl5dX0BhRe2xhFBZz+Gq+hb3T4iJIC0+mtSEaMYO7c8F8dEkxkUSGR5GmAjhYbiPznR0XoSwsGbb3XURYUJkeBiR4WFERRybjwwPc5JD07qwY/MRYdJjEkRrQuKe49OmTdPmN3Lavn07Y8eODVJE3S/UXq8JXUdqG3js7d0880ku4SIMGhBNWnw0aQnOlNo07yaJpm3BqCvo6URknaqe0PbfrjiMMX2CqvL6pv08+NZ2iivrWDg9k59cOpqkfq13pjUnxxKHMSYgKusaqahpYGhibMCfa1fhEX7x2hY+yznMhPQB/Pnr05iUkRjw5w1VljiMMZ3m9SrZxVVs2FfK+n1lbNhXyq7CI3gVpg5P4sbpGVw5YQhxUV37lVNZ18jv3tnF0x/l0i86gofmn87C6ZmEh/XsOoLezhKHMabDymsa2JRXxvp9pWzYV8bGvDLKa5x+RgkxEUzKSOSS8YOJiQzjpXX5/OSlzfzqjW1cPWkoC6dnckZ658ZOUlXe3HyAB9/aRmFFHQunZ/CTy8aQbMVS3cIShzGmTR6vsudQpZsknCuKPYcqARCB0wYmcPkZg5mckcTkzEROTYsnzOcX/x0Xnsqa3FKWfL6Pl9bl8/fP9jF+aH8Wzshk3qSh9O/gWEp7Dh3hvn9u5eO9JZw+rD9//NpUpmQmdelrNm2zxGGMOUFOcRUrth7kg91FbMorp7LO6c+QFBfJ5Mwk5k0cypThSUxIH9DuIHoiwowRycwYkcwvrxrPPzcV8PznefzitS38+1vbuWLCEBbNyGBKZttDZlTVNfL4e7t5cnUOcVHh/GreeL5y5nArlgoCSxxBMmvWLO6++24uvfTSo+t++9vfsnPnTv74xz+esP/MmTN59NFHmTZtGpdffjnPPfcciYmJx+3T0ii7xvjD61W+KChnxbaDrNhayG73imLM4ATmTx7G5MxEJmcmkZUS16k+BgPiIvn62VncdNZwNueXs2TNPl7fuJ+X1uUzamA8N07P4Lop6ce1hFJVlm05yK/e3MaB8lqun5rOT+eOITU+uo1nMoFkiSNIFi1axJIlS45LHEuWLOGRRx5p99ilS5cGMjQTIuobvXyWU8KKrYW8va2QgxW1hIcJZ45I5qtnZnLx+MEMC1CLKBFhYkYiEzMS+dkV43hz036eX5PHg29t55F/7eTS0wezcHoGg/rHcP8bW1m9u5ixQ/rz+69MZurw5IDEZPxniSNIFixYwM9//nPq6+uJiooiNzeX/fv38/zzz/OjH/2ImpoaFixYwP3333/CsVlZWaxdu5bU1FQeeughnnnmGQYOHEhGRgZTp04NwqsxvUVlXSPv7yxixbaDvLfjEEdqG4mNDOfC09K4ZPwgZo8ZSGJc91Ywx0dHsHBGJgtnZLL9QAUvrMnjlfX5vLFpPwAJ0REsvmocXztrOBHhdtPSniCgiUNELgN+B4QDf1HVXzfbPhx4CkgDDgNfU9V8d9tvgKZxNn6lqi+460cAS4AUYB1wk6q2PIaAv5bdDQe/6NQpTjD4DJj761Y3JycnM2PGDJYtW8a8efNYsmQJN9xwA/feey/Jycl4PB7mzJnD5s2bmTBhQovnWLduHUuWLGHjxo00NjYyZcoUSxzmBEVH6nh3eyHLtx7koz0l1Hu8JPeLYu7pg7lk3GDOG5XaY3pNjx3Sn8VXj+fuuWNYtuUAOUVV3HR2FmkJVizVkwQscYhIOPAH4GIgH1gjIq+r6jaf3R4FnlXVZ0RkNvAwcJOIXAFMASYB0cAqEVmmqhXAb4DHVHWJiPwJ+CZwYqVAL9BUXNWUOJ588klefPFFnnjiCRobGzlw4ADbtm1rNXGsXr2a+fPnExfn3LTl6quv7s7wTQ9zpLaBvMM15JdWk1fqPH6RX866faWoQkZyLF8/eziXjB/M1OFJPbpSOSYynPmT7dYAPVUgrzhmAHtUNRtARJYA8wDfxDEO+JE7vxJ4zWf9B6raCDSKyGbgMhH5BzAb+Iq73zPAYjqbONq4MgikefPm8cMf/pD169dTXV1NcnIyjz76KGvWrCEpKYlbbrmF2traoMRmep6aeo+bFKrJL60h77D7WFpN3uGao/0omvSLCmfkwHjumnMal4wfxJjBCT1+8DzTOwQycQwD8nyW84Ezm+2zCbgWpzhrPpAgIinu+l+KyH8CccAsnISTApS5CaXpnMNaenIRuQ24DSAzM7MrXk+Xi4+PZ9asWdx6660sWrSIiooK+vXrx4ABAygsLGTZsmWt3oMD4IILLuCWW27hnnvuobGxkTfeeIPbb7+9+16ACZj9ZTW8va2QtV+WugmimuLK40tkoyPCSE+KJSM5jkkZiWQkxZGRHOesS4ojMS7SEoUJiGBXjv8Y+L2I3AJ8ABQAHlVdISLTgY+BIuATwNORE6vqE8AT4IyO25VBd6VFixYxf/58lixZwpgxY5g8eTJjxowhIyODc889t81jp0yZwo033sjEiRMZOHAg06dP76aoTVdTVXYfqmT5loOs2FbIFwXlAAxLjCUrNY6Lxg46mhTSk+LISI4lLT7aEoMJioANqy4iZwOLVfVSd/keAFV9uJX944EdqnpCwaaIPAf8L7AMJ5EMVtXG5s/RGhtWPfReb2/g8Sob9pWyYlshK7YeJLekGoDJmYlcMm4wF48bxMiB8UGO0oSyYAyrvgYY5baCKgAWcqxuoimoVOCwqnqBe3BaWDVVrCeqaomITAAmACtUVUVkJbAAp2XVzcA/A/gajOlStQ0ePt5bzIqthbyzvZDiynoiw4VzTk3lWxecwsVjBzEwyLcFNaY9AUsc7hXB94DlOM1xn1LVrSLyALBWVV8HZgIPi4jiFFV91z08EljtXoZX4DTTbarX+CmwREQeBDYATwbqNRjTFcqrG1i58xArth1k1c4iqus9xEdHMGvMQC4ZN4iZo9PaHbbDmJ4koHUcqroUWNps3X0+8y8BL7VwXC1Oy6qWzpmN02KrK+ILiTLiULjLYzB4vMqR2gbKqhsoq2mgvKaBsup699FZ3nnwCJ9ml9DoVQYmRDN/8jAuGT+Ys05JJjqiZ/SdMKajgl05HjQxMTGUlJSQkpLSp5OHqlJSUkJMjBV/dERpVT0f7y1h+4EKymrqKa9pPC4plFXXc6SukbZycr+ocIYlxfKtC07hknGDmJieeNyoscb0ViGbONLT08nPz6eoqCjYoQRcTEwM6enWmaottQ0e1uQe5sM9xXy0p5it+ytQhTCBAbGRJMZFMSA2kuR+UYxI7UdibCQD4qKcx9hIEuOcaUCss9+A2EiiImx4DNM3hWziiIyMZMSIEcEOwwSJx6ts3V9+NFGsyS2lvtFLZLgwOTOJH150GueNSmXCsAE2PpIxzYRs4jCh58uSqqOJ4uO9JZRVOz2txwxO4OtnDefcUanMyEqmX7T9WxjTFvsPMX3WkdoG3t9VxEd7ivlwTzF5h2sAGDIghovHDuK8UamcfWoKAxOs/seYjrDEYfoUVWX9vlKe/zyPtzYfoKbBQ0JMBGefksK3zj+F80amMiK1X59uEGH6gOLdsPZp+PIjGDgO0qdBxgxIGwvhwf/aDn4ExnSB0qp6XtlQwJLP97H7UCX9osK5ZvJQrpuSzqSMRKunMD1fYz3seMNJGLmrISwC0mfA7hWw6Tlnn8h+MGwKpE8/NsWndXuoljhMr+X1Kp9ml/D8mjyWbzlIvcfLpIxEfnPdGVw5YajVVZjeoTQX1v0VNvwvVBVBYibMuQ8mfQ0SBoGqs0/+Wsj/HPLXwMePg9ftE52UdXwiGXwGhAe2Q6n9Z5le59CRWl5al88La/L4sqSa/jERfOXMTBbOyGDM4P7BDs+Y9nkaYfdyWPsU7HkXROC0uTDtG3DqbAjz6RwqAskjnGnC9c66hhrYv9FJIvlrIPdD+OIfzraIGBg62SneSp8Op8yCmK79vwjYIIc9SUuDHJrexeNVPthVxPOf7+PdHYfweJUzRySzaEYml50+uMfcwc6YNpUXwPpnnenIfkgYAlNuhik3wYBO9LVShYoCyPvcvTJZAwc2gqcevrsG0k47qdMGY5BDYzqtoKyGF9bk8Y+1eRworyWlXxT/57wR3Dg9g1PSbORY0wt4vbD3PefqYtcy50t+5By44lEYdWnXVHaLOIlnQDqcfq2zrrHOuSV2ysjOn78ZSxymxyivbmDbgQq2HahguzttO1ABwPmj0rjvynHMGTvIemT3FvVVUFkICUMhsgc3ea6rhLIvofRLKNt3/HzVIQiPduKPiIHIWJ/HaIiIdbe18lhZCOv/5pwzLhXOvdO5wkjuhs7HEdFOcVUgTh2QsxrTBq9XySutZtv+Y8lh+4EjFJTVHN0nNT6acUP7c+ecUVw3JZ2M5LggRmzaVVvh/Lo9sBEObHLK34t3AW5RePxgSBruVPwmDj9+fkB6YCtzG2qPJYSWEkTN4eP3j4xz4krMhGGTwdPg1Ck01rqPdVBb5py3scZ9dLd5G058/qzz4aLFMOZKiIgK3OvsRpY4TEDV1HvYWXjESRD7j11JVNU7N3QMDxNOSe3HtKwkbhoynLFD+jN2SIJ1yuvJakrhwObjk8Thvce2JwyBIZNg/HxIzICK/e6X9ZeQ9xlseQXU54aeEgb9h7WcVOIHOl/I9VXuVOk8NlQfmz+6vvr4feqrnC/4ysLj4w+PduJKHO7EmTTcTRTuc8elOEU/J8PrOZZkGmshLNJpGdXHWOIwXc7rVVbtOsRTH+by8d5ivO6PzoToCMYO6c+CqemMG9qfsUP6c9qgBKvY7smqSo4liKbH0txj2wdkwJCJMHGR8zhkYvtflJ5GpyK3+a//sn2QvQqOHODolUq7BKLiIaqfzxTvfPknZjqtiQZkNktGgyAsQMWdYeEQHe9MfZglDtNlqusbeXldPk9/lEt2cRWD+8dwx8xTOWNYIuOH9ic9KdZ6bPd0Xi8UrHMqcXf+Cw5tPbYtKctJDFNudpPEJOiX0vHnCI9wvsiThkNLRf2NdVCe7ySoqmKIijuWEHyTQ1Q/p77BPlPdzhKH6bT9ZTU880kuz3+2j4raRiZmJPL4osnMPX0wkdZju+erq4TslU6i2L3c6YQm4ZB5Nsz5JQybCkMmQGxS98QTEQ0ppzqT6ZEscZiTtmFfKU9+mMOyLQdRVeaePoRbzxvBlMxEu7IAp9llXYVPxapvZarPY2Nds8pXn8ewCEh2v0RTR0H/9K4pZinPh13/cpJFzgfgqYPoATDqIqcj2sg5EJfc+ecxfZIlDtMhjR4vy7cW8uSH2azfV0ZCdAS3npvFzedkkZ5kLZ/wepxOWDvehB1vQWnOSZxEjjX79NQ7lb1NImKOTyQpo5x2+qkj274i8HrhwAYnUexa5rSAAkgaAdO/CaddBsPPCfhQFaZvsMRh/FJe08ALa/bxzMdfUlBWw/CUOBZfNY4F0zKID/UxoRpqnUrdHW/CzmVQXQzhUTDiQph6i1NR2l57/wiffgLhUcfK7VWh8hCU7IaSPc6oqSV74NA22Ln02HhF4FQI+yaSlJFOi6Vdy52p8qCznHEmXHQ/jJ4LqadZHYHpsBD/jzftyS2u4umPcvjHunyq6z2cdUoyi68ez+wxAwkP5ftn15TCrhVOstjzLjRUQXR/GHUJjLkCRl7UNeMDiTitlBIGQdZ5x2/zNDitkponlT1vw8b/PbZfVIJT9DR6Loy8+OQqtI3xYYnDtKiwopaHl27nn5v2ExEmXD1xGN84N4vThw3w/yT11U4zy+Zt61trc39cG313OSzS+VWcNvrYlHqa06Kmu5Xnw46lTrL48iPn1378YJi40EkWWed3bwev8EjnyiK1hSElasuhZK9TV5I+vc90PDM9gyUOc5wGj5dnPs7lt+/spt7j5dsXnso3zs3yr0Oe1+N0Bste6RTd5H3mlNG3JSL2xCaWUXHQL82Zb6h2fknvXn58scyATGfgtrQxbmIZ4yx3ZcsfrweKdsLOt5z6iv0bnPWpp8E533d6Ag+dErg+AZ0RM8C5b4MxAWCJwxz1WXYJ9/1zKzsLjzBrdBqLrx7P8JQ2ftmrwuHsY4ki5wPnly449wQ483ZneOfo/icmh0i3bX6Yn53/PA1wOAeKdkDxTucLvWinM5x0Y+2x/eIH+SSS0ZAw2L26qTz+SuboVY7vcrPeyI3HhkAhfbozbMToK056pFFj+gpLHIZDR2p5eOkOXt1QwLDEWJ64aSoXjxvUcpPayiLIed9NFu9DeZ6zfkAGjL0aTpnpTP1SuzbI8Ej3CqPZl7bXC+X73ESyA4p2OY+bX3CawrakKWkdd5UT7ySd5p3MEgbDaZc6j8YYwBJHSGv0ePnbp1/yXyt2Udfo5fuzR/KdmSOJjfK5Cqivgi8/OZYoCt1mnDEDYMQFcN5dzo1ikk8JTuucsDCnR3NSlvMF30TVGbqiqtgnQcQ5ScPfqxxjTIsscYSotbmH+flrW9hx8AgXnJbG/VeNY0RMJeS9D4e2Q+E2p8ln4RanniI8ymnGOfsXcOosZ7iJnvwFLAL9hzqTMaZLWeIIMcWVdTz2xlq2b/6cWf0O8P/GV5LZmIs8tf344aX7pcHAcXDmt52ip8yznV/sxpiQZ4mjL/M0OOX9hdvwFm5j/651hBVv5yGKIRpoBPLiYeBYGHslDBzvzA8cB/FpwY7eGNNDWeLoa+oqYe+7sP1Npwmr28rJQwRHvEMo7XcG0RPPJGXEJCdBDMjomc1JjTE9liWOvqCyyBl/aMdbsHelM2BdbBK1p17GK+Wn8fTeBGrih3PPVRO5/IzBNgChMaZTLHH0VoeznUSx4y3Y9ymgTqe4abdSP2ouz+QP5vFVudTUe/jmBSP4wexR9Av1MaWMMV3Cvkl6C1XnDmxNyeLQNmf9oDPgwp/CmCvQQaezfNshHn51O1+W7GHW6DR+dsVYRg5MCGroxpi+xRJHT+ZpcMZE2vGWM0ZSRb4zumnmOXDpwzDmcqf/ArCloJwH//IZn2YfZtTAeJ65dQYXnmYV3MaYrmeJoyc6sBk2PQ9f/MO5G1tEDJw6B2bd69w3wWd000MVtTy6Yif/WJdPUlwUv7rmdBZNzyDC7rxnjAkQSxw9ReUh2PyikzAKtzijwo6+DM64wRkSu9losLUNHv6yOpv/WbWXBo+Xb51/Ct+dNZIBsXYjHmNMYFniCKaGWudmPJued+7poB7n/s6XPwqnX9firTtVlTc2H+A3y3ZQUFbDpeMHcc/csWSlBmGYcWNMSApo4hCRy4DfAeHAX1T11822DweeAtKAw8DXVDXf3fYIcAUQBrwN3KmqKiKrgCFA09Cll6jqoUC+ji6lCvlrYONzsPUVp59FwlA49wcwcZEzomsrNuwr5VdvbmP9vjLGDenPo9dP5OxT7aY8xpjuFbDEISLhwB+Ai4F8YI2IvK6q23x2exR4VlWfEZHZwMPATSJyDnAuMMHd70PgQmCVu/xVVV0bqNgDoiwPNi+BTUucu7RFxMLYq2DSIucWo22M+7S/rIZH/rWD1zbuJy0hmkeum8B1U9ND+w58xpigCeQVxwxgj6pmA4jIEmAe4Js4xgE/cudXAq+58wrEAFGAAJFAYQBjDYz6Ktj2Omx6DnJWAwrDz4Vz74Jx89q9tWh1fSN/WrWXJ1Zn41X47qxTuWPmSLvHtzEmqAL5DTQMyPNZzgfObLbPJuBanOKs+UCCiKSo6icishI4gJM4fq+q232Oe1pEPMDLwIOqqoF6ESfF63GKot77FVQWOk1mZ97t3GLUbT7bHo9XueXpNXyec5irJg7lp5eNJj3JBhk0xgRfsH+6/hj4vYjcAnwAFAAeERkJjAXS3f3eFpHzVXU1TjFVgYgk4CSOm4Bnm59YRG4DbgPIzMwM+As5Kmc1LL8HDn7h3DXuuich67wO36vif1bu4fOcwzyyYAI3TMsIULDGGNNxgWzsXwD4fuOlu+uOUtX9qnqtqk4GfuauK8O5+vhUVStVtRJYBpztbi9wH48Az+EUiZ1AVZ9Q1WmqOi0trRs6wh3OhiVfhWeuhOpSJ2F8820YcX6Hk8b6faX89t3dXD1xKNdPTW//AGOM6UaBTBxrgFEiMkJEooCFwOu+O4hIqog0xXAPTgsrgH3AhSISISKROBXj293lVPfYSOBKYEsAX0P7asthxc/h9zOcAQZn/xy+vxbOWHBSd8SrrGvkriUbGdw/hgfnn24DEhpjepyAFVWpaqOIfA9YjtMc9ylV3SoiDwBrVfV1YCbwsIgoTlHVd93DXwJmA1/gVJT/S1XfEJF+wHI3aYQD7wB/DtRraJOnEdb/FVb+O1QfhklfhTm/6PS9qX/5z63kl1bzwu1n0z/GOvMZY3qegNZxqOpSYGmzdff5zL+EkySaH+cBbm9hfRUwtesj7aA978Lyn0HRdhh+Hlz6EAyd1OnTvrFpPy+vz+cHs0cyPevEzn/GGNMTBLtyvHcp2gUrfga7Vzito274m9MXowuKkwrKarj31S+YnJnID+aM6nysxhgTIJY4/FF9GFb9Gtb8xRkz6uJfwZm3Q0R0l5ze41V+uGQjXq/yuxsn2wCFxpgezRJHWxrrnWTx/m+grgKmfsMZobZfapc+zR9X7eHz3MP85/UTyUyxvhrGmJ7NEkdb/r4Act6HU2fDJQ/BoHFd/hQb88p47J3dXDVxKNdOGdbl5zfGmK5miaMtZ90BZ38PRl3cJfUYzVXWNXLnkg1O09trrOmtMaZ3sMTRltFzA3r6xa9vJe9wNUtuO9vuo2GM6TXarYUVkat8OumZLvLm5v28tC6f784ayYwR1vTWGNN7+JMQbgR2i8gjIjIm0AGFgoKyGu595QsmZVjTW2NM79Nu4lDVrwGTgb3AX0XkExG5zR1k0HSQx6v88IWNeLzK7xZOItKa3hpjehm/vrVUtQKnh/cSnLvvzQfWi8j3Axhbn/Sn9/fyec5h7p93OsNT7Havxpjex586jqtF5FWcu+9FAjNUdS4wEfi/gQ2vb9mUV8Zjb+/iyglDuM6a3hpjeil/WlVdBzymqh/4rlTVahH5ZmDC6nuq3Ka3AxOieeiaM6zprTGm1/IncSzGuRMfACISCwxS1VxVfTdQgfU197+xlX2Hq3n+W2cxIM6a3hpjei9/6jj+AXh9lj3uOuOnpV8c4MW1+Xxn5kjOPCUl2OEYY0yn+JM4IlS1vmnBnY8KXEh9y/6yGu5+eTMTMxK58yJremuM6f38SRxFInJ104KIzAOKAxdS33Fc09sbremtMaZv8KeO49vA30Xk94AAecDXAxpVH/H0Rzl8lnOY/1gwgaxUa3prjOkb2k0cqroXOEtE4t3lyoBH1Uf8a8tBJmYksmBqerBDMcaYLuPXIIcicgUwHohpakaqqg8EMK4+Iae4ikvGD7Kmt8aYPsWfDoB/whmv6vs4RVXXA8MDHFevV17dQElVPSOsiMoY08f4U1t7jqp+HShV1fuBs4HTAhtW75dTUgXAiNT4IEdijDFdy5/EUes+VovIUKABZ7wq04acYqcqyK44jDF9jT91HG+ISCLwH8B6QIE/BzKoviCnqIowgcxku4e4MaZvaTNxuDdweldVy4CXReRNIEZVy7sjuN4su7iKjOQ4oiKs74Yxpm9p81tNVb3AH3yW6yxp+CenuMqKqYwxfZI/P4ffFZHrxNqU+k1VLXEYY/osfxLH7TiDGtaJSIWIHBGRigDH1asdOlJHdb2HUyxxGGP6IH96jtstYjsou8ia4hpj+q52E4eIXNDS+uY3djLH5BS7iSPNrjiMMX2PP81x/81nPgaYAawDZgckoj4gp7iS6IgwhvSPCXYoxhjT5fwpqrrKd1lEMoDfBiqgvqCpYjwszNoTGGP6npPpZJAPjO3qQPqSbGtRZYzpw/yp4/hvnN7i4CSaSTg9yE0LGj1e9pVUc9n4wcEOxRhjAsKfOo61PvONwPOq+lGA4un18ktraPSqXXEYY/osfxLHS0CtqnoARCRcROJUtTqwofVOTS2qTrEWVcaYPsqvnuNArM9yLPBOYMLp/bKLrQ+HMaZv8ydxxPjeLtadtyFfW5FTXMmA2EiS4iKDHYoxxgSEP4mjSkSmNC2IyFSgJnAh9W5NTXFtaC9jTF/lT+K4C/iHiKwWkQ+BF4Dv+XNyEblMRHaKyB4RubuF7cNF5F0R2Swiq0Qk3WfbIyKyVUS2i8jjTYMsishUEfnCPefR9T1FTlGVjVFljOnT2k0cqroGGAPcAXwbGKuq69o7TkTCcYZknwuMAxaJyLhmuz0KPKuqE4AHgIfdY88BzgUmAKcD04EL3WP+CHwLGOVOl7UXS3epqfewv7zWWlQZY/q0dhOHiHwX6KeqW1R1CxAvIt/x49wzgD2qmq2q9cASYF6zfcYB77nzK322K87wJlFANBAJFIrIEKC/qn6qqgo8C1zjRyzdIrfExqgyxvR9/hRVfcu9AyAAqlqK84u/PcOAPJ/lfHedr03Ate78fCBBRFJU9ROcRHLAnZar6nb3+Px2zgmAiNwmImtFZG1RUZEf4Xbe0cEN7YrDGNOH+ZM4wn3rEdwiqKguev4fAxeKyAacoqgCwCMiI3GGNUnHSQyzReT8jpxYVZ9Q1WmqOi0tLa2Lwm1bU+LISrHEYYzpu/zpAPgv4AUR+X/u8u3AMj+OKwAyfJbT3XVHqep+3CsOEYkHrlPVMhH5FvBpUzNgEVkGnA38zT1Pq+cMpuyiKgb3j6FftD9vqzHG9E7+XHH8FKce4tvu9AXHdwhszRpglIiMEJEoYCHwuu8OIpIqIk0x3AM85c7vw7kSiRCRSJyrke2qegCoEJGz3KugrwP/9COWbpFTXGnFVMaYPs+fVlVe4DMgF6fCezaw3Y/jGnGa7S53939RVbeKyAMicrW720xgp4jsAgYBD7nrXwL24iSpTcAmVX3D3fYd4C/AHncff65+ukVOcZVVjBtj+rxWy1RE5DRgkTsV4/TfQFVn+XtyVV0KLG227j6f+ZdwkkTz4zw4RWItnXMtThPdHqW0qp7S6gbrw2GM6fPaKozfAawGrlTVPQAi8sNuiaoXyimxFlXGmNDQVlHVtThNYVeKyJ9FZA7Qo3pp9yQ5RZY4jDGhodXEoaqvqepCnF7jK3GGHhkoIn8UkUu6Kb5eI6e4ivAwISPZxn80xvRt/lSOV6nqc+69x9OBDTgtrYyPnOIqMpPjiAw/mbvxGmNM79GhbzlVLXU71s0JVEC9ld1n3BgTKuzncRfwepVcSxzGmBBhiaMLFB6ppabBY4nDGBMSLHF0gaYWVdaHwxgTCixxdIGj9xm3XuPGmBBgiaML5BRXERsZzqCEmGCHYowxAWeJowvkFFeRldqPsDDrH2mM6fsscXSBnGK7z7gxJnRY4uik+kYv+w5XW4sqY0zIsMTRSXml1Xi8aonDGBMyLHF00tHBDa1FlTEmRFji6KSm+4xbHYcxJlRY4uik7OIqkuIiSYyLCnYoxhjTLSxxdJLdZ9wYE2oscXRSTnEVI1Ljgx2GMcZ0G0scnVBV10hhRR2nWMW4MSaEWOLohKaKcSuqMsaEEkscnWCJwxgTiixxdEJT4shKscRhjAkdljg6Iae4iqEDYoiNCg92KMYY020scXRCdnGV9Rg3xoQcSxwnSVXJKbI+HMaY0GOJ4yQdrqqnorbR+nAYY0KOJY6TZGNUGWNClSWOk5RtTXGNMSHKEsdJyimuIjJcSE+KDXYoxhjTrSxxnKScoioyk+OICLe30BgTWuxb7yTZ4IbGmFBlieMkeL1KTkmVDW5ojAlJljhOwv7yGuobvVYxbowJSZY4ToINbmiMCWWWOE6C9eEwxoQySxwnIbuoin5R4aQlRAc7FGOM6XYBTRwicpmI7BSRPSJydwvbh4vIuyKyWURWiUi6u36WiGz0mWpF5Bp3219FJMdn26RAvoaW5LiDG4pIdz+1McYEXcASh4iEA38A5gLjgEUiMq7Zbo8Cz6rqBOAB4GEAVV2pqpNUdRIwG6gGVvgc929N21V1Y6BeQ2usKa4xJpQF8opjBrBHVbNVtR5YAsxrts844D13fmUL2wEWAMtUtTpgkXZAXaOH/NJqqxg3xoSsQCaOYUCez3K+u87XJuBad34+kCAiKc32WQg832zdQ27x1mMi0mJFg4jcJiJrRWRtUVHRyb2CFuQdrsarVjFujAldwa4c/zFwoYhsAC4ECgBP00YRGQKcASz3OeYeYAwwHUgGftrSiVX1CVWdpqrT0tLSuizg7CJrimuMCW0RATx3AZDhs5zurjtKVffjXnGISDxwnaqW+exyA/Cqqjb4HHPAna0Tkadxkk+3OXqfcUscxpgQFcgrjjXAKBEZISJROEVOr/vuICKpItIUwz3AU83OsYhmxVTuVQjiNGm6BtjS9aG3Lqe4itT4KAbERnbn0xpjTI8RsMShqo3A93CKmbYDL6rqVhF5QESudnebCewUkV3AIOChpuNFJAvniuX9Zqf+u4h8AXwBpAIPBuo1tCS7uMqKqYwxIS2QRVWo6lJgabN19/nMvwS81MqxuZxYmY6qzu7aKDsmp7iKWaO7rs7EGGN6m2BXjvcqR2obKDpSZ304jDEhzRJHB+QWO11JrKjKGBPKLHF0QHZxJYDdh8MYE9IscXRATnEVIpCZHBfsUIwxJmgscXRATnEVwxJjiYkMD3YoxhgTNJY4OiDHmuIaY4wlDn+pKjlFVTZGlTEm5Fni8FNxZT1H6hrtisMYE/Iscfjp6H3G06wPhzEmtFni8FNOU1Ncu+IwxoQ4Sxx+yi6uIio8jKGJscEOxRhjgsoSh59yiqoYnhJHeJjdZ9wYE9oscfjJmuIaY4zDEocfPF7ly5JqRthQI8YYY4nDH/vLaqj3eK1i3BhjsMThl+ymprg2nLoxxlji8EdOkdMU1+o4jDHGEodfcoqrSIiOIDU+KtihGGNM0Fni8EN2cRUj0vohYk1xjTHGEocfrCmuMcYcY4mjHbUNHgrKaixxGGOMyxJHO74sqUbVKsaNMaaJJY52HBvc0JriGmMMWOJoV1MfjqxUu8+4McaAJY525RRVkZYQTUJMZLBDMcaYHsESRzusRZUxxhzPEkc7cortPuPGGOPLEkcbyqsbKKmqtysOY4zxYYmjDTklTYMbWuIwxpgmljjacLQprt2HwxhjjrLE0YacoirCBDKSrSmuMcY0scTRhuziKtKT4oiOCA92KMYY02NEBDuAnmzskP6kJ9nVhjHG+LLE0YbvzhoZ7BCMMabHsaIqY4wxHWKJwxhjTIdY4jDGGNMhljiMMcZ0SEATh4hcJiI7RWSPiNzdwvbhIvKuiGwWkVUiku6unyUiG32mWhG5xt02QkQ+c8/5gohEBfI1GGOMOV7AEoeIhAN/AOYC44BFIjKu2W6PAs+q6gTgAeBhAFVdqaqTVHUSMBuoBla4x/wGeExVRwKlwDcD9RqMMcacKJBXHDOAPaqarar1wBJgXrN9xgHvufMrW9gOsABYpqrVIiI4ieQld9szwDVdHbgxxpjWBTJxDAPyfJbz3XW+NgHXuvPzgQQRSWm2z0LgeXc+BShT1cY2zgmAiNwmImtFZG1RUdFJvgRjjDHNBbsD4I+B34vILcAHQAHgadooIkOAM4DlHT2xqj4BPOGep0hEvjzJGFOB4pM8tjtYfJ1j8XWOxdc5PT2+4S2tDGTiKAAyfJbT3XVHqep+3CsOEYkHrlPVMp9dbgBeVdUGd7kESBSRCPeq44RztkRV0072RYjIWlWddrLHB5rF1zkWX+dYfJ3T0+NrTSCLqtYAo9xWUFE4RU6v++4gIqki0hTDPcBTzc6xiGPFVKiq4tSFLHBX3Qz8MwCxG2OMaUXAEod7RfA9nGKm7cCLqrpVRB4Qkavd3WYCO0VkFzAIeKjpeBHJwrlieb/ZqX8K/EhE9uDUeTwZqNdgjDHmRAGt41DVpcDSZuvu85l/iWMtpJofm0sLFd+qmo3TYqu7PNGNz3UyLL7Osfg6x+LrnJ4eX4vEKf0xxhhj/GNDjhhjjOkQSxzGGGM6xBKHy49xtaLdsbH2uGNlZXVjbBkislJEtonIVhG5s4V9ZopIuc/4Xve1dK4AxpgrIl+4z722he0iIo+7799mEZnSjbGNbjb2WYWI3NVsn259/0TkKRE5JCJbfNYli8jbIrLbfUxq5dib3X12i8jN3Rjff4jIDvfv96qIJLZybJufhQDGt1hECnz+hpe3cmyb/+sBjO8Fn9hyRWRjK8cG/P3rNFUN+QkIB/YCpwBROD3axzXb5zvAn9z5hcAL3RjfEGCKO58A7GohvpnAm0F8D3OB1Da2Xw4sAwQ4C/gsiH/rg8DwYL5/wAXAFGCLz7pHgLvd+buB37RwXDKQ7T4mufNJ3RTfJUCEO/+bluLz57MQwPgWAz/24+/f5v96oOJrtv0/gfuC9f51drIrDoc/42rNwxkbC5yWYHPcsbMCTlUPqOp6d/4ITvPmFoda6cHm4Qxoqar6KU5HziFBiGMOsFdVT3YkgS6hqh8Ah5ut9v2MtTYO26XA26p6WFVLgbeBy7ojPlVdoceG+/kUpwNuULTy/vnDn//1TmsrPvd74wZ8+qj1NpY4HP6Mq3V0H/efpxynH0m3covIJgOftbD5bBHZJCLLRGR890aGAitEZJ2I3NbCdn/e4+7gO/ZZc8F8/wAGqeoBd/4gTt+m5nrK+3grzhVkS9r7LATS99yitKdaKerrCe/f+UChqu5uZXsw3z+/WOLoRcQZluVl4C5VrWi2eT1O8ctE4L+B17o5vPNUdQrOMPrfFZELuvn52+WOYHA18I8WNgf7/TuOOmUWPbKtvIj8DGgE/t7KLsH6LPwROBWYBBzAKQ7qiY4bEaMFPf5/yRKHo91xtXz3EZEIYADO2FndQkQicZLG31X1lebbVbVCVSvd+aVApIikdld8qlrgPh4CXuXETpr+vMeBNhdYr6qFzTcE+/1zFTYV37mPh1rYJ6jvozgDkl4JfNVNbifw47MQEKpaqKoeVfUCf27leYP9/kXgjM/3Qmv7BOv96whLHI52x9Vyl5tasCwA3mvtH6eruWWiTwLbVfW/WtlncFOdi4jMwPnbdktiE5F+IpLQNI9Tibql2W6vA193W1edBZT7FMt0l1Z/6QXz/fPh+xlrbRy25cAlIpLkFsVcwkmMHn0yROQy4CfA1apa3co+/nwWAhWfb53Z/Fae15//9UC6CNihqvktbQzm+9chwa6d7ykTTqufXTgtLn7mrnsA558EIAaniGMP8DlwSjfGdh5OscVmYKM7XQ58G/i2u8/3gK04rUQ+Bc7pxvhOcZ93kxtD0/vnG5/g3BFyL/AFMK2b/779cBLBAJ91QXv/cBLYAaABp5z9mzh1Zu8Cu4F3gGR332nAX3yOvdX9HO4BvtGN8e3BqR9o+gw2tTIcCixt67PQTfH9zf1sbcZJBkOax+cun/C/3h3xuev/2vSZ89m329+/zk425IgxxpgOsaIqY4wxHWKJwxhjTIdY4jDGGNMhljiMMcZ0iCUOY4wxHWKJw5guICIeOX4E3i4bdVVEsnxHWTUm2AJ661hjQkiNqk4KdhDGdAe74jAmgNx7Kzzi3l/hcxEZ6a7PEpH33AH53hWRTHf9IPdeF5vc6Rz3VOEi8mdx7seyQkRig/aiTMizxGFM14htVlR1o8+2clU9A/g98Ft33X8Dz6jqBJzBAh931z8OvK/OYItTcHoPA4wC/qCq44Ey4LqAvhpj2mA9x43pAiJSqarxLazPBWararY7UOVBVU0RkWKcITEa3PUHVDVVRIqAdFWt8zlHFs49OEa5yz8FIlX1wW54acacwK44jAk8bWW+I+p85j1Y/aQJIkscxgTejT6Pn7jzH+OMzArwVWC1O/8ucAeAiISLyIDuCtIYf9mvFmO6RqyIbPRZ/peqNjXJTRKRzThXDYvcdd8HnhaRfwOKgG+46+8EnhCRb+JcWdyBM8qqMT2G1XEYE0BuHcc0VS0OdizGdBUrqjLGGNMhdsVhjDGmQ+yKwxhjTIdY4jDGGNMhljiMMcZ0iCUOY4wxHWKJwxhjTIf8f8WFQw4Hf6ILAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# Draw accuracy values for training & validation\n",
    "plt.plot(history.global_history['multiclassaccuracy'])\n",
    "plt.plot(history.global_history['val_multiclassaccuracy'])\n",
    "plt.title('FLModel accuracy')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend(['Train', 'Valid'], loc='upper left')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e330d34",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13 ('3.8')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "ae1fdd5fd034b7d694352220485921694ff89198520409089b4646721fce11ca"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}