Doc tweaks for paper resubmission

docs/images/Kiosk_Architecture.xml

@@ -1 +1 @@
-<mxfile modified="2020-03-23T21:49:19.823Z" host="app.diagrams.net" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:74.0) Gecko/20100101 Firefox/74.0" etag="R6B_OFPcHXYypOXCqkrY" version="12.9.0" type="device"><diagram id="f106602c-feb2-e66a-4537-3a34d633f6aa" name="Page-1">7Vxrd6I4GP41fiyHEAjwsWrbnZ2Zczrb2euXHoSoTNG4gK3ur98EEiQkqFWsbad6zox5cyHkvTxPXkJ7cDBb3aTBYvqVRDjpWWa06sFhz7Icz6L/MsG6FNimXwomaRyVIrAR3MX/YS40uXQZRziTGuaEJHm8kIUhmc9xmEuyIE3Jk9xsTBL5qotgghXBXRgkqvTPOMqnXAqQv6n4BceTKb+0Z7llxSgIHyYpWc759XoWHBefsnoWiLH4jWbTICJPNRG86sFBSkhe/pqtBjhhSyuWrex33VJbzTvF83yfDlxPj0Gy5Lc+xHjBGw3xIiHrGRvKMj/HJHug/1+m4TTO6aIvU8zvIV+LdRvHSTIgCUmLIrSHl+Da6cF+lqfkAYuaOZljJhT3DmhhkgZRTK/UaDMm85ybB7BpOUjiyZwWEjzO2RCLIIznky9FaeiYvEdtCnz1YX+azxJ+LXWR+Lo94jTHq5qIL9oNJjOcp2vahNd6Ttmjsu+y+LQxFs/mTaY1OxHtAm6fk2rgjY7oD64mvco8RWWKGnBErZkXcTIiT1cbgbSklrwweB5dMu+hRULHLyXXcSLqC8PGbB5mpVThH1bZmo+MivogFVdiuouCbNroXLcVumLXkA3CJkxlTBlxWMyiVWHsRreqq6YQYPLFT3ES5PGj7Oo6jfDhbklceABXveWaku6h7xs+9B3fhcCxIbKgPGJGlmmI+SB1V2yO65kNm2oYC13OCc6VgQp7qW55LxOCqtv3WdDEqWJKip6i4sPsSPb1ytEkt5ZtRLYhVgpGGUmWOaZRRdgkk1YlZlVJMMLJLcniPCbM90NqBXSmsC8s5EujwSyOosLURbRQelzyihHJczJTAokmjvjFR+M+Tywa3tHuTPJE0XAzWp+PXtxFB0GnsmBuIZbtGo4SeIR51uOOCFfHxB0IdwceycU5OAejotaUQ03DelyHfbcgxYKNMVtNGN0wJuHCMpJgkRO22nRN89bIXpn63qtsatbU0aypWI8jw8mFJUMJ8IyGssh4nOFjXZ6ip6o9lDAQHdEfE/bjhpAJ1RWjDGUNHbSqHCRkGdG6W3q3Y5LOqt6paCEkUfzIZs9djEr/XTI60+deWJWb/ZhnMXMSbFC0A+1dYiEoqQpmExwxXeZTdh8PjLNcUDukQQbXbipujkNl5dUVcXEzQrqd7Vwj9t3Jdsxj2Q6r1UQoF7AvlftX7KuJdqW78FG+s8LwwhaX5bdlKsHQbnhuF4HMlwNZFdjqYczS+ZzXAYECzu5Apo04k8I/7kPmCfcL4QdKLDveDo5Z4toSCiZSX0GrGxLUhCKnoZdOYpan6qXObEmaT8mEzIOkTm9ljiqRW3rhv5jccETx74qmUAwR1DdMgiyLQyGu8d8fOM/X3EGDZU6oaDOJL6TwKKAovk5uWzVbskS+mDxWl4SPy9CejHdvxe7rLsg5jRbqOihVImlBuEtNBWYnKqg4ytZgJukDqfpA9rn0IWYjIzm9w9WlDqReAYHvADGgSzdZPtp8ZPxAnoEc5PnIcYBr2UBlcAAhw4W1j6XGRgQME9q2Z7mOhywTeR0oy22lXZzuhJUGNiQHmsVH5T03t7+zFAy1+4wlq9Id/GXDzQZktqBaop2v5pN4rmZuDmbuDuB+tT9zD8vJ3ONyKhKDb2720lI9++/1OFdqcp+qXSNl9IydnmrGlSvutuN1++bCtjVEx+8EpuWtxQXwT7K38IBiTi+BEGfHaWRpcPpsuJBE9g+4+GMKwMN38xu+uXv4p38BTHS0brRLp1niPXVYFIereuVwLUqrOK9RNFoSDI393nRiBdGnXYlNs2ixnoN1TYLbmCw/x5B8HQ2CNADOp88XAJlnY2pqWk+lA7qAHJElDfW/FU9S1PU8ljB0AP+W1YB/VwpsluMYNqAcAYISwJGK/55p1AcQI0j47xsFe6gG6kAluqTZ4fh/ndJOeB61J18aTOAD90+B+5Wj/dy47/onwf2qcAbcrwCoBj8bMNID0DZA2wtbXM3+8nw8ogXVjs8CHMIjDuAD2+xom6pehkfodP3aeISryzC8Cx6BGolny7cMqEkW6J6hIceAHaxttwmBAZlny9lHIuDMhMD9SATUQWtr4HgX+UgbmYZdTyfajbhiG7U8oo9eT0JS9/Dr8PjzKxnRLt+WeIn3DkG/4SjOaK9hQANLkJ05BBUP8yI6lSwn6VuOQZXz/dwxCKkxqHk4YJNFv+NZ9IYBPj+k1GxSa3stD3qlh8GS2Tf13bQL8Yx/h6VJj/x7/OCA5gRA4wk2vHKKc5tdUC5gGnK2BkD1aT9FUMPRPK4G0DNgB3kYPcUWT63fIdF15IRZQwOUy7rOBpw06OS5hllPlyFVN65j1BHOOdneU4yswSzpeM7hMCYOsN/h9HEnmZZPBX3H84yk1wnVZ9mfOdo5ES0vJjSmE7pP6Hb7DSPaDqd91lkY2zI04UWc0e4U2dwToJrrKTZ1gqMwZnty44Sptufmdo5JtW03qXpSxu38wI3edmj8NEzQ+mQD2gYCVYR1rQYWthwup/oL1rVmC9Yga5+E6zT2L6jx2kizvetsa09/lDM41Ny3Z8R+lu2ltMRQnIaT9pCuAXQPsti+8VRojNrR+BDovaPbLvY2lmX2l+EDC5T7ZrL4geRqgDNi7hSvPoUMMPuLtPwxLDeWGZ/cewNhtD8Iv+fNZsuZC/dosH7BMxf7PfdQqMGuo8p11BW5LekApdU1vh53UAbAt6W03Udljzzd1GLb3utXpam+Ivnxetvbe71Nlyc61cttLYbk7yacr++Vtx1O8ZO/CNcS/VXFfkR/zTJpNsqvLfoDdZf4Ef3fXvSHrmmY6pb3ZQEA6Pa6bxQAwP5btvcEALS4+XMqZfPNn6yBV/8D</diagram></mxfile>
+<mxfile modified="2020-08-20T06:53:55.265Z" host="app.diagrams.net" agent="5.0 (X11)" etag="pjCZdwwoV25vgs_PPxLP" version="13.6.3" type="device"><diagram id="f106602c-feb2-e66a-4537-3a34d633f6aa" name="Page-1">7V3ddps4F32aXJolIRBwmd+2k3Ym07TfdHqThY1sM8GWB+Mknqf/JBAYCWFjQxKcxl6rNQIE6Oyzz4+OyAk6nz19iP3F9AsNSHRiguDpBF2cmKblOOxf3rDOGkwMs4ZJHAZZU6nhNvyPiEYgWldhQJbSgQmlURIu5MYRnc/JKJHa/Dimj/JhYxrJV134E1JpuB35UbX1rzBIpqIVYm+z4yMJJ1NxadcUDzz0R/eTmK7m4nonJhqnn2z3zM/7Eg+6nPoBfSw1ocsTdB5TmmS/Zk/nJOJDmw9bdt5Vzd7ivmMyT5qcYGYnPPjRSjz6BSELcdAFWUR0PeNdmeA6pMt79v9pPJqGCRv0VUzEMyTrfNzGYRSd04jG6SayLk7hlX2CzpZJTO9JvmdO54Q35s8O2cYk9oOQXUk5ZkzniYAHtNi2H4WTOduIyDjhXSz8UTiffE63Lmwgzijdghh9dDZNZpG4VnWQxLg9kDghT6UmMWgfCJ2RJF6zQ8Re187OEPi2hDgfN2DxgDhkWsJJfpwv8DkpOt7IiP0QYtKLzK2IrCIGEjA0i00SDenj5aZBGlJTHhgyD0659rBNyvrPWq7CKN+fApvw+wCFUHP9MLOjRc843e/H+ZW47AJ/OVVOLmPFdK0rxDvhN8zauDDCUXoXtQLjD7pVXCWBQCAGPyaRn4QPsqrrJCK6u6FhqgFC9KYDJNkjzzM85Nmeg6BtIWwiucclXcUjIjopq6LarwsUTClgYcM5IUmloxQvxSM3ghCqqv2nGaO031jbTbSahPMKpCryCtIPx5Os84XCSeotY0XGEt/yh0sarRLC2CXHJm8ttji6In9Iohu6DJOQcg4YMTQQdtWzHCmflQNmYRCkkM9Zo3LGqdgxpElCZxVC0fCJl340avTIWfGWnc5bHplV3PR2JnpPn6ID8vFkoDBza9gV/snRVKafnLXa0A9Cu/lH0nRho/1huhfIjKOAx7H5d4vBWPA+Zk8T7nUYk9HCNCJ/kVA+2GxIk1qCLxDfeJCBZkxtzZjm49GSVQambFGgayjCouPxkrTVfGZEq9LDEbelQ/Zjwn98oHTCZMU9h2wP67TYeR7RVZDShJ+MaTwrzo7zI/KWIHzgdy80jLX+u+JezZlQwmJbPY8rFodT7hTmx8H6U8K8IfNYCL/BIZdlMuXPcc9dlwHDIeMYUnqoUO2HtWVXrzSnD5O3bnd6rjD/lgkQtPVv+F4NFzmQf1m7d8m/Gl7LNEP08o1vXAys/LLiCUCF9ixFSbXK2AGPQYXICgtdUjlHx2PI7cCPgvZuItMyziTVj7sR14S7Ra4HFS7b4GCr87sFHG2GuDSEuUNSHkGzG1+okNhaJshOOcvd7uDSOJnSCZ37UdnLlV1VycdlF/7B2w3XxHnD32mD53q535r7waPIXy7DUd5ccob/IUmyFrrrrxLKmja38pmmylbVHsnT5Ve+eBL3mG2ty1s3JA7ZQHGXJWt8CpPs1pGFxDa/8wEwAMwbNh3yjXVpQ+2uFl+ZyypEKixG5n2KNtzQ/W4Mr6ZKi+3nwYKMA1tFQa60JQiATiBQeEpb41JJHrgqD2y9ljzyu5H9CfaET6c6U9mDKKIDu4UcFvF5ePORrRh2DWxj18O2DR3TglU/EmJsOKj0MasMjaEBkGW5pmO72ATY7UBYTq3zJ5yuUSGBjauFQPqpel8fbr7zfBDD/ZJnzuIdXtTGQ7xeDUk8Z0S05I4kDbqLH2wo9Kp5/DCiswWDzB2Zs7CXyHGEGnHGmXiaB5zCjVPdsuI4JX+1R7hZhXGhirtxLJCqC3EsS+NueZ04C3KAM4Des0Q4LqzAqQsLwU0sKFoyM4Hwy7oLwvrDkuUHRhoub7H7+/oYjYzQx7tlOL27Al/N4Out9+Ov03n4+8DVeQqdWyY9tiyk5EQ8xROtycIxufnr0mELfsCyJQSjwPoHLf43hfD+G/iTfLi9/3k2gAC3BqYWMBpgNXRxtmKjcDXtEtzgVqzVQ1dVhhqd6RZ7M/wdou/JN+cs9u+/jdHPi8vbgYteCJDMxFsQFB8owZPtYR67Qn/d5Xcd71k4EFYh9OLkB07K5Hc49zWCmqPxtbtntKbuG/VvQrq6DhH9Mjz3Yx/an64HsIOIqA2tqCZxx7C3pogNL5nNiakFr7g94xWMkGG7G14xD+aVSteOLXtolSxbjQ09gKBsXRr6TcaKFgaGVQ71LMVPsYxSjOfh/gSLuvTo4cHiVxKEPNz7jQ7Zv3+uyIocabyYpn0DP/GXCY2POmC0GoP5LQeMuMpF6jTSJtNxKzIdCgD3p5Zd0ys1UwLStIEEe1XeKi7yKaIdSJNmjE7EvJNmAkmZ60CXdlro0wFjYmbDHEnyEFXnhRDwDM28BkSukVNsG/Zz3GfxoJktAKbkMb2oD90o3tvDr+o6gWBWnS2n86kGPdcws8k8p01KV0YgG0QDQ2Gh06xuI8/oJbMLzquGAXtmF5qFARXl2DUvWkYY0oAJm12DqSmf6IUG0XEJbfeMWL1gGwmtBtuaIKxvogTVsswzXoescRZ6ENi8l9LpS+l0nsZzFdLV4MjThF1dhTvPVl63Qyd+8aK7GvKvCvad/DXDBPpP/vk9lpT2nM5m/jwYRHxS2QSfOFuOOaO9W4OjsQbIAUaeYH01gwBhBVtHaxAKPfnFDcLWGYQm9aijNeOVINU4nXqUBD7M2ODzsGgoVqj9sUpSdtLRhrw2i+ckCnTNeHYi/b1T+zOgHE4v6ZnMyrEnVaFZzkiltueeJKNpnSG6xPzbDUkgl+fjmxAFxBqkeptD25CFHkLaaryDc/a3CY35QkkTnK1G9xz1DRP2+SKBooP3fH0H+fr9554AL1qqm3qyANKBdls+vw1gh1cLe06t39cemg7s4b/eEl0UZRv9mLx2OslaHVLf0pvpZRtDldwsGxkoT8PuP6esLGpEzVZHHmBU9QDrtF785atdDM8uF/pDprHeybPU+TfOyOvHufOIrC5NL1OaDFXTtg3LTIuukeMhCOxGYKtP09fchLoCGCuL7Pc7nv3I7qDubFc+Wy3L6K5u7Gn68wt2f5x+mcT/XYOfH6/HPx4GHSy/aZWnMFzTLvM01wHruZn64HKzg3Rvq0q9RjltOgermAHHNoBtNYJeFcO24nvwghW1Tqk7U7CVy/Ze7xfQFfNNv6bvGakiq226psap02Cmfsbcg4aF6+YuoQMNG20KjDQVRq5jAIwrC1rKLiCTfrlKye4gl6aXki4KLi0o1q4V3iuq+UbmSxpfRUwmLEIh8QMvdNhnAXCvypGS9GnG7GnuIkaxRxbe7ID9Fj2uKkM5hjF1tSGWQH6nVUmOfHon6SF9oa9dTTL3h7/aVfy4W/kL95S/asSky9e+E9gbJbAy7neo7V5r9o+cwbau0eklgbVywBhM5AXBSvUYZmGpiEjTKm2soTAW6uNqD1KRt2eUQltkWfVyb+eCdVv5fRWzk8g8qH9TjEJavaKno14efJB7Zemh/qZLwPscGrZyrUxzOzPZfWUmXB8HvpPQUZFQGb+Fov3aS062LjXsJQm1y08pb7ZEgHm3mjDO0UyXM9cJ1QuxnZ+jK7U8nGLO6Xy5mvF6X3B6pEvbjplmDvJ13F+PexxdjUhfaKZdGslR3wHy0jTjdPuCpXdG6Y3j4ry/XKmWRb0+J6ZbOS4OVhwX1zNcZ/PyBaAJl17YifHq080tKWfXW3HfKacfTowH9Bh+yzy0tWCilzzUyrNRq+j4Cyhfj4f0g99tvuadh3rj+mzXtV/KH2Kbmz/Nkx2++fNH6PL/</diagram></mxfile>

docs/source/DEVELOPER.rst

@@ -144,3 +144,32 @@ The Deepcell Kiosk uses Google Kubernetes Engine to requisition resources on Goo
 5. any Persistent Disks associated with your cluster
 
 While we hope this list is comprehensive, there could be some lingering resources used by Google Cloud and not deleted automatically that we're not aware of.
+
+Benchmarking the DeepCell Kiosk
+---------------------------------------------------------
+
+The DeepCell Kiosk comes with a utility for benchmarking the scalability and performance of a deep learning workflow. To reproduce the cost and timing benchmarks reported in the DeepCell Kiosk paper, please refer to the ``2020-Bannon_et_al-Kiosk`` folder of our `figure creation repository <https://github.com/vanvalenlab/publication-figures>`_. To run your own benchmarking, please read below.
+
+1. If you don't already have a cloud storage bucket for use with the DeepCell Kiosk, you should create one now. It's fine to reuse this bucket across multiple DeepCell Kiosk clusters.
+
+2. There are three variables in the benchmarking pod's YAML file, ``conf/helmfile.d/0410.benchmarking.yaml``, that may need to be customized before benchmarking:
+
+    - ``MODEL`` is the model name and version that will be used in benchmarking. The model you choose should be present in the ``models/`` folder of your benchmarking bucket. See the `Van Valen Lab's benchmarking bucket <https://console.cloud.google.com/storage/browser/kiosk-benchmarking>`_ for an example.
+    - ``FILE`` is the name of the file that will be used for benchmarking. A file by this name should be in your benchmarking bucket in the ``uploads/`` folder.    
+    - ``COUNT`` specifies how many times the ``FILE`` will be submitted to the cluster for processing.
+
+3. Deploy a DeepCell Kiosk as you normally would. While navigating the cluster configuration menu, pay special attention to two configuration settings:
+
+    - The bucket name you provide should be that of the benchmarking bucket from step 1.
+    - The Maximum Number of GPUs has a strong effect on benchmarking time by effectively limiting how large the cluster can scale.
+
+4. Once the cluster has deployed successfully, drop to the ``Shell`` via the DeepCell Kiosk main menu and begin the benchmarking process by executing the following command: ``kubectl scale deployment benchmarking --replicas=1``.
+
+5. Benchmarking jobs can take a day or more, depending on the conditions (# of images and max # of GPUs) chosen. To monitor the status of your benchmarking job, drop to the ``Shell`` within the DeepCell Kiosk main menu and execute the command ``stern benchmarking -s 10m``. This will show you the most recent log output from the `benchmarking` pod. When benchmarking has finished, the final line in the log should be ``Uploaded [FILEPATH] to [BUCKET] in [SECONDS] seconds.``, where ``[FILEPATH]`` is the location in ``[BUCKET]`` where the benchmarking data has been saved.
+
+6. Now that the benchmarking process has finished, clean up the benchmarking resources by executing ``kubectl scale deployment benchmarking --replicas 0`` at the DeepCell Kiosk's ``Shell``. This prevents the benchmarking process from executing multiple times in long-lived clusters.
+
+7. Finally, you can download and analyze your benchmarking data. Two top-level fields in this large JSON file that are of interest are:
+
+    - ``time_elapsed``: the exact running time of the benchmarking procedure (seconds)
+    - ``total_node_and_networking_costs``: a slight underestimate of the total costs of the benchmarking run. (This total does not include Storage, Operation, or Storage Egress Fees. These extra fees `can be calculated <https://github.com/vanvalenlab/publication-figures/blob/383a90149eb86d4a0a697395edffb32d383bb1ca/figure_generation/data_extractor.py#L318>`_ after the fact by using the `Google Cloud guidelines <https://cloud.google.com/vpc/network-pricing#general>`_.)