notebooks/split_fashionmnist_multihead.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TOC:\n",
    "* [Environment Setup](#setup)\n",
    "* [Results](#results)\n",
    "    * [S-FSVI](#res1)\n",
    "    * [S-FSVI (larger networks)](#res2)\n",
    "    * [S-FSVI (no coreset)](#res3)\n",
    "    * [FRCL (with random-choice coreset)](#res4)\n",
    "    * [FROMP (with lambda-descend coreset)](#res5)\n",
    "    * [VCL (random-choice coreset)](#res6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Environment Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Run as Colab notebook\n",
    "\n",
    "**Important: Before connecting to a kernel, select a GPU runtime. To do so, open the `Runtime` tab above, click `Change runtime type`, and select `GPU`. Run the setup cell below only after you've done this.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# pull S-FSVI repository\n",
    "!git clone https://github.com/timrudner/S-FSVI.git\n",
    "# patch required packages\n",
    "!pip install -r ./S-FSVI/colab_requirements.txt"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**After successfully running the cell above, you need to restart the runtime. To do so, open the “Runtime” tab above and and click “Restart runtime”. Once the runtime was restarted, run the cell below. There is no need to re-run the installation in the cell above.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# add the repo to path\n",
    "import os\n",
    "import sys\n",
    "root = os.path.abspath(os.path.join(os.getcwd(), \"S-FSVI\"))\n",
    "if root not in sys.path:\n",
    "    sys.path.insert(0, root)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Run as Jupyter notebook (-->skip ahead to “Results” if you are running this as a Colab notebook<--)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Install conda environment `fsvi`"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!conda env update -f ../environment.yml"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Troubleshooting:\n",
    "\n",
    " - In case there is an error when installing sklearn: run `pip install Cython==0.29.23` manually and then run the above command again.\n",
    " - In case you have access to a GPU, see instructions [here](https://github.com/google/jax#pip-installation-gpu-cuda) for installing the GPU version of `jaxlib`. This will make the experiment run significantly faster."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Run the command below to install the conda environment as a kernel of the jupyter notebook. Then switch to this kernel using the Jupyter Notebook menu bar by selecting `Kernel`, `Change kernel`, and then selecting `fsvi`."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!python -m ipykernel install --user --name=fsvi"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Troubleshooting: For further details, see [here](https://medium.com/@nrk25693/how-to-add-your-conda-environment-to-your-jupyter-notebook-in-just-4-steps-abeab8b8d084)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "# assuming os.getcwd() returns the directory containing this jupyter notebook\n",
    "root = os.path.abspath(os.path.join(os.getcwd(), \"..\"))\n",
    "if root not in sys.path:\n",
    "    sys.path.insert(0, root)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results <a name=\"results\"></a>\n",
    "\n",
    "To read a model checkpoint instead of training the model from scratch, pass load_chkpt=True to the function read_config_and_run .\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/scratch/timner/applications/anaconda3/envs/fsvi/lib/python3.8/site-packages/tensorflow_addons/utils/ensure_tf_install.py:53: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.4.0 and strictly below 2.7.0 (nightly versions are not supported). \n",
      " The versions of TensorFlow you are currently using is 2.8.0 and is not supported. \n",
      "Some things might work, some things might not.\n",
      "If you were to encounter a bug, do not file an issue.\n",
      "If you want to make sure you're using a tested and supported configuration, either change the TensorFlow version or the TensorFlow Addons's version. \n",
      "You can find the compatibility matrix in TensorFlow Addon's readme:\n",
      "https://github.com/tensorflow/addons\n",
      "  warnings.warn(\n",
      "/home/scratch/timner/applications/anaconda3/envs/fsvi/lib/python3.8/site-packages/sklearn/feature_extraction/image.py:167: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  dtype=np.int):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: TensorFlow is set to only use CPU.\n",
      "Jax is running on gpu\n",
      "WARNING: TensorFlow is set to only use CPU.\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%config Completer.use_jedi = False\n",
    "import os\n",
    "import sys\n",
    "root = os.path.abspath(os.path.join(os.getcwd(), \"..\"))\n",
    "if root not in sys.path:\n",
    "    sys.path.insert(0, root)\n",
    "\n",
    "\n",
    "from notebooks.nb_utils.common import read_config_and_run, show_final_average_accuracy\n",
    "import sfsvi.exps.utils.load_utils as lutils\n",
    "\n",
    "task_sequence = \"sfashionmnist_mh\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## S-FSVI <a name=\"res1\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading experiments: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 997.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading from cache:\n",
      "Running on clpc158.cs.ox.ac.uk\n",
      "Jax is running on gpu\n",
      "\n",
      "\n",
      "Input arguments:\n",
      " {\n",
      "    \"command\":\"cl\",\n",
      "    \"data_training\":\"continual_learning_sfashionmnist\",\n",
      "    \"data_ood\":[\n",
      "        \"not_specified\"\n",
      "    ],\n",
      "    \"model_type\":\"fsvi_mlp\",\n",
      "    \"optimizer\":\"adam\",\n",
      "    \"optimizer_var\":\"not_specified\",\n",
      "    \"momentum\":0.0,\n",
      "    \"momentum_var\":0.0,\n",
      "    \"schedule\":\"not_specified\",\n",
      "    \"architecture\":[\n",
      "        256,\n",
      "        256\n",
      "    ],\n",
      "    \"activation\":\"relu\",\n",
      "    \"prior_mean\":\"0.0\",\n",
      "    \"prior_cov\":\"0.001\",\n",
      "    \"prior_covs\":[\n",
      "        0.0\n",
      "    ],\n",
      "    \"prior_type\":\"bnn_induced\",\n",
      "    \"epochs\":60,\n",
      "    \"start_var_opt\":0,\n",
      "    \"batch_size\":128,\n",
      "    \"learning_rate\":0.0005,\n",
      "    \"learning_rate_var\":0.001,\n",
      "    \"dropout_rate\":0.0,\n",
      "    \"regularization\":0.0,\n",
      "    \"inducing_points\":0,\n",
      "    \"n_marginals\":1,\n",
      "    \"n_condition\":128,\n",
      "    \"inducing_input_type\":\"uniform_rand\",\n",
      "    \"inducing_input_ood_data\":[\n",
      "        \"not_specified\"\n",
      "    ],\n",
      "    \"inducing_input_ood_data_size\":50000,\n",
      "    \"kl_scale\":\"equal\",\n",
      "    \"feature_map_jacobian\":false,\n",
      "    \"feature_map_jacobian_train_only\":false,\n",
      "    \"feature_map_type\":\"not_specified\",\n",
      "    \"td_prior_scale\":0.0,\n",
      "    \"feature_update\":1,\n",
      "    \"full_cov\":false,\n",
      "    \"n_samples\":5,\n",
      "    \"n_samples_eval\":5,\n",
      "    \"tau\":1.0,\n",
      "    \"noise_std\":1.0,\n",
      "    \"ind_lim\":\"ind_-1_1\",\n",
      "    \"logging_frequency\":10,\n",
      "    \"figsize\":[\n",
      "        \"10\",\n",
      "        \"4\"\n",
      "    ],\n",
      "    \"seed\":8,\n",
      "    \"save\":false,\n",
      "    \"name\":\"\",\n",
      "    \"evaluate\":false,\n",
      "    \"resume_training\":false,\n",
      "    \"no_final_layer_bias\":false,\n",
      "    \"extra_linear_layer\":false,\n",
      "    \"map_initialization\":false,\n",
      "    \"stochastic_linearization\":false,\n",
      "    \"grad_flow_jacobian\":false,\n",
      "    \"stochastic_prior_mean\":\"not_specified\",\n",
      "    \"batch_normalization\":false,\n",
      "    \"batch_normalization_mod\":\"not_specified\",\n",
      "    \"final_layer_variational\":false,\n",
      "    \"kl_sup\":\"not_specified\",\n",
      "    \"kl_sampled\":false,\n",
      "    \"fixed_inner_layers_variational_var\":false,\n",
      "    \"init_logvar\":[\n",
      "        0.0,\n",
      "        0.0\n",
      "    ],\n",
      "    \"init_logvar_lin\":[\n",
      "        0.0,\n",
      "        0.0\n",
      "    ],\n",
      "    \"init_logvar_conv\":[\n",
      "        0.0,\n",
      "        0.0\n",
      "    ],\n",
      "    \"perturbation_param\":0.01,\n",
      "    \"logroot\":\"ablation\",\n",
      "    \"subdir\":\"reproduce_main_results_2\",\n",
      "    \"wandb_project\":\"not_specified\",\n",
      "    \"n_inducing_inputs\":40,\n",
      "    \"n_inducing_inputs_first_task\":\"not_specified\",\n",
      "    \"n_inducing_inputs_second_task\":\"not_specified\",\n",
      "    \"inducing_inputs_bound\":[\n",
      "        0.0,\n",
      "        1.0\n",
      "    ],\n",
      "    \"fix_shuffle\":false,\n",
      "    \"use_generative_model\":false,\n",
      "    \"inducing_inputs_add_mode\":0,\n",
      "    \"inducing_input_adjustment\":false,\n",
      "    \"not_use_coreset\":false,\n",
      "    \"inducing_input_augmentation\":false,\n",
      "    \"plotting\":false,\n",
      "    \"logging\":1,\n",
      "    \"use_val_split\":false,\n",
      "    \"not_use_val_split\":true,\n",
      "    \"coreset\":\"random\",\n",
      "    \"coreset_entropy_mode\":\"soft_highest\",\n",
      "    \"coreset_entropy_offset\":\"0.0\",\n",
      "    \"coreset_kl_heuristic\":\"lowest\",\n",
      "    \"coreset_kl_offset\":\"0.0\",\n",
      "    \"coreset_elbo_heuristic\":\"lowest\",\n",
      "    \"coreset_elbo_offset\":\"0.0\",\n",
      "    \"coreset_elbo_n_samples\":5,\n",
      "    \"coreset_n_tasks\":\"not_specified\",\n",
      "    \"coreset_entropy_n_mixed\":1,\n",
      "    \"n_coreset_inputs_per_task\":\"not_specified\",\n",
      "    \"full_ntk\":false,\n",
      "    \"constant_inducing_points\":false,\n",
      "    \"n_permuted_tasks\":10,\n",
      "    \"n_omniglot_tasks\":20,\n",
      "    \"epochs_first_task\":60,\n",
      "    \"identity_cov\":false,\n",
      "    \"n_epochs_save_params\":\"not_specified\",\n",
      "    \"n_augment\":\"not_specified\",\n",
      "    \"augment_mode\":\"constant\",\n",
      "    \"n_valid\":\"same\",\n",
      "    \"learning_rate_first_task\":\"not_specified\",\n",
      "    \"save_alt\":true,\n",
      "    \"save_first_task\":false,\n",
      "    \"first_task_load_exp_path\":\"not_specified\",\n",
      "    \"only_task_id\":\"not_specified\",\n",
      "    \"loss_type\":1,\n",
      "    \"only_trainable_head\":false,\n",
      "    \"n_omniglot_coreset_chars\":2,\n",
      "    \"omniglot_randomize_test_split\":false,\n",
      "    \"omniglot_randomize_task_sequence\":false,\n",
      "    \"n_inducing_input_adjust_amount\":\"not_specified\",\n",
      "    \"save_all_params\":false,\n",
      "    \"task\":\"continual_learning_sfashionmnist\",\n",
      "    \"architecture_arg\":\"fc_256_256\",\n",
      "    \"init_logvar_minval\":0.0,\n",
      "    \"init_logvar_maxval\":0.0,\n",
      "    \"init_logvar_lin_minval\":0.0,\n",
      "    \"init_logvar_lin_maxval\":0.0,\n",
      "    \"init_logvar_conv_minval\":0.0,\n",
      "    \"init_logvar_conv_maxval\":0.0\n",
      "} \n",
      "\n",
      "\n",
      "MAP initialization: False\n",
      "Full NTK computation: False\n",
      "Stochastic linearization (posterior): False\n",
      "init_logvar_lin_minval: -10.0\n",
      "init_logvar_lin_maxval: -8.0\n",
      "Full NTK computation: False\n",
      "Stochastic linearization (prior): False\n",
      "\n",
      "\n",
      "Learning task 1\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------\n",
      "  epoch    mean acc    t1 acc\n",
      "-------  ----------  --------\n",
      "      0       95.25     95.25\n",
      "      1       97.70     97.70\n",
      "      2       98.25     98.25\n",
      "      3       98.35     98.35\n",
      "      4       98.60     98.60\n",
      "      5       98.70     98.70\n",
      "      6       98.90     98.90\n",
      "      7       98.90     98.90\n",
      "      8       98.95     98.95\n",
      "      9       99.00     99.00\n",
      "     10       99.00     99.00\n",
      "     11       98.95     98.95\n",
      "     12       99.00     99.00\n",
      "     13       99.00     99.00\n",
      "     14       99.20     99.20\n",
      "     15       99.30     99.30\n",
      "     16       99.00     99.00\n",
      "     17       99.20     99.20\n",
      "     18       99.10     99.10\n",
      "     19       99.15     99.15\n",
      "     20       99.20     99.20\n",
      "     21       99.40     99.40\n",
      "     22       99.30     99.30\n",
      "     23       99.20     99.20\n",
      "     24       99.10     99.10\n",
      "     25       98.95     98.95\n",
      "     26       99.25     99.25\n",
      "     27       99.25     99.25\n",
      "     28       98.85     98.85\n",
      "     29       98.80     98.80\n",
      "     30       99.20     99.20\n",
      "     31       99.25     99.25\n",
      "     32       99.40     99.40\n",
      "     33       99.30     99.30\n",
      "     34       99.05     99.05\n",
      "     35       99.15     99.15\n",
      "     36       99.25     99.25\n",
      "     37       99.00     99.00\n",
      "     38       99.00     99.00\n",
      "     39       99.05     99.05\n",
      "     40       99.45     99.45\n",
      "     41       99.35     99.35\n",
      "     42       99.35     99.35\n",
      "     43       99.20     99.20\n",
      "     44       99.35     99.35\n",
      "     45       99.25     99.25\n",
      "     46       99.30     99.30\n",
      "     47       99.35     99.35\n",
      "     48       99.30     99.30\n",
      "     49       99.30     99.30\n",
      "     50       99.40     99.40\n",
      "     51       99.35     99.35\n",
      "     52       99.30     99.30\n",
      "     53       99.35     99.35\n",
      "     54       99.25     99.25\n",
      "     55       99.25     99.25\n",
      "     56       99.35     99.35\n",
      "     57       99.30     99.30\n",
      "     58       99.35     99.35\n",
      "     59       99.35     99.35\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9935 \n",
      "Accuracies (test): [0.9935]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 2\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc\n",
      "-------  ----------  --------  --------\n",
      "      0       97.40     98.70     96.10\n",
      "      1       97.52     98.75     96.30\n",
      "      2       97.72     98.90     96.55\n",
      "      3       97.80     98.70     96.90\n",
      "      4       97.83     98.85     96.80\n",
      "      5       97.97     98.75     97.20\n",
      "      6       97.90     98.75     97.05\n",
      "      7       97.85     98.80     96.90\n",
      "      8       97.88     98.80     96.95\n",
      "      9       98.10     98.90     97.30\n",
      "     10       98.28     98.90     97.65\n",
      "     11       98.20     98.85     97.55\n",
      "     12       98.17     98.85     97.50\n",
      "     13       98.25     98.85     97.65\n",
      "     14       98.35     98.95     97.75\n",
      "     15       98.28     98.85     97.70\n",
      "     16       98.28     98.95     97.60\n",
      "     17       98.30     98.95     97.65\n",
      "     18       98.32     98.90     97.75\n",
      "     19       98.32     98.85     97.80\n",
      "     20       98.40     99.00     97.80\n",
      "     21       98.30     98.95     97.65\n",
      "     22       98.25     99.00     97.50\n",
      "     23       98.35     98.85     97.85\n",
      "     24       98.43     99.00     97.85\n",
      "     25       98.43     99.00     97.85\n",
      "     26       98.43     99.00     97.85\n",
      "     27       98.35     98.90     97.80\n",
      "     28       98.45     99.00     97.90\n",
      "     29       98.32     98.85     97.80\n",
      "     30       98.43     98.95     97.90\n",
      "     31       98.40     99.00     97.80\n",
      "     32       98.35     98.95     97.75\n",
      "     33       98.28     98.85     97.70\n",
      "     34       98.43     99.00     97.85\n",
      "     35       98.35     98.75     97.95\n",
      "     36       98.35     99.00     97.70\n",
      "     37       98.40     99.00     97.80\n",
      "     38       98.28     99.00     97.55\n",
      "     39       98.17     98.90     97.45\n",
      "     40       98.15     98.90     97.40\n",
      "     41       98.35     98.85     97.85\n",
      "     42       98.23     98.85     97.60\n",
      "     43       98.20     98.80     97.60\n",
      "     44       98.35     98.90     97.80\n",
      "     45       98.40     98.90     97.90\n",
      "     46       98.32     98.90     97.75\n",
      "     47       98.38     98.90     97.85\n",
      "     48       98.40     98.90     97.90\n",
      "     49       98.38     98.80     97.95\n",
      "     50       98.32     98.75     97.90\n",
      "     51       98.38     98.85     97.90\n",
      "     52       98.30     98.80     97.80\n",
      "     53       98.28     98.80     97.75\n",
      "     54       98.40     98.95     97.85\n",
      "     55       98.38     98.95     97.80\n",
      "     56       98.20     98.90     97.50\n",
      "     57       98.25     98.80     97.70\n",
      "     58       98.43     98.95     97.90\n",
      "     59       98.35     98.85     97.85\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9835 \n",
      "Accuracies (test): [0.9885, 0.9785]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 3\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc    t3 acc\n",
      "-------  ----------  --------  --------  --------\n",
      "      0       98.77     98.85     97.50     99.95\n",
      "      1       98.72     98.90     97.30     99.95\n",
      "      2       98.68     98.80     97.30     99.95\n",
      "      3       98.77     98.85     97.50     99.95\n",
      "      4       98.77     98.85     97.45    100.00\n",
      "      5       98.75     98.85     97.40    100.00\n",
      "      6       98.72     98.85     97.30    100.00\n",
      "      7       98.80     98.80     97.60    100.00\n",
      "      8       98.82     98.95     97.50    100.00\n",
      "      9       98.80     98.90     97.50    100.00\n",
      "     10       98.68     98.80     97.25    100.00\n",
      "     11       98.77     98.85     97.45    100.00\n",
      "     12       98.72     98.80     97.35    100.00\n",
      "     13       98.63     98.75     97.15    100.00\n",
      "     14       98.60     98.75     97.05    100.00\n",
      "     15       98.70     98.75     97.35    100.00\n",
      "     16       98.73     98.80     97.40    100.00\n",
      "     17       98.75     98.90     97.35    100.00\n",
      "     18       98.77     98.90     97.40    100.00\n",
      "     19       98.77     98.85     97.45    100.00\n",
      "     20       98.77     98.80     97.50    100.00\n",
      "     21       98.73     98.75     97.45    100.00\n",
      "     22       98.77     98.80     97.50    100.00\n",
      "     23       98.63     98.75     97.15    100.00\n",
      "     24       98.70     98.75     97.35    100.00\n",
      "     25       98.82     98.85     97.60    100.00\n",
      "     26       98.72     98.75     97.40    100.00\n",
      "     27       98.77     98.80     97.50    100.00\n",
      "     28       98.75     98.80     97.45    100.00\n",
      "     29       98.72     98.85     97.30    100.00\n",
      "     30       98.73     98.75     97.45    100.00\n",
      "     31       98.70     98.80     97.30    100.00\n",
      "     32       98.73     98.80     97.40    100.00\n",
      "     33       98.62     98.80     97.05    100.00\n",
      "     34       98.77     98.90     97.40    100.00\n",
      "     35       98.78     98.80     97.55    100.00\n",
      "     36       98.67     98.85     97.15    100.00\n",
      "     37       98.75     98.95     97.30    100.00\n",
      "     38       98.75     98.90     97.35    100.00\n",
      "     39       98.67     98.85     97.15    100.00\n",
      "     40       98.72     98.85     97.30    100.00\n",
      "     41       98.73     98.85     97.35    100.00\n",
      "     42       98.73     98.85     97.35    100.00\n",
      "     43       98.77     98.85     97.45    100.00\n",
      "     44       98.77     98.90     97.40    100.00\n",
      "     45       98.68     98.85     97.20    100.00\n",
      "     46       98.72     98.85     97.30    100.00\n",
      "     47       98.75     98.90     97.35    100.00\n",
      "     48       98.73     98.90     97.30    100.00\n",
      "     49       98.70     98.80     97.30    100.00\n",
      "     50       98.63     98.80     97.10    100.00\n",
      "     51       98.75     98.90     97.35    100.00\n",
      "     52       98.75     98.80     97.50     99.95\n",
      "     53       98.68     98.85     97.25     99.95\n",
      "     54       98.75     98.80     97.45    100.00\n",
      "     55       98.77     98.90     97.40    100.00\n",
      "     56       98.73     98.80     97.40    100.00\n",
      "     57       98.75     98.85     97.40    100.00\n",
      "     58       98.72     98.90     97.25    100.00\n",
      "     59       98.78     98.90     97.45    100.00\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9878 \n",
      "Accuracies (test): [0.989, 0.9745, 1.0]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 4\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc    t3 acc    t4 acc\n",
      "-------  ----------  --------  --------  --------  --------\n",
      "      0       99.10     98.90     97.50    100.00    100.00\n",
      "      1       98.96     98.75     97.10    100.00    100.00\n",
      "      2       98.99     98.80     97.20     99.95    100.00\n",
      "      3       99.03     98.70     97.45     99.95    100.00\n",
      "      4       99.01     98.85     97.25     99.95    100.00\n",
      "      5       99.03     98.90     97.25     99.95    100.00\n",
      "      6       99.06     98.85     97.45     99.95    100.00\n",
      "      7       99.01     98.80     97.35     99.90    100.00\n",
      "      8       99.04     98.90     97.35     99.90    100.00\n",
      "      9       98.99     98.80     97.25     99.90    100.00\n",
      "     10       99.05     98.80     97.50     99.90    100.00\n",
      "     11       98.90     98.70     97.00     99.90    100.00\n",
      "     12       99.05     98.85     97.45     99.90    100.00\n",
      "     13       99.02     98.80     97.40     99.90    100.00\n",
      "     14       98.98     98.80     97.20     99.90    100.00\n",
      "     15       98.96     98.75     97.20     99.90    100.00\n",
      "     16       98.98     98.75     97.25     99.90    100.00\n",
      "     17       99.04     98.70     97.50     99.95    100.00\n",
      "     18       98.99     98.65     97.40     99.90    100.00\n",
      "     19       98.98     98.70     97.30     99.90    100.00\n",
      "     20       98.94     98.60     97.25     99.90    100.00\n",
      "     21       99.02     98.70     97.50     99.90    100.00\n",
      "     22       98.95     98.60     97.30     99.90    100.00\n",
      "     23       98.96     98.65     97.30     99.90    100.00\n",
      "     24       98.94     98.70     97.15     99.90    100.00\n",
      "     25       98.95     98.70     97.20     99.90    100.00\n",
      "     26       99.00     98.65     97.45     99.90    100.00\n",
      "     27       98.98     98.75     97.25     99.90    100.00\n",
      "     28       98.96     98.70     97.25     99.90    100.00\n",
      "     29       98.99     98.70     97.35     99.90    100.00\n",
      "     30       99.03     98.65     97.55     99.90    100.00\n",
      "     31       98.98     98.65     97.35     99.90    100.00\n",
      "     32       99.02     98.70     97.50     99.90    100.00\n",
      "     33       98.92     98.70     97.10     99.90    100.00\n",
      "     34       99.01     98.70     97.45     99.90    100.00\n",
      "     35       98.99     98.65     97.40     99.90    100.00\n",
      "     36       98.95     98.70     97.20     99.90    100.00\n",
      "     37       98.92     98.60     97.20     99.90    100.00\n",
      "     38       98.94     98.65     97.20     99.90    100.00\n",
      "     39       99.01     98.75     97.40     99.90    100.00\n",
      "     40       98.96     98.70     97.25     99.90    100.00\n",
      "     41       98.89     98.60     97.05     99.90    100.00\n",
      "     42       98.98     98.70     97.30     99.90    100.00\n",
      "     43       98.99     98.65     97.40     99.90    100.00\n",
      "     44       98.96     98.70     97.25     99.90    100.00\n",
      "     45       98.93     98.75     97.05     99.90    100.00\n",
      "     46       98.94     98.70     97.15     99.90    100.00\n",
      "     47       98.90     98.60     97.10     99.90    100.00\n",
      "     48       99.04     98.90     97.35     99.90    100.00\n",
      "     49       99.02     98.80     97.40     99.90    100.00\n",
      "     50       99.03     98.85     97.30     99.95    100.00\n",
      "     51       99.00     98.70     97.40     99.90    100.00\n",
      "     52       99.00     98.85     97.20     99.95    100.00\n",
      "     53       99.01     98.75     97.35     99.95    100.00\n",
      "     54       99.04     98.80     97.45     99.90    100.00\n",
      "     55       99.08     98.70     97.65     99.95    100.00\n",
      "     56       98.99     98.80     97.25     99.90    100.00\n",
      "     57       99.01     98.90     97.25     99.90    100.00\n",
      "     58       99.00     98.80     97.30     99.90    100.00\n",
      "     59       98.99     98.65     97.35     99.95    100.00\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9899 \n",
      "Accuracies (test): [0.9865, 0.9735, 0.9995, 1.0]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 5\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------  --------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc    t3 acc    t4 acc    t5 acc\n",
      "-------  ----------  --------  --------  --------  --------  --------\n",
      "      0       99.11     98.75     97.20     99.85    100.00     99.75\n",
      "      1       99.04     98.70     96.90     99.85    100.00     99.75\n",
      "      2       99.08     98.70     97.10     99.85    100.00     99.75\n",
      "      3       99.06     98.75     96.95     99.85    100.00     99.75\n",
      "      4       99.08     98.80     97.00     99.85    100.00     99.75\n",
      "      5       99.11     98.65     97.25     99.85    100.00     99.80\n",
      "      6       99.09     98.65     97.10     99.85    100.00     99.85\n",
      "      7       99.09     98.80     96.95     99.85    100.00     99.85\n",
      "      8       99.13     98.75     97.20     99.85    100.00     99.85\n",
      "      9       99.10     98.65     97.15     99.85    100.00     99.85\n",
      "     10       99.09     98.65     97.10     99.85    100.00     99.85\n",
      "     11       99.10     98.75     97.05     99.85    100.00     99.85\n",
      "     12       99.05     98.65     96.90     99.85    100.00     99.85\n",
      "     13       99.07     98.70     97.05     99.85    100.00     99.75\n",
      "     14       99.10     98.65     97.20     99.85    100.00     99.80\n",
      "     15       99.07     98.70     97.05     99.85    100.00     99.75\n",
      "     16       99.08     98.70     97.05     99.85    100.00     99.80\n",
      "     17       99.10     98.65     97.15     99.85    100.00     99.85\n",
      "     18       99.09     98.60     97.20     99.85    100.00     99.80\n",
      "     19       99.08     98.85     96.90     99.85    100.00     99.80\n",
      "     20       99.05     98.65     96.95     99.85    100.00     99.80\n",
      "     21       99.14     98.90     97.15     99.85    100.00     99.80\n",
      "     22       99.11     98.75     97.15     99.85    100.00     99.80\n",
      "     23       99.08     98.70     97.05     99.85    100.00     99.80\n",
      "     24       99.11     98.75     97.15     99.85    100.00     99.80\n",
      "     25       99.08     98.65     97.10     99.85    100.00     99.80\n",
      "     26       99.08     98.55     97.20     99.85    100.00     99.80\n",
      "     27       99.10     98.75     97.10     99.85    100.00     99.80\n",
      "     28       99.09     98.65     97.15     99.85    100.00     99.80\n",
      "     29       99.08     98.75     97.00     99.85    100.00     99.80\n",
      "     30       99.05     98.70     96.90     99.85    100.00     99.80\n",
      "     31       99.11     98.75     97.15     99.85    100.00     99.80\n",
      "     32       99.10     98.60     97.25     99.85    100.00     99.80\n",
      "     33       99.04     98.65     96.90     99.85    100.00     99.80\n",
      "     34       99.14     98.80     97.20     99.90    100.00     99.80\n",
      "     35       99.16     98.85     97.30     99.85    100.00     99.80\n",
      "     36       99.12     98.65     97.30     99.85    100.00     99.80\n",
      "     37       99.12     98.75     97.20     99.85    100.00     99.80\n",
      "     38       99.14     98.80     97.25     99.85    100.00     99.80\n",
      "     39       99.11     98.70     97.20     99.85    100.00     99.80\n",
      "     40       99.12     98.75     97.20     99.85    100.00     99.80\n",
      "     41       99.10     98.80     97.05     99.85    100.00     99.80\n",
      "     42       99.12     98.70     97.35     99.85    100.00     99.70\n",
      "     43       99.12     98.75     97.20     99.85    100.00     99.80\n",
      "     44       99.11     98.60     97.25     99.90    100.00     99.80\n",
      "     45       99.06     98.80     96.85     99.85    100.00     99.80\n",
      "     46       99.07     98.65     97.05     99.85    100.00     99.80\n",
      "     47       99.11     98.70     97.20     99.85    100.00     99.80\n",
      "     48       99.09     98.75     97.05     99.85    100.00     99.80\n",
      "     49       99.13     98.75     97.25     99.90    100.00     99.75\n",
      "     50       99.08     98.70     97.15     99.85    100.00     99.70\n",
      "     51       99.11     98.80     97.10     99.85    100.00     99.80\n",
      "     52       99.08     98.85     96.90     99.85    100.00     99.80\n",
      "     53       99.17     98.85     97.35     99.85    100.00     99.80\n",
      "     54       99.14     98.85     97.15     99.90    100.00     99.80\n",
      "     55       99.13     98.85     97.20     99.85    100.00     99.75\n",
      "     56       99.16     98.85     97.30     99.85    100.00     99.80\n",
      "     57       99.13     98.80     97.20     99.85    100.00     99.80\n",
      "     58       99.08     98.75     97.00     99.85    100.00     99.80\n",
      "     59       99.14     98.90     97.15     99.85    100.00     99.80\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9914 \n",
      "Accuracies (test): [0.989, 0.9715, 0.9985, 1.0, 0.998]\n",
      "---\n",
      "\n",
      "\n",
      "------------------- DONE -------------------\n",
      "\n",
      "\n",
      "0.9914\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "logdir = read_config_and_run(\"fsvi_match.pkl\", task_sequence)\n",
    "exp = lutils.read_exp(logdir)\n",
    "show_final_average_accuracy(exp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## S-FSVI (larger networks) <a name=\"res2\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading experiments: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 1151.78it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading from cache:\n",
      "Running on oat1.cs.ox.ac.uk\n",
      "Jax is running on gpu\n",
      "\n",
      "\n",
      "Input arguments:\n",
      " {\n",
      "    \"command\":\"cl\",\n",
      "    \"data_training\":\"continual_learning_sfashionmnist\",\n",
      "    \"data_ood\":[\n",
      "        \"not_specified\"\n",
      "    ],\n",
      "    \"model_type\":\"fsvi_mlp\",\n",
      "    \"optimizer\":\"adam\",\n",
      "    \"optimizer_var\":\"not_specified\",\n",
      "    \"momentum\":0.0,\n",
      "    \"momentum_var\":0.0,\n",
      "    \"schedule\":\"not_specified\",\n",
      "    \"architecture\":[\n",
      "        200,\n",
      "        200,\n",
      "        200,\n",
      "        200\n",
      "    ],\n",
      "    \"activation\":\"relu\",\n",
      "    \"prior_mean\":\"0.0\",\n",
      "    \"prior_cov\":\"0.001\",\n",
      "    \"prior_covs\":[\n",
      "        0.0\n",
      "    ],\n",
      "    \"prior_type\":\"bnn_induced\",\n",
      "    \"epochs\":60,\n",
      "    \"start_var_opt\":0,\n",
      "    \"batch_size\":128,\n",
      "    \"learning_rate\":0.0005,\n",
      "    \"learning_rate_var\":0.001,\n",
      "    \"dropout_rate\":0.0,\n",
      "    \"regularization\":0.0,\n",
      "    \"inducing_points\":0,\n",
      "    \"n_marginals\":1,\n",
      "    \"n_condition\":128,\n",
      "    \"inducing_input_type\":\"uniform_rand\",\n",
      "    \"inducing_input_ood_data\":[\n",
      "        \"not_specified\"\n",
      "    ],\n",
      "    \"inducing_input_ood_data_size\":50000,\n",
      "    \"kl_scale\":\"equal\",\n",
      "    \"feature_map_jacobian\":false,\n",
      "    \"feature_map_jacobian_train_only\":false,\n",
      "    \"feature_map_type\":\"not_specified\",\n",
      "    \"td_prior_scale\":0.0,\n",
      "    \"feature_update\":1,\n",
      "    \"full_cov\":false,\n",
      "    \"n_samples\":5,\n",
      "    \"n_samples_eval\":5,\n",
      "    \"tau\":1.0,\n",
      "    \"noise_std\":1.0,\n",
      "    \"ind_lim\":\"ind_-1_1\",\n",
      "    \"logging_frequency\":10,\n",
      "    \"figsize\":[\n",
      "        \"10\",\n",
      "        \"4\"\n",
      "    ],\n",
      "    \"seed\":5,\n",
      "    \"save\":false,\n",
      "    \"name\":\"\",\n",
      "    \"evaluate\":false,\n",
      "    \"resume_training\":false,\n",
      "    \"no_final_layer_bias\":false,\n",
      "    \"extra_linear_layer\":false,\n",
      "    \"map_initialization\":false,\n",
      "    \"stochastic_linearization\":false,\n",
      "    \"grad_flow_jacobian\":false,\n",
      "    \"stochastic_prior_mean\":\"not_specified\",\n",
      "    \"batch_normalization\":false,\n",
      "    \"batch_normalization_mod\":\"not_specified\",\n",
      "    \"final_layer_variational\":false,\n",
      "    \"kl_sup\":\"not_specified\",\n",
      "    \"kl_sampled\":false,\n",
      "    \"fixed_inner_layers_variational_var\":false,\n",
      "    \"init_logvar\":[\n",
      "        0.0,\n",
      "        0.0\n",
      "    ],\n",
      "    \"init_logvar_lin\":[\n",
      "        0.0,\n",
      "        0.0\n",
      "    ],\n",
      "    \"init_logvar_conv\":[\n",
      "        0.0,\n",
      "        0.0\n",
      "    ],\n",
      "    \"perturbation_param\":0.01,\n",
      "    \"logroot\":\"ablation\",\n",
      "    \"subdir\":\"reproduce_main_results_2\",\n",
      "    \"wandb_project\":\"not_specified\",\n",
      "    \"n_inducing_inputs\":40,\n",
      "    \"n_inducing_inputs_first_task\":\"not_specified\",\n",
      "    \"n_inducing_inputs_second_task\":\"not_specified\",\n",
      "    \"inducing_inputs_bound\":[\n",
      "        0.0,\n",
      "        1.0\n",
      "    ],\n",
      "    \"fix_shuffle\":false,\n",
      "    \"use_generative_model\":false,\n",
      "    \"inducing_inputs_add_mode\":0,\n",
      "    \"inducing_input_adjustment\":false,\n",
      "    \"not_use_coreset\":false,\n",
      "    \"inducing_input_augmentation\":false,\n",
      "    \"plotting\":false,\n",
      "    \"logging\":1,\n",
      "    \"use_val_split\":false,\n",
      "    \"not_use_val_split\":true,\n",
      "    \"coreset\":\"random\",\n",
      "    \"coreset_entropy_mode\":\"soft_highest\",\n",
      "    \"coreset_entropy_offset\":\"0.0\",\n",
      "    \"coreset_kl_heuristic\":\"lowest\",\n",
      "    \"coreset_kl_offset\":\"0.0\",\n",
      "    \"coreset_elbo_heuristic\":\"lowest\",\n",
      "    \"coreset_elbo_offset\":\"0.0\",\n",
      "    \"coreset_elbo_n_samples\":5,\n",
      "    \"coreset_n_tasks\":\"not_specified\",\n",
      "    \"coreset_entropy_n_mixed\":1,\n",
      "    \"n_coreset_inputs_per_task\":\"not_specified\",\n",
      "    \"full_ntk\":false,\n",
      "    \"constant_inducing_points\":false,\n",
      "    \"n_permuted_tasks\":10,\n",
      "    \"n_omniglot_tasks\":20,\n",
      "    \"epochs_first_task\":60,\n",
      "    \"identity_cov\":false,\n",
      "    \"n_epochs_save_params\":\"not_specified\",\n",
      "    \"n_augment\":\"not_specified\",\n",
      "    \"augment_mode\":\"constant\",\n",
      "    \"n_valid\":\"same\",\n",
      "    \"learning_rate_first_task\":\"not_specified\",\n",
      "    \"save_alt\":true,\n",
      "    \"save_first_task\":false,\n",
      "    \"first_task_load_exp_path\":\"not_specified\",\n",
      "    \"only_task_id\":\"not_specified\",\n",
      "    \"loss_type\":1,\n",
      "    \"only_trainable_head\":false,\n",
      "    \"n_omniglot_coreset_chars\":2,\n",
      "    \"omniglot_randomize_test_split\":false,\n",
      "    \"omniglot_randomize_task_sequence\":false,\n",
      "    \"n_inducing_input_adjust_amount\":\"not_specified\",\n",
      "    \"save_all_params\":false,\n",
      "    \"task\":\"continual_learning_sfashionmnist\",\n",
      "    \"architecture_arg\":\"fc_200_200_200_200\",\n",
      "    \"init_logvar_minval\":0.0,\n",
      "    \"init_logvar_maxval\":0.0,\n",
      "    \"init_logvar_lin_minval\":0.0,\n",
      "    \"init_logvar_lin_maxval\":0.0,\n",
      "    \"init_logvar_conv_minval\":0.0,\n",
      "    \"init_logvar_conv_maxval\":0.0\n",
      "} \n",
      "\n",
      "\n",
      "MAP initialization: False\n",
      "Full NTK computation: False\n",
      "Stochastic linearization (posterior): False\n",
      "init_logvar_lin_minval: -10.0\n",
      "init_logvar_lin_maxval: -8.0\n",
      "Full NTK computation: False\n",
      "Stochastic linearization (prior): False\n",
      "\n",
      "\n",
      "Learning task 1\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------\n",
      "  epoch    mean acc    t1 acc\n",
      "-------  ----------  --------\n",
      "      0       95.20     95.20\n",
      "      1       98.40     98.40\n",
      "      2       98.30     98.30\n",
      "      3       98.60     98.60\n",
      "      4       98.90     98.90\n",
      "      5       99.05     99.05\n",
      "      6       99.10     99.10\n",
      "      7       99.15     99.15\n",
      "      8       99.35     99.35\n",
      "      9       99.15     99.15\n",
      "     10       99.20     99.20\n",
      "     11       99.40     99.40\n",
      "     12       99.30     99.30\n",
      "     13       99.15     99.15\n",
      "     14       99.25     99.25\n",
      "     15       98.65     98.65\n",
      "     16       99.25     99.25\n",
      "     17       99.15     99.15\n",
      "     18       99.25     99.25\n",
      "     19       99.15     99.15\n",
      "     20       99.35     99.35\n",
      "     21       99.45     99.45\n",
      "     22       99.10     99.10\n",
      "     23       99.30     99.30\n",
      "     24       99.25     99.25\n",
      "     25       99.15     99.15\n",
      "     26       99.10     99.10\n",
      "     27       99.35     99.35\n",
      "     28       99.15     99.15\n",
      "     29       98.45     98.45\n",
      "     30       99.30     99.30\n",
      "     31       99.30     99.30\n",
      "     32       99.30     99.30\n",
      "     33       99.20     99.20\n",
      "     34       99.20     99.20\n",
      "     35       99.05     99.05\n",
      "     36       99.15     99.15\n",
      "     37       99.20     99.20\n",
      "     38       99.20     99.20\n",
      "     39       99.45     99.45\n",
      "     40       99.30     99.30\n",
      "     41       98.75     98.75\n",
      "     42       99.30     99.30\n",
      "     43       99.35     99.35\n",
      "     44       99.15     99.15\n",
      "     45       99.35     99.35\n",
      "     46       99.30     99.30\n",
      "     47       99.15     99.15\n",
      "     48       98.85     98.85\n",
      "     49       99.15     99.15\n",
      "     50       99.35     99.35\n",
      "     51       99.10     99.10\n",
      "     52       99.20     99.20\n",
      "     53       99.30     99.30\n",
      "     54       99.25     99.25\n",
      "     55       99.25     99.25\n",
      "     56       99.15     99.15\n",
      "     57       99.20     99.20\n",
      "     58       98.90     98.90\n",
      "     59       99.30     99.30\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9930 \n",
      "Accuracies (test): [0.993]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 2\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc\n",
      "-------  ----------  --------  --------\n",
      "      0       97.17     98.40     95.95\n",
      "      1       97.25     98.35     96.15\n",
      "      2       97.38     98.35     96.40\n",
      "      3       97.50     98.20     96.80\n",
      "      4       97.55     98.35     96.75\n",
      "      5       97.58     98.30     96.85\n",
      "      6       97.75     98.40     97.10\n",
      "      7       97.60     98.35     96.85\n",
      "      8       97.67     98.35     97.00\n",
      "      9       97.75     98.35     97.15\n",
      "     10       97.88     98.30     97.45\n",
      "     11       97.75     98.25     97.25\n",
      "     12       97.85     98.30     97.40\n",
      "     13       97.78     98.15     97.40\n",
      "     14       97.95     98.25     97.65\n",
      "     15       97.90     98.15     97.65\n",
      "     16       97.95     98.20     97.70\n",
      "     17       97.60     98.15     97.05\n",
      "     18       97.80     98.20     97.40\n",
      "     19       97.85     98.15     97.55\n",
      "     20       97.95     98.20     97.70\n",
      "     21       97.78     98.15     97.40\n",
      "     22       97.65     98.00     97.30\n",
      "     23       97.62     97.80     97.45\n",
      "     24       97.88     98.05     97.70\n",
      "     25       97.70     97.90     97.50\n",
      "     26       97.88     97.85     97.90\n",
      "     27       97.90     97.85     97.95\n",
      "     28       97.75     98.00     97.50\n",
      "     29       97.80     97.95     97.65\n",
      "     30       97.70     97.75     97.65\n",
      "     31       97.92     98.00     97.85\n",
      "     32       97.75     98.05     97.45\n",
      "     33       97.88     98.05     97.70\n",
      "     34       97.83     97.95     97.70\n",
      "     35       97.78     98.00     97.55\n",
      "     36       97.88     98.15     97.60\n",
      "     37       97.83     97.90     97.75\n",
      "     38       97.83     98.05     97.60\n",
      "     39       97.67     98.10     97.25\n",
      "     40       97.85     98.10     97.60\n",
      "     41       98.00     98.10     97.90\n",
      "     42       97.92     98.10     97.75\n",
      "     43       97.97     98.15     97.80\n",
      "     44       98.00     98.15     97.85\n",
      "     45       98.05     98.35     97.75\n",
      "     46       98.03     98.20     97.85\n",
      "     47       98.03     98.25     97.80\n",
      "     48       98.03     98.35     97.70\n",
      "     49       97.95     98.10     97.80\n",
      "     50       98.03     98.15     97.90\n",
      "     51       97.78     98.10     97.45\n",
      "     52       97.90     98.25     97.55\n",
      "     53       97.95     98.25     97.65\n",
      "     54       97.92     98.05     97.80\n",
      "     55       97.95     98.25     97.65\n",
      "     56       98.03     98.40     97.65\n",
      "     57       98.10     98.35     97.85\n",
      "     58       97.97     98.20     97.75\n",
      "     59       98.03     98.25     97.80\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9803 \n",
      "Accuracies (test): [0.9825, 0.978]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 3\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc    t3 acc\n",
      "-------  ----------  --------  --------  --------\n",
      "      0       98.32     97.95     97.00    100.00\n",
      "      1       98.18     97.90     96.70     99.95\n",
      "      2       98.10     97.85     96.50     99.95\n",
      "      3       98.10     97.95     96.35    100.00\n",
      "      4       98.17     97.95     96.55    100.00\n",
      "      5       98.07     97.90     96.30    100.00\n",
      "      6       98.12     97.90     96.45    100.00\n",
      "      7       98.15     97.85     96.60    100.00\n",
      "      8       98.08     97.85     96.40    100.00\n",
      "      9       98.18     98.05     96.50    100.00\n",
      "     10       98.08     97.80     96.45    100.00\n",
      "     11       98.18     97.90     96.65    100.00\n",
      "     12       98.13     97.80     96.60    100.00\n",
      "     13       98.17     97.85     96.65    100.00\n",
      "     14       98.12     97.85     96.50    100.00\n",
      "     15       98.13     97.95     96.45    100.00\n",
      "     16       98.08     97.55     96.70    100.00\n",
      "     17       98.12     97.95     96.40    100.00\n",
      "     18       98.13     97.95     96.45    100.00\n",
      "     19       98.07     97.85     96.35    100.00\n",
      "     20       98.15     97.75     96.70    100.00\n",
      "     21       98.08     97.85     96.45     99.95\n",
      "     22       98.15     97.95     96.55     99.95\n",
      "     23       98.13     98.10     96.30    100.00\n",
      "     24       98.13     97.70     96.70    100.00\n",
      "     25       98.05     97.60     96.55    100.00\n",
      "     26       98.18     98.00     96.55    100.00\n",
      "     27       98.15     97.80     96.65    100.00\n",
      "     28       98.02     97.65     96.40    100.00\n",
      "     29       98.05     97.65     96.50    100.00\n",
      "     30       98.18     98.05     96.50    100.00\n",
      "     31       98.10     97.80     96.50    100.00\n",
      "     32       98.02     97.70     96.35    100.00\n",
      "     33       98.15     98.00     96.45    100.00\n",
      "     34       98.12     97.90     96.45    100.00\n",
      "     35       98.15     98.00     96.45    100.00\n",
      "     36       98.20     98.15     96.45    100.00\n",
      "     37       98.15     98.05     96.40    100.00\n",
      "     38       98.22     97.70     96.95    100.00\n",
      "     39       98.25     97.95     96.80    100.00\n",
      "     40       98.15     97.75     96.70    100.00\n",
      "     41       98.05     98.35     95.80    100.00\n",
      "     42       98.15     98.05     96.40    100.00\n",
      "     43       98.15     97.85     96.60    100.00\n",
      "     44       98.28     98.40     96.45    100.00\n",
      "     45       98.12     98.00     96.35    100.00\n",
      "     46       98.12     97.75     96.60    100.00\n",
      "     47       98.12     97.50     96.85    100.00\n",
      "     48       98.18     98.05     96.50    100.00\n",
      "     49       98.08     97.55     96.70    100.00\n",
      "     50       98.25     98.25     96.50    100.00\n",
      "     51       98.18     98.10     96.45    100.00\n",
      "     52       98.30     98.15     96.75    100.00\n",
      "     53       98.22     98.00     96.65    100.00\n",
      "     54       98.22     97.90     96.75    100.00\n",
      "     55       98.23     97.95     96.75    100.00\n",
      "     56       98.02     98.00     96.05    100.00\n",
      "     57       98.15     98.25     96.20    100.00\n",
      "     58       98.15     97.80     96.65    100.00\n",
      "     59       98.22     98.05     96.60    100.00\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9822 \n",
      "Accuracies (test): [0.9805, 0.966, 1.0]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 4\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc    t3 acc    t4 acc\n",
      "-------  ----------  --------  --------  --------  --------\n",
      "      0       98.16     97.85     94.90     99.90    100.00\n",
      "      1       98.16     97.70     95.00     99.95    100.00\n",
      "      2       98.24     97.60     95.50     99.85    100.00\n",
      "      3       98.16     97.80     94.95     99.90    100.00\n",
      "      4       98.17     97.55     95.30     99.85    100.00\n",
      "      5       98.16     97.95     94.90     99.80    100.00\n",
      "      6       98.10     97.70     94.85     99.85    100.00\n",
      "      7       98.14     97.50     95.25     99.80    100.00\n",
      "      8       98.30     97.60     95.75     99.85    100.00\n",
      "      9       98.24     97.80     95.30     99.85    100.00\n",
      "     10       98.19     97.75     95.15     99.85    100.00\n",
      "     11       98.15     97.75     95.00     99.85    100.00\n",
      "     12       97.88     97.40     94.25     99.85    100.00\n",
      "     13       98.14     97.55     95.15     99.85    100.00\n",
      "     14       97.92     97.65     94.20     99.85    100.00\n",
      "     15       98.11     97.65     94.95     99.85    100.00\n",
      "     16       97.80     97.65     93.70     99.85    100.00\n",
      "     17       97.90     97.60     94.15     99.85    100.00\n",
      "     18       98.22     97.25     95.80     99.85    100.00\n",
      "     19       98.06     97.50     94.90     99.85    100.00\n",
      "     20       97.96     97.30     94.70     99.85    100.00\n",
      "     21       98.16     97.45     95.35     99.85    100.00\n",
      "     22       98.19     97.65     95.25     99.85    100.00\n",
      "     23       98.01     97.50     94.70     99.85    100.00\n",
      "     24       98.11     97.50     95.10     99.85    100.00\n",
      "     25       97.84     97.55     93.95     99.85    100.00\n",
      "     26       98.30     97.40     95.95     99.85    100.00\n",
      "     27       98.21     97.30     95.70     99.85    100.00\n",
      "     28       98.05     97.55     94.80     99.85    100.00\n",
      "     29       97.99     97.40     94.70     99.85    100.00\n",
      "     30       98.02     97.50     94.70     99.90    100.00\n",
      "     31       98.11     97.40     95.20     99.85    100.00\n",
      "     32       98.01     97.45     94.75     99.85    100.00\n",
      "     33       98.14     97.60     95.10     99.85    100.00\n",
      "     34       97.99     97.45     94.65     99.85    100.00\n",
      "     35       97.94     97.55     94.35     99.85    100.00\n",
      "     36       97.95     97.65     94.30     99.85    100.00\n",
      "     37       98.19     97.35     95.55     99.85    100.00\n",
      "     38       97.92     97.35     94.50     99.85    100.00\n",
      "     39       98.01     97.75     94.45     99.85    100.00\n",
      "     40       98.05     97.30     95.00     99.90    100.00\n",
      "     41       98.22     97.50     95.50     99.90    100.00\n",
      "     42       98.17     97.40     95.45     99.85    100.00\n",
      "     43       98.01     97.35     94.85     99.85    100.00\n",
      "     44       97.96     97.35     94.60     99.90    100.00\n",
      "     45       98.11     97.50     95.05     99.90    100.00\n",
      "     46       98.16     97.45     95.35     99.85    100.00\n",
      "     47       97.91     97.50     94.25     99.90    100.00\n",
      "     48       98.03     97.55     94.65     99.90    100.00\n",
      "     49       98.08     97.65     94.75     99.90    100.00\n",
      "     50       98.24     97.35     95.70     99.90    100.00\n",
      "     51       97.96     97.45     94.50     99.90    100.00\n",
      "     52       97.82     97.55     93.90     99.85    100.00\n",
      "     53       98.01     97.40     94.75     99.90    100.00\n",
      "     54       98.06     97.50     94.85     99.90    100.00\n",
      "     55       97.95     97.55     94.40     99.85    100.00\n",
      "     56       97.75     97.55     93.60     99.85    100.00\n",
      "     57       97.88     97.35     94.25     99.90    100.00\n",
      "     58       98.11     97.55     95.00     99.90    100.00\n",
      "     59       97.94     97.60     94.25     99.90    100.00\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9794 \n",
      "Accuracies (test): [0.976, 0.9425, 0.999, 1.0]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 5\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------  --------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc    t3 acc    t4 acc    t5 acc\n",
      "-------  ----------  --------  --------  --------  --------  --------\n",
      "      0       98.31     96.80     95.30     99.75     99.95     99.75\n",
      "      1       98.31     96.70     95.50     99.65     99.95     99.75\n",
      "      2       98.29     96.60     95.45     99.70     99.95     99.75\n",
      "      3       98.21     96.70     95.05     99.60     99.95     99.75\n",
      "      4       98.15     96.55     94.90     99.60     99.95     99.75\n",
      "      5       98.26     96.80     95.25     99.60     99.95     99.70\n",
      "      6       98.35     96.85     95.60     99.65     99.95     99.70\n",
      "      7       98.30     96.80     95.35     99.60     99.95     99.80\n",
      "      8       98.10     96.85     94.35     99.55     99.90     99.85\n",
      "      9       98.22     96.85     94.85     99.60     99.95     99.85\n",
      "     10       98.27     96.75     95.30     99.55     99.95     99.80\n",
      "     11       98.34     96.80     95.55     99.60     99.95     99.80\n",
      "     12       98.35     96.70     95.60     99.65     99.95     99.85\n",
      "     13       98.24     96.85     94.90     99.65     99.95     99.85\n",
      "     14       98.28     96.75     95.25     99.60     99.95     99.85\n",
      "     15       98.20     96.80     94.80     99.65     99.95     99.80\n",
      "     16       98.35     96.70     95.65     99.65     99.95     99.80\n",
      "     17       98.15     96.80     94.60     99.65     99.95     99.75\n",
      "     18       98.17     96.75     94.70     99.65     99.95     99.80\n",
      "     19       98.31     96.80     95.40     99.65     99.95     99.75\n",
      "     20       98.34     96.90     95.45     99.65     99.95     99.75\n",
      "     21       98.38     96.85     95.65     99.65     99.95     99.80\n",
      "     22       98.27     96.65     95.25     99.65     99.95     99.85\n",
      "     23       98.29     96.85     95.20     99.65     99.95     99.80\n",
      "     24       98.32     96.60     95.60     99.65     99.95     99.80\n",
      "     25       98.33     96.70     95.60     99.60     99.95     99.80\n",
      "     26       98.23     96.50     95.20     99.65     99.95     99.85\n",
      "     27       98.24     96.85     94.95     99.65     99.95     99.80\n",
      "     28       98.35     96.60     95.70     99.65     99.95     99.85\n",
      "     29       98.40     96.50     96.10     99.65     99.95     99.80\n",
      "     30       98.12     96.80     94.40     99.65     99.95     99.80\n",
      "     31       98.23     96.65     95.10     99.65     99.95     99.80\n",
      "     32       98.13     96.75     94.50     99.65     99.95     99.80\n",
      "     33       98.33     96.55     95.70     99.65     99.95     99.80\n",
      "     34       98.30     96.70     95.40     99.65     99.95     99.80\n",
      "     35       98.37     96.60     95.90     99.65     99.95     99.75\n",
      "     36       98.27     96.65     95.30     99.65     99.95     99.80\n",
      "     37       98.24     96.50     95.30     99.65     99.95     99.80\n",
      "     38       98.20     96.70     94.90     99.65     99.95     99.80\n",
      "     39       98.42     96.35     96.35     99.65     99.95     99.80\n",
      "     40       98.33     96.75     95.50     99.65     99.95     99.80\n",
      "     41       98.26     96.75     95.15     99.65     99.95     99.80\n",
      "     42       98.36     96.50     95.90     99.65     99.95     99.80\n",
      "     43       98.35     96.70     95.65     99.65     99.95     99.80\n",
      "     44       98.42     96.65     96.00     99.65     99.95     99.85\n",
      "     45       98.39     96.75     95.80     99.65     99.95     99.80\n",
      "     46       98.30     96.60     95.50     99.65     99.95     99.80\n",
      "     47       98.37     96.75     95.70     99.65     99.95     99.80\n",
      "     48       98.30     96.50     95.60     99.65     99.95     99.80\n",
      "     49       98.42     96.50     96.20     99.65     99.95     99.80\n",
      "     50       98.31     96.80     95.35     99.65     99.95     99.80\n",
      "     51       98.46     96.80     96.10     99.65     99.95     99.80\n",
      "     52       98.29     96.50     95.50     99.65     99.95     99.85\n",
      "     53       98.30     96.55     95.50     99.65     99.95     99.85\n",
      "     54       98.33     96.75     95.45     99.65     99.95     99.85\n",
      "     55       98.18     96.70     94.80     99.65     99.95     99.80\n",
      "     56       98.39     96.60     95.95     99.65     99.95     99.80\n",
      "     57       98.25     96.70     95.15     99.65     99.95     99.80\n",
      "     58       98.40     96.60     96.00     99.65     99.95     99.80\n",
      "     59       98.30     96.75     95.35     99.65     99.95     99.80\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9830 \n",
      "Accuracies (test): [0.9675, 0.9535, 0.9965, 0.9995, 0.998]\n",
      "---\n",
      "\n",
      "\n",
      "------------------- DONE -------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.983\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "logdir = read_config_and_run(\"fsvi_optimized.pkl\", task_sequence)\n",
    "exp = lutils.read_exp(logdir)\n",
    "show_final_average_accuracy(exp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## S-FSVI (no coreset) <a name=\"res3\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading experiments: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 1152.80it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading from cache:\n",
      "Running on oat3.cs.ox.ac.uk\n",
      "Jax is running on gpu\n",
      "\n",
      "\n",
      "Input arguments:\n",
      " {\n",
      "    \"command\":\"cl\",\n",
      "    \"data_training\":\"continual_learning_sfashionmnist\",\n",
      "    \"data_ood\":[\n",
      "        \"not_specified\"\n",
      "    ],\n",
      "    \"model_type\":\"fsvi_mlp\",\n",
      "    \"optimizer\":\"adam\",\n",
      "    \"optimizer_var\":\"not_specified\",\n",
      "    \"momentum\":0.0,\n",
      "    \"momentum_var\":0.0,\n",
      "    \"schedule\":\"not_specified\",\n",
      "    \"architecture\":[\n",
      "        256,\n",
      "        256\n",
      "    ],\n",
      "    \"activation\":\"relu\",\n",
      "    \"prior_mean\":\"0.0\",\n",
      "    \"prior_cov\":\"100.0\",\n",
      "    \"prior_covs\":[\n",
      "        0.0\n",
      "    ],\n",
      "    \"prior_type\":\"bnn_induced\",\n",
      "    \"epochs\":60,\n",
      "    \"start_var_opt\":0,\n",
      "    \"batch_size\":128,\n",
      "    \"learning_rate\":0.0005,\n",
      "    \"learning_rate_var\":0.001,\n",
      "    \"dropout_rate\":0.0,\n",
      "    \"regularization\":0.0,\n",
      "    \"inducing_points\":0,\n",
      "    \"n_marginals\":1,\n",
      "    \"n_condition\":128,\n",
      "    \"inducing_input_type\":\"train_pixel_rand_1.0\",\n",
      "    \"inducing_input_ood_data\":[\n",
      "        \"not_specified\"\n",
      "    ],\n",
      "    \"inducing_input_ood_data_size\":50000,\n",
      "    \"kl_scale\":\"equal\",\n",
      "    \"feature_map_jacobian\":false,\n",
      "    \"feature_map_jacobian_train_only\":false,\n",
      "    \"feature_map_type\":\"not_specified\",\n",
      "    \"td_prior_scale\":0.0,\n",
      "    \"feature_update\":1,\n",
      "    \"full_cov\":false,\n",
      "    \"n_samples\":5,\n",
      "    \"n_samples_eval\":5,\n",
      "    \"tau\":1.0,\n",
      "    \"noise_std\":1.0,\n",
      "    \"ind_lim\":\"ind_-1_1\",\n",
      "    \"logging_frequency\":10,\n",
      "    \"figsize\":[\n",
      "        \"10\",\n",
      "        \"4\"\n",
      "    ],\n",
      "    \"seed\":3,\n",
      "    \"save\":false,\n",
      "    \"name\":\"\",\n",
      "    \"evaluate\":false,\n",
      "    \"resume_training\":false,\n",
      "    \"no_final_layer_bias\":false,\n",
      "    \"extra_linear_layer\":false,\n",
      "    \"map_initialization\":false,\n",
      "    \"stochastic_linearization\":false,\n",
      "    \"grad_flow_jacobian\":false,\n",
      "    \"stochastic_prior_mean\":\"not_specified\",\n",
      "    \"batch_normalization\":false,\n",
      "    \"batch_normalization_mod\":\"not_specified\",\n",
      "    \"final_layer_variational\":false,\n",
      "    \"kl_sup\":\"not_specified\",\n",
      "    \"kl_sampled\":false,\n",
      "    \"fixed_inner_layers_variational_var\":false,\n",
      "    \"init_logvar\":[\n",
      "        0.0,\n",
      "        0.0\n",
      "    ],\n",
      "    \"init_logvar_lin\":[\n",
      "        0.0,\n",
      "        0.0\n",
      "    ],\n",
      "    \"init_logvar_conv\":[\n",
      "        0.0,\n",
      "        0.0\n",
      "    ],\n",
      "    \"perturbation_param\":0.01,\n",
      "    \"logroot\":\"ablation\",\n",
      "    \"subdir\":\"reproduce_main_results_2\",\n",
      "    \"wandb_project\":\"not_specified\",\n",
      "    \"n_inducing_inputs\":40,\n",
      "    \"n_inducing_inputs_first_task\":\"not_specified\",\n",
      "    \"n_inducing_inputs_second_task\":\"not_specified\",\n",
      "    \"inducing_inputs_bound\":[\n",
      "        0.0,\n",
      "        1.0\n",
      "    ],\n",
      "    \"fix_shuffle\":false,\n",
      "    \"use_generative_model\":false,\n",
      "    \"inducing_inputs_add_mode\":0,\n",
      "    \"inducing_input_adjustment\":false,\n",
      "    \"not_use_coreset\":true,\n",
      "    \"inducing_input_augmentation\":false,\n",
      "    \"plotting\":false,\n",
      "    \"logging\":1,\n",
      "    \"use_val_split\":false,\n",
      "    \"not_use_val_split\":true,\n",
      "    \"coreset\":\"random\",\n",
      "    \"coreset_entropy_mode\":\"soft_highest\",\n",
      "    \"coreset_entropy_offset\":\"0.0\",\n",
      "    \"coreset_kl_heuristic\":\"lowest\",\n",
      "    \"coreset_kl_offset\":\"0.0\",\n",
      "    \"coreset_elbo_heuristic\":\"lowest\",\n",
      "    \"coreset_elbo_offset\":\"0.0\",\n",
      "    \"coreset_elbo_n_samples\":5,\n",
      "    \"coreset_n_tasks\":\"not_specified\",\n",
      "    \"coreset_entropy_n_mixed\":1,\n",
      "    \"n_coreset_inputs_per_task\":\"not_specified\",\n",
      "    \"full_ntk\":false,\n",
      "    \"constant_inducing_points\":true,\n",
      "    \"n_permuted_tasks\":10,\n",
      "    \"n_omniglot_tasks\":20,\n",
      "    \"epochs_first_task\":60,\n",
      "    \"identity_cov\":false,\n",
      "    \"n_epochs_save_params\":\"not_specified\",\n",
      "    \"n_augment\":\"not_specified\",\n",
      "    \"augment_mode\":\"constant\",\n",
      "    \"n_valid\":\"same\",\n",
      "    \"learning_rate_first_task\":\"not_specified\",\n",
      "    \"save_alt\":true,\n",
      "    \"save_first_task\":false,\n",
      "    \"first_task_load_exp_path\":\"not_specified\",\n",
      "    \"only_task_id\":\"not_specified\",\n",
      "    \"loss_type\":1,\n",
      "    \"only_trainable_head\":false,\n",
      "    \"n_omniglot_coreset_chars\":2,\n",
      "    \"omniglot_randomize_test_split\":false,\n",
      "    \"omniglot_randomize_task_sequence\":false,\n",
      "    \"n_inducing_input_adjust_amount\":\"not_specified\",\n",
      "    \"save_all_params\":false,\n",
      "    \"task\":\"continual_learning_sfashionmnist\",\n",
      "    \"architecture_arg\":\"fc_256_256\",\n",
      "    \"init_logvar_minval\":0.0,\n",
      "    \"init_logvar_maxval\":0.0,\n",
      "    \"init_logvar_lin_minval\":0.0,\n",
      "    \"init_logvar_lin_maxval\":0.0,\n",
      "    \"init_logvar_conv_minval\":0.0,\n",
      "    \"init_logvar_conv_maxval\":0.0\n",
      "} \n",
      "\n",
      "\n",
      "MAP initialization: False\n",
      "Full NTK computation: False\n",
      "Stochastic linearization (posterior): False\n",
      "init_logvar_lin_minval: -10.0\n",
      "init_logvar_lin_maxval: -8.0\n",
      "Full NTK computation: False\n",
      "Stochastic linearization (prior): False\n",
      "\n",
      "\n",
      "Learning task 1\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------\n",
      "  epoch    mean acc    t1 acc\n",
      "-------  ----------  --------\n",
      "      0       95.45     95.45\n",
      "      1       90.75     90.75\n",
      "      2       96.10     96.10\n",
      "      3       98.50     98.50\n",
      "      4       96.85     96.85\n",
      "      5       96.90     96.90\n",
      "      6       92.85     92.85\n",
      "      7       98.45     98.45\n",
      "      8       98.45     98.45\n",
      "      9       98.15     98.15\n",
      "     10       98.40     98.40\n",
      "     11       98.40     98.40\n",
      "     12       98.50     98.50\n",
      "     13       98.65     98.65\n",
      "     14       93.70     93.70\n",
      "     15       98.35     98.35\n",
      "     16       98.25     98.25\n",
      "     17       98.60     98.60\n",
      "     18       98.70     98.70\n",
      "     19       98.10     98.10\n",
      "     20       98.40     98.40\n",
      "     21       98.65     98.65\n",
      "     22       96.65     96.65\n",
      "     23       98.65     98.65\n",
      "     24       97.00     97.00\n",
      "     25       98.70     98.70\n",
      "     26       98.10     98.10\n",
      "     27       98.70     98.70\n",
      "     28       77.55     77.55\n",
      "     29       98.65     98.65\n",
      "     30       98.65     98.65\n",
      "     31       98.85     98.85\n",
      "     32       98.10     98.10\n",
      "     33       98.85     98.85\n",
      "     34       98.65     98.65\n",
      "     35       98.70     98.70\n",
      "     36       98.85     98.85\n",
      "     37       98.75     98.75\n",
      "     38       98.10     98.10\n",
      "     39       98.40     98.40\n",
      "     40       98.65     98.65\n",
      "     41       98.80     98.80\n",
      "     42       98.85     98.85\n",
      "     43       99.00     99.00\n",
      "     44       98.90     98.90\n",
      "     45       99.00     99.00\n",
      "     46       97.50     97.50\n",
      "     47       98.90     98.90\n",
      "     48       95.50     95.50\n",
      "     49       98.95     98.95\n",
      "     50       95.40     95.40\n",
      "     51       98.85     98.85\n",
      "     52       98.00     98.00\n",
      "     53       92.70     92.70\n",
      "     54       98.70     98.70\n",
      "     55       99.00     99.00\n",
      "     56       99.00     99.00\n",
      "     57       98.35     98.35\n",
      "     58       98.40     98.40\n",
      "     59       98.55     98.55\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9855 \n",
      "Accuracies (test): [0.9855]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 2\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc\n",
      "-------  ----------  --------  --------\n",
      "      0       96.47     97.35     95.60\n",
      "      1       97.58     98.70     96.45\n",
      "      2       97.45     98.35     96.55\n",
      "      3       97.78     98.65     96.90\n",
      "      4       95.50     95.95     95.05\n",
      "      5       97.28     98.25     96.30\n",
      "      6       97.47     98.10     96.85\n",
      "      7       98.10     98.85     97.35\n",
      "      8       98.03     99.00     97.05\n",
      "      9       97.47     97.75     97.20\n",
      "     10       97.32     98.80     95.85\n",
      "     11       98.17     99.00     97.35\n",
      "     12       98.23     98.95     97.50\n",
      "     13       97.72     98.50     96.95\n",
      "     14       97.67     97.95     97.40\n",
      "     15       97.75     98.20     97.30\n",
      "     16       97.88     98.70     97.05\n",
      "     17       97.60     98.35     96.85\n",
      "     18       97.35     97.20     97.50\n",
      "     19       98.08     98.55     97.60\n",
      "     20       97.83     98.55     97.10\n",
      "     21       97.90     98.15     97.65\n",
      "     22       97.40     97.25     97.55\n",
      "     23       98.12     98.85     97.40\n",
      "     24       98.23     98.80     97.65\n",
      "     25       97.05     96.60     97.50\n",
      "     26       97.65     97.70     97.60\n",
      "     27       96.88     96.40     97.35\n",
      "     28       97.40     97.75     97.05\n",
      "     29       98.17     98.75     97.60\n",
      "     30       97.30     97.10     97.50\n",
      "     31       98.17     99.00     97.35\n",
      "     32       97.40     97.55     97.25\n",
      "     33       98.10     98.70     97.50\n",
      "     34       97.92     98.75     97.10\n",
      "     35       97.97     99.00     96.95\n",
      "     36       97.43     97.50     97.35\n",
      "     37       97.32     97.45     97.20\n",
      "     38       98.17     98.85     97.50\n",
      "     39       96.20     95.75     96.65\n",
      "     40       97.85     98.50     97.20\n",
      "     41       97.88     98.30     97.45\n",
      "     42       97.20     97.20     97.20\n",
      "     43       98.03     98.50     97.55\n",
      "     44       97.38     97.35     97.40\n",
      "     45       97.08     97.30     96.85\n",
      "     46       97.83     98.50     97.15\n",
      "     47       98.28     98.95     97.60\n",
      "     48       97.65     98.30     97.00\n",
      "     49       97.15     98.95     95.35\n",
      "     50       98.35     98.95     97.75\n",
      "     51       97.62     97.60     97.65\n",
      "     52       98.30     98.90     97.70\n",
      "     53       97.62     97.70     97.55\n",
      "     54       98.15     98.70     97.60\n",
      "     55       97.08     96.50     97.65\n",
      "     56       98.20     98.75     97.65\n",
      "     57       97.55     97.50     97.60\n",
      "     58       98.20     98.70     97.70\n",
      "     59       98.28     98.80     97.75\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9828 \n",
      "Accuracies (test): [0.988, 0.9775]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 3\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc    t3 acc\n",
      "-------  ----------  --------  --------  --------\n",
      "      0       98.47     98.65     97.25     99.50\n",
      "      1       98.77     99.00     97.40     99.90\n",
      "      2       98.78     98.65     97.75     99.95\n",
      "      3       98.75     98.60     97.75     99.90\n",
      "      4       98.78     98.65     97.70    100.00\n",
      "      5       98.93     99.00     97.80    100.00\n",
      "      6       98.73     98.40     97.80    100.00\n",
      "      7       98.85     98.95     97.60    100.00\n",
      "      8       98.45     97.70     97.65    100.00\n",
      "      9       97.98     96.20     97.75    100.00\n",
      "     10       98.62     98.10     97.75    100.00\n",
      "     11       98.93     98.95     97.85    100.00\n",
      "     12       98.27     97.15     97.65    100.00\n",
      "     13       98.90     98.90     97.80    100.00\n",
      "     14       98.33     97.30     97.70    100.00\n",
      "     15       98.72     98.55     97.60    100.00\n",
      "     16       98.75     98.45     97.80    100.00\n",
      "     17       98.60     97.95     97.85    100.00\n",
      "     18       98.77     98.65     97.75     99.90\n",
      "     19       98.82     98.65     97.80    100.00\n",
      "     20       98.78     98.45     97.90    100.00\n",
      "     21       98.58     98.00     97.75    100.00\n",
      "     22       98.37     97.25     97.85    100.00\n",
      "     23       98.80     98.55     97.85    100.00\n",
      "     24       97.70     95.35     97.75    100.00\n",
      "     25       98.00     96.25     97.75    100.00\n",
      "     26       98.75     98.60     97.65    100.00\n",
      "     27       98.45     97.65     97.70    100.00\n",
      "     28       98.30     97.05     97.85    100.00\n",
      "     29       98.83     98.65     97.85    100.00\n",
      "     30       98.82     98.80     97.65    100.00\n",
      "     31       98.77     98.65     97.65    100.00\n",
      "     32       98.63     98.20     97.70    100.00\n",
      "     33       98.78     98.70     97.65    100.00\n",
      "     34       98.82     98.70     97.75    100.00\n",
      "     35       98.63     98.10     97.80    100.00\n",
      "     36       98.75     98.55     97.70    100.00\n",
      "     37       98.47     97.75     97.70     99.95\n",
      "     38       98.90     98.95     97.75    100.00\n",
      "     39       98.87     98.90     97.70    100.00\n",
      "     40       98.78     98.60     97.75    100.00\n",
      "     41       98.70     98.40     97.70    100.00\n",
      "     42       98.70     98.40     97.70    100.00\n",
      "     43       98.20     96.80     97.80    100.00\n",
      "     44       98.78     98.60     97.75    100.00\n",
      "     45       97.95     96.20     97.65    100.00\n",
      "     46       98.83     98.80     97.70    100.00\n",
      "     47       98.80     98.65     97.75    100.00\n",
      "     48       98.80     98.65     97.75    100.00\n",
      "     49       97.15     93.65     97.80    100.00\n",
      "     50       98.77     98.50     97.80    100.00\n",
      "     51       98.65     98.25     97.70    100.00\n",
      "     52       97.87     95.90     97.70    100.00\n",
      "     53       98.80     98.55     97.85    100.00\n",
      "     54       98.72     98.35     97.80    100.00\n",
      "     55       98.73     98.45     97.75    100.00\n",
      "     56       98.78     98.60     97.75    100.00\n",
      "     57       98.80     98.65     97.75    100.00\n",
      "     58       98.87     98.85     97.75    100.00\n",
      "     59       98.80     98.65     97.75    100.00\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9880 \n",
      "Accuracies (test): [0.9865, 0.9775, 1.0]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 4\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc    t3 acc    t4 acc\n",
      "-------  ----------  --------  --------  --------  --------\n",
      "      0       99.16     98.90     97.80    100.00     99.95\n",
      "      1       98.92     98.15     97.60    100.00     99.95\n",
      "      2       98.83     97.70     97.70     99.95     99.95\n",
      "      3       99.15     98.80     97.85    100.00     99.95\n",
      "      4       99.04     98.55     97.70     99.95     99.95\n",
      "      5       98.23     95.25     97.75    100.00     99.90\n",
      "      6       98.56     96.40     97.85    100.00    100.00\n",
      "      7       99.10     98.60     97.80    100.00    100.00\n",
      "      8       99.08     98.60     97.75    100.00     99.95\n",
      "      9       99.06     98.55     97.70    100.00    100.00\n",
      "     10       99.09     98.60     97.75    100.00    100.00\n",
      "     11       98.91     97.85     97.80    100.00    100.00\n",
      "     12       99.19     98.95     97.80    100.00    100.00\n",
      "     13       98.39     95.75     97.80    100.00    100.00\n",
      "     14       99.21     99.00     97.85    100.00    100.00\n",
      "     15       98.44     96.00     97.75    100.00    100.00\n",
      "     16       98.75     97.20     97.80    100.00    100.00\n",
      "     17       99.19     98.95     97.80    100.00    100.00\n",
      "     18       98.80     97.50     97.70    100.00    100.00\n",
      "     19       99.08     98.60     97.70    100.00    100.00\n",
      "     20       98.42     95.90     97.80    100.00    100.00\n",
      "     21       99.14     98.80     97.75    100.00    100.00\n",
      "     22       98.58     96.45     97.85    100.00    100.00\n",
      "     23       99.08     98.55     97.75    100.00    100.00\n",
      "     24       99.10     98.60     97.80    100.00    100.00\n",
      "     25       99.08     98.60     97.70    100.00    100.00\n",
      "     26       99.14     98.80     97.75    100.00    100.00\n",
      "     27       98.01     94.30     97.75    100.00    100.00\n",
      "     28       98.96     98.20     97.65    100.00    100.00\n",
      "     29       99.09     98.60     97.75    100.00    100.00\n",
      "     30       99.01     98.40     97.65    100.00    100.00\n",
      "     31       95.35     83.50     97.90    100.00    100.00\n",
      "     32       99.08     98.65     97.65    100.00    100.00\n",
      "     33       99.04     98.55     97.60    100.00    100.00\n",
      "     34       99.04     98.45     97.70    100.00    100.00\n",
      "     35       98.84     97.40     97.95    100.00    100.00\n",
      "     36       98.08     94.65     97.65    100.00    100.00\n",
      "     37       99.06     98.65     97.60    100.00    100.00\n",
      "     38       99.04     98.60     97.55    100.00    100.00\n",
      "     39       99.02     98.40     97.70    100.00    100.00\n",
      "     40       99.15     98.70     97.90    100.00    100.00\n",
      "     41       99.03     98.35     97.75    100.00    100.00\n",
      "     42       99.15     98.75     97.85    100.00    100.00\n",
      "     43       98.47     96.05     97.85    100.00    100.00\n",
      "     44       99.08     98.55     97.75    100.00    100.00\n",
      "     45       99.09     98.60     97.75    100.00    100.00\n",
      "     46       99.08     98.55     97.75    100.00    100.00\n",
      "     47       99.09     98.55     97.80    100.00    100.00\n",
      "     48       99.06     98.85     97.40    100.00    100.00\n",
      "     49       98.75     97.50     97.50    100.00    100.00\n",
      "     50       98.83     97.45     97.85    100.00    100.00\n",
      "     51       98.92     97.95     97.75    100.00    100.00\n",
      "     52       99.08     98.65     97.65    100.00    100.00\n",
      "     53       99.09     98.60     97.80     99.95    100.00\n",
      "     54       99.12     98.60     97.90    100.00    100.00\n",
      "     55       98.86     97.80     97.70     99.95    100.00\n",
      "     56       99.05     98.55     97.65    100.00    100.00\n",
      "     57       99.09     98.70     97.70     99.95    100.00\n",
      "     58       99.12     98.75     97.75    100.00    100.00\n",
      "     59       99.10     98.60     97.80    100.00    100.00\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9910 \n",
      "Accuracies (test): [0.986, 0.978, 1.0, 1.0]\n",
      "---\n",
      "\n",
      "\n",
      "Learning task 5\n",
      "Nomenclature:\n",
      "\tacc: accuracy in %\n",
      "\tt1: the first task\n",
      "\n",
      "-------  ----------  --------  --------  --------  --------  --------\n",
      "  epoch    mean acc    t1 acc    t2 acc    t3 acc    t4 acc    t5 acc\n",
      "-------  ----------  --------  --------  --------  --------  --------\n",
      "      0       99.17     98.50     97.75    100.00    100.00     99.60\n",
      "      1       99.22     98.60     97.75    100.00    100.00     99.75\n",
      "      2       99.18     98.60     97.80    100.00    100.00     99.50\n",
      "      3       99.11     98.05     97.80    100.00    100.00     99.70\n",
      "      4       99.28     98.80     97.90    100.00    100.00     99.70\n",
      "      5       99.17     98.15     97.95    100.00    100.00     99.75\n",
      "      6       99.22     98.50     97.85    100.00    100.00     99.75\n",
      "      7       99.25     98.60     97.90    100.00    100.00     99.75\n",
      "      8       99.27     98.65     97.90    100.00    100.00     99.80\n",
      "      9       98.92     97.05     97.75    100.00    100.00     99.80\n",
      "     10       99.24     98.60     97.80    100.00    100.00     99.80\n",
      "     11       99.07     97.70     97.90    100.00    100.00     99.75\n",
      "     12       99.13     98.25     97.60    100.00    100.00     99.80\n",
      "     13       99.17     98.20     97.85    100.00    100.00     99.80\n",
      "     14       98.94     97.10     97.80    100.00    100.00     99.80\n",
      "     15       98.74     96.05     97.85    100.00    100.00     99.80\n",
      "     16       99.16     98.20     97.80    100.00    100.00     99.80\n",
      "     17       99.27     98.70     97.85    100.00    100.00     99.80\n",
      "     18       99.11     98.00     97.75    100.00    100.00     99.80\n",
      "     19       99.27     98.80     97.75    100.00    100.00     99.80\n",
      "     20       99.05     97.60     97.80    100.00    100.00     99.85\n",
      "     21       99.19     98.35     97.80    100.00    100.00     99.80\n",
      "     22       99.28     98.80     97.80    100.00    100.00     99.80\n",
      "     23       98.03     92.45     97.90    100.00    100.00     99.80\n",
      "     24       99.04     97.70     97.65    100.00    100.00     99.85\n",
      "     25       99.32     98.95     97.80    100.00    100.00     99.85\n",
      "     26       99.21     98.60     97.70    100.00    100.00     99.75\n",
      "     27       98.78     96.55     97.55    100.00    100.00     99.80\n",
      "     28       99.12     98.25     97.55    100.00    100.00     99.80\n",
      "     29       99.23     98.60     97.75    100.00    100.00     99.80\n",
      "     30       98.83     96.55     97.75    100.00    100.00     99.85\n",
      "     31       99.06     97.65     97.85    100.00    100.00     99.80\n",
      "     32       98.70     95.90     97.75    100.00    100.00     99.85\n",
      "     33       99.01     97.45     97.80    100.00    100.00     99.80\n",
      "     34       99.32     98.95     97.80    100.00    100.00     99.85\n",
      "     35       99.09     98.15     97.55    100.00    100.00     99.75\n",
      "     36       99.10     97.90     97.75    100.00    100.00     99.85\n",
      "     37       99.22     98.50     97.85    100.00    100.00     99.75\n",
      "     38       99.02     98.15     97.10    100.00    100.00     99.85\n",
      "     39       99.08     98.00     97.55    100.00    100.00     99.85\n",
      "     40       99.31     99.00     97.80    100.00    100.00     99.75\n",
      "     41       99.07     97.80     97.75    100.00    100.00     99.80\n",
      "     42       99.15     98.20     97.80    100.00    100.00     99.75\n",
      "     43       99.27     98.80     97.80    100.00    100.00     99.75\n",
      "     44       99.28     98.90     97.75    100.00    100.00     99.75\n",
      "     45       97.93     92.10     97.80    100.00    100.00     99.75\n",
      "     46       99.26     98.80     97.75    100.00    100.00     99.75\n",
      "     47       99.26     98.75     97.80    100.00    100.00     99.75\n",
      "     48       98.91     97.10     97.70    100.00    100.00     99.75\n",
      "     49       99.10     98.00     97.75    100.00    100.00     99.75\n",
      "     50       99.29     98.70     97.90    100.00    100.00     99.85\n",
      "     51       99.05     97.60     97.85    100.00    100.00     99.80\n",
      "     52       99.17     98.65     97.45    100.00    100.00     99.75\n",
      "     53       98.99     97.40     97.80    100.00    100.00     99.75\n",
      "     54       99.29     98.85     97.80    100.00    100.00     99.80\n",
      "     55       98.95     97.20     97.80    100.00    100.00     99.75\n",
      "     56       99.33     99.00     97.80    100.00    100.00     99.85\n",
      "     57       98.91     97.10     97.65    100.00    100.00     99.80\n",
      "     58       99.14     98.20     97.70    100.00    100.00     99.80\n",
      "     59       99.07     97.65     97.85    100.00    100.00     99.85\n",
      "Adding inducing inputs to the coreset randomly\n",
      "For tasks seen so far, \n",
      "---\n",
      "Mean accuracy (test): 0.9907 \n",
      "Accuracies (test): [0.9765, 0.9785, 1.0, 1.0, 0.9985]\n",
      "---\n",
      "\n",
      "\n",
      "------------------- DONE -------------------\n",
      "\n",
      "\n",
      "0.9907\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "logdir = read_config_and_run(\"fsvi_no_coreset.pkl\", task_sequence)\n",
    "exp = lutils.read_exp(logdir)\n",
    "show_final_average_accuracy(exp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FRCL (with random-choice coreset) <a name=\"res4\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading experiments: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 5422.94it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading from cache:\n",
      "Running on oat18.cs.ox.ac.uk\n",
      "Running with: python /auto/users/timner/qixuan/function-space-variational-inference/fsvi_cl/baselines/frcl/run_frcl.py --dataset sfashionmnist --batch_size 128 --hidden_size 256 --n_layers 2 --learning_rate 0.001 --n_iterations_train 2000 --n_iterations_discr_search 1000 --seed 6 --n_seeds 1 --select_method random_choice --n_permuted_tasks 10 --logroot ablation --subdir reproduce_main_results_3 --save_alt --n_coreset_inputs_per_task 40 --n_omniglot_inducing_chars 2 --n_omniglot_tasks 50\n",
      "\n",
      "  0%|          | 0/2000 [00:00<?, ?it/s]\n",
      "  0%|          | 1/2000 [00:01<36:39,  1.10s/it]\n",
      "  0%|          | 7/2000 [00:01<25:45,  1.29it/s]\n",
      "  1%|          | 13/2000 [00:01<18:09,  1.82it/s]\n",
      "  1%|          | 19/2000 [00:01<12:51,  2.57it/s]\n",
      "  1%|▏         | 25/2000 [00:01<09:08,  3.60it/s]\n",
      "  2%|▏         | 31/2000 [00:01<06:33,  5.01it/s]\n",
      "  2%|▏         | 37/2000 [00:01<04:44,  6.90it/s]\n",
      "  2%|▏         | 43/2000 [00:01<03:28,  9.37it/s]\n",
      "  2%|▏         | 49/2000 [00:01<02:35, 12.51it/s]\n",
      "  3%|▎         | 55/2000 [00:02<01:59, 16.28it/s]\n",
      "  3%|▎         | 61/2000 [00:02<01:33, 20.73it/s]\n",
      "  3%|▎         | 67/2000 [00:02<01:15, 25.63it/s]\n",
      "  4%|▎         | 73/2000 [00:02<01:02, 30.70it/s]\n",
      "  4%|▍         | 79/2000 [00:02<00:53, 35.64it/s]\n",
      "  4%|▍         | 85/2000 [00:02<00:47, 40.07it/s]\n",
      "  5%|▍         | 91/2000 [00:02<00:43, 44.00it/s]\n",
      "  5%|▍         | 97/2000 [00:02<00:40, 47.24it/s]\n",
      "  5%|▌         | 103/2000 [00:02<00:38, 49.75it/s]\n",
      "  5%|▌         | 109/2000 [00:03<00:36, 51.75it/s]\n",
      "  6%|▌         | 115/2000 [00:03<00:35, 53.25it/s]\n",
      "  6%|▌         | 121/2000 [00:03<00:34, 54.35it/s]\n",
      "  6%|▋         | 127/2000 [00:03<00:34, 55.09it/s]\n",
      "  7%|▋         | 133/2000 [00:03<00:33, 55.68it/s]\n",
      "  7%|▋         | 139/2000 [00:03<00:33, 55.92it/s]\n",
      "  7%|▋         | 145/2000 [00:03<00:32, 56.22it/s]\n",
      "  8%|▊         | 151/2000 [00:03<00:32, 56.48it/s]\n",
      "  8%|▊         | 157/2000 [00:03<00:32, 56.68it/s]\n",
      "  8%|▊         | 163/2000 [00:03<00:32, 56.83it/s]\n",
      "  8%|▊         | 169/2000 [00:04<00:32, 56.92it/s]\n",
      "  9%|▉         | 175/2000 [00:04<00:32, 56.98it/s]\n",
      "  9%|▉         | 181/2000 [00:04<00:31, 57.05it/s]\n",
      "  9%|▉         | 187/2000 [00:04<00:31, 57.02it/s]\n",
      " 10%|▉         | 193/2000 [00:04<00:31, 57.00it/s]\n",
      " 10%|▉         | 199/2000 [00:04<00:31, 56.78it/s]\n",
      " 10%|█         | 205/2000 [00:04<00:31, 56.84it/s]\n",
      " 11%|█         | 211/2000 [00:04<00:31, 56.87it/s]\n",
      " 11%|█         | 217/2000 [00:04<00:31, 56.87it/s]\n",
      " 11%|█         | 223/2000 [00:05<00:31, 56.91it/s]\n",
      " 11%|█▏        | 229/2000 [00:05<00:31, 56.93it/s]\n",
      " 12%|█▏        | 235/2000 [00:05<00:31, 56.91it/s]\n",
      " 12%|█▏        | 241/2000 [00:05<00:30, 56.92it/s]\n",
      " 12%|█▏        | 247/2000 [00:05<00:30, 56.93it/s]\n",
      " 13%|█▎        | 253/2000 [00:05<00:30, 56.93it/s]\n",
      " 13%|█▎        | 259/2000 [00:05<00:30, 56.76it/s]\n",
      " 13%|█▎        | 265/2000 [00:05<00:30, 56.83it/s]\n",
      " 14%|█▎        | 271/2000 [00:05<00:30, 56.88it/s]\n",
      " 14%|█▍        | 277/2000 [00:05<00:30, 56.91it/s]\n",
      " 14%|█▍        | 283/2000 [00:06<00:30, 56.92it/s]\n",
      " 14%|█▍        | 289/2000 [00:06<00:30, 56.94it/s]\n",
      " 15%|█▍        | 295/2000 [00:06<00:29, 56.95it/s]\n",
      " 15%|█▌        | 301/2000 [00:06<00:29, 56.96it/s]\n",
      " 15%|█▌        | 307/2000 [00:06<00:29, 56.98it/s]\n",
      " 16%|█▌        | 313/2000 [00:06<00:29, 56.78it/s]\n",
      " 16%|█▌        | 319/2000 [00:06<00:29, 56.83it/s]\n",
      " 16%|█▋        | 325/2000 [00:06<00:29, 56.86it/s]\n",
      " 17%|█▋        | 331/2000 [00:06<00:29, 56.88it/s]\n",
      " 17%|█▋        | 337/2000 [00:07<00:29, 56.90it/s]\n",
      " 17%|█▋        | 343/2000 [00:07<00:29, 56.90it/s]\n",
      " 17%|█▋        | 349/2000 [00:07<00:29, 56.43it/s]\n",
      " 18%|█▊        | 355/2000 [00:07<00:28, 56.83it/s]\n",
      " 18%|█▊        | 361/2000 [00:07<00:28, 57.14it/s]\n",
      " 18%|█▊        | 367/2000 [00:07<00:28, 57.35it/s]\n",
      " 19%|█▊        | 373/2000 [00:07<00:28, 57.25it/s]\n",
      " 19%|█▉        | 379/2000 [00:07<00:28, 57.29it/s]\n",
      " 19%|█▉        | 385/2000 [00:07<00:28, 57.33it/s]\n",
      " 20%|█▉        | 391/2000 [00:07<00:28, 57.39it/s]\n",
      " 20%|█▉        | 397/2000 [00:08<00:27, 57.40it/s]\n",
      " 20%|██        | 403/2000 [00:08<00:27, 57.51it/s]\n",
      " 20%|██        | 409/2000 [00:08<00:27, 57.61it/s]\n",
      " 21%|██        | 415/2000 [00:08<00:27, 57.68it/s]\n",
      " 21%|██        | 421/2000 [00:08<00:27, 57.72it/s]\n",
      " 21%|██▏       | 427/2000 [00:08<00:27, 57.61it/s]\n",
      " 22%|██▏       | 433/2000 [00:08<00:27, 56.41it/s]\n",
      " 22%|██▏       | 439/2000 [00:08<00:27, 56.67it/s]\n",
      " 22%|██▏       | 445/2000 [00:08<00:27, 56.73it/s]\n",
      " 23%|██▎       | 451/2000 [00:09<00:27, 56.78it/s]\n",
      " 23%|██▎       | 457/2000 [00:09<00:27, 56.84it/s]\n",
      " 23%|██▎       | 463/2000 [00:09<00:27, 56.87it/s]\n",
      " 23%|██▎       | 469/2000 [00:09<00:27, 56.70it/s]\n",
      " 24%|██▍       | 475/2000 [00:09<00:26, 56.76it/s]\n",
      " 24%|██▍       | 481/2000 [00:09<00:26, 56.81it/s]\n",
      " 24%|██▍       | 487/2000 [00:09<00:26, 56.83it/s]\n",
      " 25%|██▍       | 493/2000 [00:09<00:26, 56.65it/s]\n",
      " 25%|██▍       | 499/2000 [00:09<00:26, 56.73it/s]\n",
      " 25%|██▌       | 505/2000 [00:09<00:26, 56.78it/s]\n",
      " 26%|██▌       | 511/2000 [00:10<00:26, 56.78it/s]\n",
      " 26%|██▌       | 517/2000 [00:10<00:26, 56.80it/s]\n",
      " 26%|██▌       | 523/2000 [00:10<00:25, 56.86it/s]\n",
      " 26%|██▋       | 529/2000 [00:10<00:25, 56.90it/s]\n",
      " 27%|██▋       | 535/2000 [00:10<00:25, 56.93it/s]\n",
      " 27%|██▋       | 541/2000 [00:10<00:25, 56.95it/s]\n",
      " 27%|██▋       | 547/2000 [00:10<00:25, 56.89it/s]\n",
      " 28%|██▊       | 553/2000 [00:10<00:25, 56.93it/s]\n",
      " 28%|██▊       | 559/2000 [00:10<00:25, 56.93it/s]\n",
      " 28%|██▊       | 565/2000 [00:11<00:26, 55.15it/s]\n",
      " 29%|██▊       | 571/2000 [00:11<00:25, 55.87it/s]\n",
      " 29%|██▉       | 577/2000 [00:11<00:25, 56.38it/s]\n",
      " 29%|██▉       | 583/2000 [00:11<00:24, 56.71it/s]\n",
      " 29%|██▉       | 589/2000 [00:11<00:24, 56.96it/s]\n",
      " 30%|██▉       | 595/2000 [00:11<00:24, 57.14it/s]\n",
      " 30%|███       | 601/2000 [00:11<00:24, 57.27it/s]\n",
      " 30%|███       | 607/2000 [00:11<00:24, 57.04it/s]\n",
      " 31%|███       | 613/2000 [00:11<00:24, 57.21it/s]\n",
      " 31%|███       | 619/2000 [00:11<00:24, 57.31it/s]\n",
      " 31%|███▏      | 625/2000 [00:12<00:24, 56.57it/s]\n",
      " 32%|███▏      | 631/2000 [00:12<00:24, 56.85it/s]\n",
      " 32%|███▏      | 637/2000 [00:12<00:23, 57.06it/s]\n",
      " 32%|███▏      | 643/2000 [00:12<00:23, 57.22it/s]\n",
      " 32%|███▏      | 649/2000 [00:12<00:23, 57.33it/s]\n",
      " 33%|███▎      | 655/2000 [00:12<00:23, 57.42it/s]\n",
      " 33%|███▎      | 661/2000 [00:12<00:23, 57.44it/s]\n",
      " 33%|███▎      | 667/2000 [00:12<00:23, 56.86it/s]\n",
      " 34%|███▎      | 673/2000 [00:12<00:23, 57.04it/s]\n",
      " 34%|███▍      | 679/2000 [00:13<00:23, 57.17it/s]\n",
      " 34%|███▍      | 685/2000 [00:13<00:22, 57.26it/s]\n",
      " 35%|███▍      | 691/2000 [00:13<00:22, 57.29it/s]\n",
      " 35%|███▍      | 697/2000 [00:13<00:22, 57.27it/s]\n",
      " 35%|███▌      | 703/2000 [00:13<00:22, 57.25it/s]\n",
      " 35%|███▌      | 709/2000 [00:13<00:22, 57.26it/s]\n",
      " 36%|███▌      | 715/2000 [00:13<00:22, 57.27it/s]\n",
      " 36%|███▌      | 721/2000 [00:13<00:22, 57.26it/s]\n",
      " 36%|███▋      | 727/2000 [00:13<00:22, 57.26it/s]\n",
      " 37%|███▋      | 733/2000 [00:13<00:22, 57.26it/s]\n",
      " 37%|███▋      | 739/2000 [00:14<00:22, 57.05it/s]\n",
      " 37%|███▋      | 745/2000 [00:14<00:21, 57.08it/s]\n",
      " 38%|███▊      | 751/2000 [00:14<00:21, 57.10it/s]\n",
      " 38%|███▊      | 757/2000 [00:14<00:21, 57.15it/s]\n",
      " 38%|███▊      | 763/2000 [00:14<00:21, 57.19it/s]\n",
      " 38%|███▊      | 769/2000 [00:14<00:21, 57.22it/s]\n",
      " 39%|███▉      | 775/2000 [00:14<00:21, 57.22it/s]\n",
      " 39%|███▉      | 781/2000 [00:14<00:21, 57.21it/s]\n",
      " 39%|███▉      | 787/2000 [00:14<00:21, 57.21it/s]\n",
      " 40%|███▉      | 793/2000 [00:15<00:21, 55.60it/s]\n",
      " 40%|███▉      | 799/2000 [00:15<00:21, 56.17it/s]\n",
      " 40%|████      | 805/2000 [00:15<00:21, 56.54it/s]\n",
      " 41%|████      | 811/2000 [00:15<00:20, 56.76it/s]\n",
      " 41%|████      | 817/2000 [00:15<00:20, 56.91it/s]\n",
      " 41%|████      | 823/2000 [00:15<00:20, 57.08it/s]\n",
      " 41%|████▏     | 829/2000 [00:15<00:20, 57.20it/s]\n",
      " 42%|████▏     | 835/2000 [00:15<00:20, 57.27it/s]\n",
      " 42%|████▏     | 841/2000 [00:15<00:20, 57.32it/s]\n",
      " 42%|████▏     | 847/2000 [00:15<00:20, 57.36it/s]\n",
      " 43%|████▎     | 853/2000 [00:16<00:19, 57.43it/s]\n",
      " 43%|████▎     | 859/2000 [00:16<00:19, 57.53it/s]\n",
      " 43%|████▎     | 865/2000 [00:16<00:19, 57.62it/s]\n",
      " 44%|████▎     | 871/2000 [00:16<00:19, 57.52it/s]\n",
      " 44%|████▍     | 877/2000 [00:16<00:19, 56.69it/s]\n",
      " 44%|████▍     | 883/2000 [00:16<00:19, 56.75it/s]\n",
      " 44%|████▍     | 889/2000 [00:16<00:19, 56.93it/s]\n",
      " 45%|████▍     | 895/2000 [00:16<00:19, 56.86it/s]\n",
      " 45%|████▌     | 901/2000 [00:16<00:19, 57.05it/s]\n",
      " 45%|████▌     | 907/2000 [00:17<00:19, 57.18it/s]\n",
      " 46%|████▌     | 913/2000 [00:17<00:18, 57.28it/s]\n",
      " 46%|████▌     | 919/2000 [00:17<00:18, 57.37it/s]\n",
      " 46%|████▋     | 925/2000 [00:17<00:18, 57.42it/s]\n",
      " 47%|████▋     | 931/2000 [00:17<00:18, 57.44it/s]\n",
      " 47%|████▋     | 937/2000 [00:17<00:18, 57.47it/s]\n",
      " 47%|████▋     | 943/2000 [00:17<00:18, 56.77it/s]\n",
      " 47%|████▋     | 949/2000 [00:17<00:18, 56.99it/s]\n",
      " 48%|████▊     | 955/2000 [00:17<00:18, 57.23it/s]\n",
      " 48%|████▊     | 961/2000 [00:17<00:18, 57.41it/s]\n",
      " 48%|████▊     | 967/2000 [00:18<00:18, 57.30it/s]\n",
      " 49%|████▊     | 973/2000 [00:18<00:17, 57.27it/s]\n",
      " 49%|████▉     | 979/2000 [00:18<00:17, 57.23it/s]\n",
      " 49%|████▉     | 985/2000 [00:18<00:17, 57.20it/s]\n",
      " 50%|████▉     | 991/2000 [00:18<00:17, 57.19it/s]\n",
      " 50%|████▉     | 997/2000 [00:18<00:17, 57.19it/s]\n",
      " 50%|█████     | 1003/2000 [00:18<00:17, 57.20it/s]\n",
      " 50%|█████     | 1009/2000 [00:18<00:17, 57.09it/s]\n",
      " 51%|█████     | 1015/2000 [00:18<00:17, 57.12it/s]\n",
      " 51%|█████     | 1021/2000 [00:18<00:17, 57.14it/s]\n",
      " 51%|█████▏    | 1027/2000 [00:19<00:17, 57.15it/s]\n",
      " 52%|█████▏    | 1033/2000 [00:19<00:16, 57.20it/s]\n",
      " 52%|█████▏    | 1039/2000 [00:19<00:16, 57.23it/s]\n",
      " 52%|█████▏    | 1045/2000 [00:19<00:16, 57.00it/s]\n",
      " 53%|█████▎    | 1051/2000 [00:19<00:16, 57.08it/s]\n",
      " 53%|█████▎    | 1057/2000 [00:19<00:16, 57.14it/s]\n",
      " 53%|█████▎    | 1063/2000 [00:19<00:16, 57.20it/s]\n",
      " 53%|█████▎    | 1069/2000 [00:19<00:16, 57.23it/s]\n",
      " 54%|█████▍    | 1075/2000 [00:19<00:16, 57.26it/s]\n",
      " 54%|█████▍    | 1081/2000 [00:20<00:16, 57.25it/s]\n",
      " 54%|█████▍    | 1087/2000 [00:20<00:16, 56.53it/s]\n",
      " 55%|█████▍    | 1093/2000 [00:20<00:15, 56.83it/s]\n",
      " 55%|█████▍    | 1099/2000 [00:20<00:15, 57.04it/s]\n",
      " 55%|█████▌    | 1105/2000 [00:20<00:15, 57.10it/s]\n",
      " 56%|█████▌    | 1111/2000 [00:20<00:15, 57.15it/s]\n",
      " 56%|█████▌    | 1117/2000 [00:20<00:15, 57.18it/s]\n",
      " 56%|█████▌    | 1123/2000 [00:20<00:15, 57.01it/s]\n",
      " 56%|█████▋    | 1129/2000 [00:20<00:15, 57.11it/s]\n",
      " 57%|█████▋    | 1135/2000 [00:20<00:15, 57.14it/s]\n",
      " 57%|█████▋    | 1141/2000 [00:21<00:15, 57.18it/s]\n",
      " 57%|█████▋    | 1147/2000 [00:21<00:14, 57.14it/s]\n",
      " 58%|█████▊    | 1153/2000 [00:21<00:14, 57.13it/s]\n",
      " 58%|█████▊    | 1159/2000 [00:21<00:14, 57.11it/s]\n",
      " 58%|█████▊    | 1165/2000 [00:21<00:14, 57.09it/s]\n",
      " 59%|█████▊    | 1171/2000 [00:21<00:14, 57.07it/s]\n",
      " 59%|█████▉    | 1177/2000 [00:21<00:14, 57.05it/s]\n",
      " 59%|█████▉    | 1183/2000 [00:21<00:14, 57.04it/s]\n",
      " 59%|█████▉    | 1189/2000 [00:21<00:14, 57.03it/s]\n",
      " 60%|█████▉    | 1195/2000 [00:22<00:14, 56.92it/s]\n",
      " 60%|██████    | 1201/2000 [00:22<00:14, 56.84it/s]\n",
      " 60%|██████    | 1207/2000 [00:22<00:14, 55.37it/s]\n",
      " 61%|██████    | 1213/2000 [00:22<00:14, 55.85it/s]\n",
      " 61%|██████    | 1219/2000 [00:22<00:13, 56.23it/s]\n",
      " 61%|██████▏   | 1225/2000 [00:22<00:13, 56.62it/s]\n",
      " 62%|██████▏   | 1231/2000 [00:22<00:13, 56.89it/s]\n",
      " 62%|██████▏   | 1237/2000 [00:22<00:13, 57.08it/s]\n",
      " 62%|██████▏   | 1243/2000 [00:22<00:13, 57.10it/s]\n",
      " 62%|██████▏   | 1249/2000 [00:23<00:13, 57.12it/s]\n",
      " 63%|██████▎   | 1255/2000 [00:23<00:13, 57.12it/s]\n",
      " 63%|██████▎   | 1261/2000 [00:23<00:12, 57.13it/s]\n",
      " 63%|██████▎   | 1267/2000 [00:23<00:12, 57.13it/s]\n",
      " 64%|██████▎   | 1273/2000 [00:23<00:12, 57.00it/s]\n",
      " 64%|██████▍   | 1279/2000 [00:23<00:12, 57.04it/s]\n",
      " 64%|██████▍   | 1285/2000 [00:23<00:12, 55.88it/s]\n",
      " 65%|██████▍   | 1291/2000 [00:23<00:12, 56.23it/s]\n",
      " 65%|██████▍   | 1297/2000 [00:23<00:12, 56.54it/s]\n",
      " 65%|██████▌   | 1303/2000 [00:23<00:12, 56.75it/s]\n",
      " 65%|██████▌   | 1309/2000 [00:24<00:12, 56.83it/s]\n",
      " 66%|██████▌   | 1315/2000 [00:24<00:12, 56.88it/s]\n",
      " 66%|██████▌   | 1321/2000 [00:24<00:11, 56.92it/s]\n",
      " 66%|██████▋   | 1327/2000 [00:24<00:11, 56.35it/s]\n",
      " 67%|██████▋   | 1333/2000 [00:24<00:11, 56.79it/s]\n",
      " 67%|██████▋   | 1339/2000 [00:24<00:11, 57.09it/s]\n",
      " 67%|██████▋   | 1345/2000 [00:24<00:11, 56.95it/s]\n",
      " 68%|██████▊   | 1351/2000 [00:24<00:11, 57.12it/s]\n",
      " 68%|██████▊   | 1357/2000 [00:24<00:11, 57.17it/s]\n",
      " 68%|██████▊   | 1363/2000 [00:25<00:11, 57.20it/s]\n",
      " 68%|██████▊   | 1369/2000 [00:25<00:11, 57.21it/s]\n",
      " 69%|██████▉   | 1375/2000 [00:25<00:11, 56.60it/s]\n",
      " 69%|██████▉   | 1381/2000 [00:25<00:10, 56.85it/s]\n",
      " 69%|██████▉   | 1387/2000 [00:25<00:10, 57.06it/s]\n",
      " 70%|██████▉   | 1393/2000 [00:25<00:10, 57.14it/s]\n",
      " 70%|██████▉   | 1399/2000 [00:25<00:10, 57.16it/s]\n",
      " 70%|███████   | 1405/2000 [00:25<00:10, 57.17it/s]\n",
      " 71%|███████   | 1411/2000 [00:25<00:10, 57.16it/s]\n",
      " 71%|███████   | 1417/2000 [00:25<00:10, 57.15it/s]\n",
      " 71%|███████   | 1423/2000 [00:26<00:10, 56.98it/s]\n",
      " 71%|███████▏  | 1429/2000 [00:26<00:10, 57.03it/s]\n",
      " 72%|███████▏  | 1435/2000 [00:26<00:09, 57.07it/s]\n",
      " 72%|███████▏  | 1441/2000 [00:26<00:09, 57.08it/s]\n",
      " 72%|███████▏  | 1447/2000 [00:26<00:09, 57.11it/s]\n",
      " 73%|███████▎  | 1453/2000 [00:26<00:09, 57.18it/s]\n",
      " 73%|███████▎  | 1459/2000 [00:26<00:09, 57.28it/s]\n",
      " 73%|███████▎  | 1465/2000 [00:26<00:09, 57.22it/s]\n",
      " 74%|███████▎  | 1471/2000 [00:26<00:09, 57.11it/s]\n",
      " 74%|███████▍  | 1477/2000 [00:27<00:09, 57.05it/s]\n",
      " 74%|███████▍  | 1483/2000 [00:27<00:09, 55.47it/s]\n",
      " 74%|███████▍  | 1489/2000 [00:27<00:09, 55.96it/s]\n",
      " 75%|███████▍  | 1495/2000 [00:27<00:08, 56.31it/s]\n",
      " 75%|███████▌  | 1501/2000 [00:27<00:08, 56.56it/s]\n",
      " 75%|███████▌  | 1507/2000 [00:27<00:08, 56.75it/s]\n",
      " 76%|███████▌  | 1513/2000 [00:27<00:08, 56.67it/s]\n",
      " 76%|███████▌  | 1519/2000 [00:27<00:08, 56.79it/s]\n",
      " 76%|███████▋  | 1525/2000 [00:27<00:08, 56.90it/s]\n",
      " 77%|███████▋  | 1531/2000 [00:27<00:08, 56.97it/s]\n",
      " 77%|███████▋  | 1537/2000 [00:28<00:08, 57.04it/s]\n",
      " 77%|███████▋  | 1543/2000 [00:28<00:07, 57.13it/s]\n",
      " 77%|███████▋  | 1549/2000 [00:28<00:07, 57.24it/s]\n",
      " 78%|███████▊  | 1555/2000 [00:28<00:07, 57.32it/s]\n",
      " 78%|███████▊  | 1561/2000 [00:28<00:07, 57.40it/s]\n",
      " 78%|███████▊  | 1567/2000 [00:28<00:07, 57.43it/s]\n",
      " 79%|███████▊  | 1573/2000 [00:28<00:07, 57.43it/s]\n",
      " 79%|███████▉  | 1579/2000 [00:28<00:07, 57.47it/s]\n",
      " 79%|███████▉  | 1585/2000 [00:28<00:07, 57.51it/s]\n",
      " 80%|███████▉  | 1591/2000 [00:29<00:07, 57.53it/s]\n",
      " 80%|███████▉  | 1597/2000 [00:29<00:07, 57.55it/s]\n",
      " 80%|████████  | 1603/2000 [00:29<00:06, 57.49it/s]\n",
      " 80%|████████  | 1609/2000 [00:29<00:06, 57.48it/s]\n",
      " 81%|████████  | 1615/2000 [00:29<00:06, 57.50it/s]\n",
      " 81%|████████  | 1621/2000 [00:29<00:06, 57.36it/s]\n",
      " 81%|████████▏ | 1627/2000 [00:29<00:06, 57.23it/s]\n",
      " 82%|████████▏ | 1633/2000 [00:29<00:06, 57.14it/s]\n",
      " 82%|████████▏ | 1639/2000 [00:29<00:06, 57.06it/s]\n",
      " 82%|████████▏ | 1645/2000 [00:29<00:06, 57.01it/s]\n",
      " 83%|████████▎ | 1651/2000 [00:30<00:06, 56.24it/s]\n",
      " 83%|████████▎ | 1657/2000 [00:30<00:06, 56.62it/s]\n",
      " 83%|████████▎ | 1663/2000 [00:30<00:05, 56.79it/s]\n",
      " 83%|████████▎ | 1669/2000 [00:30<00:05, 56.93it/s]\n",
      " 84%|████████▍ | 1675/2000 [00:30<00:05, 56.94it/s]\n",
      " 84%|████████▍ | 1681/2000 [00:30<00:05, 57.02it/s]\n",
      " 84%|████████▍ | 1687/2000 [00:30<00:05, 57.07it/s]\n",
      " 85%|████████▍ | 1693/2000 [00:30<00:05, 57.01it/s]\n",
      " 85%|████████▍ | 1699/2000 [00:30<00:05, 57.06it/s]\n",
      " 85%|████████▌ | 1705/2000 [00:31<00:05, 57.10it/s]\n",
      " 86%|████████▌ | 1711/2000 [00:31<00:05, 57.14it/s]\n",
      " 86%|████████▌ | 1717/2000 [00:31<00:04, 57.16it/s]\n",
      " 86%|████████▌ | 1723/2000 [00:31<00:04, 57.18it/s]\n",
      " 86%|████████▋ | 1729/2000 [00:31<00:04, 57.19it/s]\n",
      " 87%|████████▋ | 1735/2000 [00:31<00:04, 57.19it/s]\n",
      " 87%|████████▋ | 1741/2000 [00:31<00:04, 57.18it/s]\n",
      " 87%|████████▋ | 1747/2000 [00:31<00:04, 57.18it/s]\n",
      " 88%|████████▊ | 1753/2000 [00:31<00:04, 57.18it/s]\n",
      " 88%|████████▊ | 1759/2000 [00:31<00:04, 55.64it/s]\n",
      " 88%|████████▊ | 1765/2000 [00:32<00:04, 56.09it/s]\n",
      " 89%|████████▊ | 1771/2000 [00:32<00:04, 56.40it/s]\n",
      " 89%|████████▉ | 1777/2000 [00:32<00:03, 56.62it/s]\n",
      " 89%|████████▉ | 1783/2000 [00:32<00:03, 56.78it/s]\n",
      " 89%|████████▉ | 1789/2000 [00:32<00:03, 56.87it/s]\n",
      " 90%|████████▉ | 1795/2000 [00:32<00:03, 57.05it/s]\n",
      " 90%|█████████ | 1801/2000 [00:32<00:03, 57.07it/s]\n",
      " 90%|█████████ | 1807/2000 [00:32<00:03, 57.02it/s]\n",
      " 91%|█████████ | 1813/2000 [00:32<00:03, 56.99it/s]\n",
      " 91%|█████████ | 1819/2000 [00:33<00:03, 56.97it/s]\n",
      " 91%|█████████▏| 1825/2000 [00:33<00:03, 55.69it/s]\n",
      " 92%|█████████▏| 1831/2000 [00:33<00:03, 56.20it/s]\n",
      " 92%|█████████▏| 1837/2000 [00:33<00:02, 56.57it/s]\n",
      " 92%|█████████▏| 1843/2000 [00:33<00:02, 56.83it/s]\n",
      " 92%|█████████▏| 1849/2000 [00:33<00:02, 57.00it/s]\n",
      " 93%|█████████▎| 1855/2000 [00:33<00:02, 57.12it/s]\n",
      " 93%|█████████▎| 1861/2000 [00:33<00:02, 57.22it/s]\n",
      " 93%|█████████▎| 1867/2000 [00:33<00:02, 57.30it/s]\n",
      " 94%|█████████▎| 1873/2000 [00:33<00:02, 57.34it/s]\n",
      " 94%|█████████▍| 1879/2000 [00:34<00:02, 57.31it/s]\n",
      " 94%|█████████▍| 1885/2000 [00:34<00:02, 57.33it/s]\n",
      " 95%|█████████▍| 1891/2000 [00:34<00:01, 57.38it/s]\n",
      " 95%|█████████▍| 1897/2000 [00:34<00:01, 57.39it/s]\n",
      " 95%|█████████▌| 1903/2000 [00:34<00:01, 57.40it/s]\n",
      " 95%|█████████▌| 1909/2000 [00:34<00:01, 57.41it/s]\n",
      " 96%|█████████▌| 1915/2000 [00:34<00:01, 57.41it/s]\n",
      " 96%|█████████▌| 1921/2000 [00:34<00:01, 57.42it/s]\n",
      " 96%|█████████▋| 1927/2000 [00:34<00:01, 57.44it/s]\n",
      " 97%|█████████▋| 1933/2000 [00:35<00:01, 57.44it/s]\n",
      " 97%|█████████▋| 1939/2000 [00:35<00:01, 57.43it/s]\n",
      " 97%|█████████▋| 1945/2000 [00:35<00:00, 57.43it/s]\n",
      " 98%|█████████▊| 1951/2000 [00:35<00:00, 57.42it/s]\n",
      " 98%|█████████▊| 1957/2000 [00:35<00:00, 57.43it/s]\n",
      " 98%|█████████▊| 1963/2000 [00:35<00:00, 57.43it/s]\n",
      " 98%|█████████▊| 1969/2000 [00:35<00:00, 57.17it/s]\n",
      " 99%|█████████▉| 1975/2000 [00:35<00:00, 57.13it/s]\n",
      " 99%|█████████▉| 1981/2000 [00:35<00:00, 57.11it/s]\n",
      " 99%|█████████▉| 1987/2000 [00:35<00:00, 57.11it/s]\n",
      "100%|█████████▉| 1993/2000 [00:36<00:00, 57.10it/s]\n",
      "100%|█████████▉| 1999/2000 [00:36<00:00, 57.08it/s]\n",
      "100%|██████████| 2000/2000 [00:36<00:00, 55.29it/s]\n",
      "\n",
      "  0%|          | 0/2000 [00:00<?, ?it/s]\n",
      "  0%|          | 4/2000 [00:00<00:52, 37.77it/s]\n",
      "  0%|          | 9/2000 [00:00<00:51, 38.82it/s]\n",
      "  1%|          | 14/2000 [00:00<00:50, 39.58it/s]\n",
      "  1%|          | 19/2000 [00:00<00:49, 40.11it/s]\n",
      "  1%|          | 24/2000 [00:00<00:48, 40.48it/s]\n",
      "  1%|▏         | 29/2000 [00:00<00:48, 40.75it/s]\n",
      "  2%|▏         | 33/2000 [00:00<00:49, 40.08it/s]\n",
      "  2%|▏         | 38/2000 [00:00<00:48, 40.51it/s]\n",
      "  2%|▏         | 43/2000 [00:01<00:47, 40.85it/s]\n",
      "  2%|▏         | 48/2000 [00:01<00:47, 41.07it/s]\n",
      "  3%|▎         | 53/2000 [00:01<00:47, 41.25it/s]\n",
      "  3%|▎         | 58/2000 [00:01<00:46, 41.33it/s]\n",
      "  3%|▎         | 63/2000 [00:01<00:46, 41.37it/s]\n",
      "  3%|▎         | 68/2000 [00:01<00:46, 41.47it/s]\n",
      "  4%|▎         | 73/2000 [00:01<00:46, 41.48it/s]\n",
      "  4%|▍         | 78/2000 [00:01<00:47, 40.67it/s]\n",
      "  4%|▍         | 83/2000 [00:02<00:46, 40.95it/s]\n",
      "  4%|▍         | 88/2000 [00:02<00:46, 41.04it/s]\n",
      "  5%|▍         | 93/2000 [00:02<00:46, 41.22it/s]\n",
      "  5%|▍         | 98/2000 [00:02<00:45, 41.35it/s]\n",
      "  5%|▌         | 103/2000 [00:02<00:46, 40.69it/s]\n",
      "  5%|▌         | 108/2000 [00:02<00:46, 40.90it/s]\n",
      "  6%|▌         | 113/2000 [00:02<00:45, 41.06it/s]\n",
      "  6%|▌         | 118/2000 [00:02<00:45, 41.17it/s]\n",
      "  6%|▌         | 123/2000 [00:02<00:45, 41.26it/s]\n",
      "  6%|▋         | 128/2000 [00:03<00:45, 41.32it/s]\n",
      "  7%|▋         | 133/2000 [00:03<00:45, 41.36it/s]\n",
      "  7%|▋         | 138/2000 [00:03<00:44, 41.38it/s]\n",
      "  7%|▋         | 143/2000 [00:03<00:44, 41.40it/s]\n",
      "  7%|▋         | 148/2000 [00:03<00:44, 41.28it/s]\n",
      "  8%|▊         | 153/2000 [00:03<00:44, 41.40it/s]\n",
      "  8%|▊         | 158/2000 [00:03<00:44, 41.35it/s]\n",
      "  8%|▊         | 163/2000 [00:03<00:44, 41.43it/s]\n",
      "  8%|▊         | 168/2000 [00:04<00:44, 41.49it/s]\n",
      "  9%|▊         | 173/2000 [00:04<00:43, 41.56it/s]\n",
      "  9%|▉         | 178/2000 [00:04<00:43, 41.59it/s]\n",
      "  9%|▉         | 183/2000 [00:04<00:43, 41.62it/s]\n",
      "  9%|▉         | 188/2000 [00:04<00:43, 41.61it/s]\n",
      " 10%|▉         | 193/2000 [00:04<00:43, 41.54it/s]\n",
      " 10%|▉         | 198/2000 [00:04<00:43, 41.57it/s]\n",
      " 10%|█         | 203/2000 [00:04<00:43, 41.59it/s]\n",
      " 10%|█         | 208/2000 [00:05<00:43, 41.62it/s]\n",
      " 11%|█         | 213/2000 [00:05<00:42, 41.63it/s]\n",
      " 11%|█         | 218/2000 [00:05<00:42, 41.65it/s]\n",
      " 11%|█         | 223/2000 [00:05<00:42, 41.40it/s]\n",
      " 11%|█▏        | 228/2000 [00:05<00:42, 41.48it/s]\n",
      " 12%|█▏        | 233/2000 [00:05<00:42, 41.55it/s]\n",
      " 12%|█▏        | 238/2000 [00:05<00:42, 41.60it/s]\n",
      " 12%|█▏        | 243/2000 [00:05<00:42, 41.43it/s]\n",
      " 12%|█▏        | 248/2000 [00:06<00:42, 41.37it/s]\n",
      " 13%|█▎        | 253/2000 [00:06<00:42, 41.33it/s]\n",
      " 13%|█▎        | 258/2000 [00:06<00:42, 41.29it/s]\n",
      " 13%|█▎        | 263/2000 [00:06<00:42, 41.28it/s]\n",
      " 13%|█▎        | 268/2000 [00:06<00:41, 41.28it/s]\n",
      " 14%|█▎        | 273/2000 [00:06<00:41, 41.28it/s]\n",
      " 14%|█▍        | 278/2000 [00:06<00:41, 41.27it/s]\n",
      " 14%|█▍        | 283/2000 [00:06<00:41, 41.27it/s]\n",
      " 14%|█▍        | 288/2000 [00:06<00:41, 41.26it/s]\n",
      " 15%|█▍        | 293/2000 [00:07<00:41, 41.26it/s]\n",
      " 15%|█▍        | 298/2000 [00:07<00:41, 41.27it/s]\n",
      " 15%|█▌        | 303/2000 [00:07<00:41, 40.79it/s]\n",
      " 15%|█▌        | 308/2000 [00:07<00:41, 41.12it/s]\n",
      " 16%|█▌        | 313/2000 [00:07<00:40, 41.34it/s]\n",
      " 16%|█▌        | 318/2000 [00:07<00:41, 40.62it/s]\n",
      " 16%|█▌        | 323/2000 [00:07<00:41, 40.85it/s]\n",
      " 16%|█▋        | 328/2000 [00:07<00:40, 41.05it/s]\n",
      " 17%|█▋        | 333/2000 [00:08<00:40, 41.21it/s]\n",
      " 17%|█▋        | 338/2000 [00:08<00:40, 41.34it/s]\n",
      " 17%|█▋        | 343/2000 [00:08<00:39, 41.43it/s]\n",
      " 17%|█▋        | 348/2000 [00:08<00:40, 41.04it/s]\n",
      " 18%|█▊        | 353/2000 [00:08<00:39, 41.25it/s]\n",
      " 18%|█▊        | 358/2000 [00:08<00:39, 41.33it/s]\n",
      " 18%|█▊        | 363/2000 [00:08<00:39, 41.43it/s]\n",
      " 18%|█▊        | 368/2000 [00:08<00:39, 41.48it/s]\n",
      " 19%|█▊        | 373/2000 [00:09<00:39, 41.54it/s]\n",
      " 19%|█▉        | 378/2000 [00:09<00:39, 41.54it/s]\n",
      " 19%|█▉        | 383/2000 [00:09<00:38, 41.56it/s]\n",
      " 19%|█▉        | 388/2000 [00:09<00:38, 41.59it/s]\n",
      " 20%|█▉        | 393/2000 [00:09<00:38, 41.61it/s]\n",
      " 20%|█▉        | 398/2000 [00:09<00:38, 41.63it/s]\n",
      " 20%|██        | 403/2000 [00:09<00:38, 41.64it/s]\n",
      " 20%|██        | 408/2000 [00:09<00:38, 41.65it/s]\n",
      " 21%|██        | 413/2000 [00:09<00:38, 41.62it/s]\n",
      " 21%|██        | 418/2000 [00:10<00:38, 40.97it/s]\n",
      " 21%|██        | 423/2000 [00:10<00:38, 41.16it/s]\n",
      " 21%|██▏       | 428/2000 [00:10<00:38, 41.29it/s]\n",
      " 22%|██▏       | 433/2000 [00:10<00:37, 41.38it/s]\n",
      " 22%|██▏       | 438/2000 [00:10<00:37, 41.46it/s]\n",
      " 22%|██▏       | 443/2000 [00:10<00:37, 41.51it/s]\n",
      " 22%|██▏       | 448/2000 [00:10<00:37, 41.55it/s]\n",
      " 23%|██▎       | 453/2000 [00:10<00:37, 41.58it/s]\n",
      " 23%|██▎       | 458/2000 [00:11<00:37, 41.61it/s]\n",
      " 23%|██▎       | 463/2000 [00:11<00:36, 41.62it/s]\n",
      " 23%|██▎       | 468/2000 [00:11<00:36, 41.63it/s]\n",
      " 24%|██▎       | 473/2000 [00:11<00:36, 41.64it/s]\n",
      " 24%|██▍       | 478/2000 [00:11<00:36, 41.62it/s]\n",
      " 24%|██▍       | 483/2000 [00:11<00:36, 41.61it/s]\n",
      " 24%|██▍       | 488/2000 [00:11<00:36, 41.60it/s]\n",
      " 25%|██▍       | 493/2000 [00:11<00:36, 41.61it/s]\n",
      " 25%|██▍       | 498/2000 [00:12<00:36, 41.61it/s]\n",
      " 25%|██▌       | 503/2000 [00:12<00:35, 41.60it/s]\n",
      " 25%|██▌       | 508/2000 [00:12<00:35, 41.59it/s]\n",
      " 26%|██▌       | 513/2000 [00:12<00:35, 41.64it/s]\n",
      " 26%|██▌       | 518/2000 [00:12<00:35, 41.70it/s]\n",
      " 26%|██▌       | 523/2000 [00:12<00:35, 41.75it/s]\n",
      " 26%|██▋       | 528/2000 [00:12<00:35, 41.78it/s]\n",
      " 27%|██▋       | 533/2000 [00:12<00:35, 41.80it/s]\n",
      " 27%|██▋       | 538/2000 [00:13<00:35, 41.73it/s]\n",
      " 27%|██▋       | 543/2000 [00:13<00:34, 41.75it/s]\n",
      " 27%|██▋       | 548/2000 [00:13<00:34, 41.68it/s]\n",
      " 28%|██▊       | 553/2000 [00:13<00:34, 41.61it/s]\n",
      " 28%|██▊       | 558/2000 [00:13<00:34, 41.55it/s]\n",
      " 28%|██▊       | 563/2000 [00:13<00:34, 41.52it/s]\n",
      " 28%|██▊       | 568/2000 [00:13<00:34, 41.49it/s]\n",
      " 29%|██▊       | 573/2000 [00:13<00:34, 40.85it/s]\n",
      " 29%|██▉       | 578/2000 [00:13<00:34, 41.02it/s]\n",
      " 29%|██▉       | 583/2000 [00:14<00:34, 41.15it/s]\n",
      " 29%|██▉       | 588/2000 [00:14<00:34, 41.24it/s]\n",
      " 30%|██▉       | 593/2000 [00:14<00:34, 41.31it/s]\n",
      " 30%|██▉       | 598/2000 [00:14<00:33, 41.28it/s]\n",
      " 30%|███       | 603/2000 [00:14<00:33, 41.25it/s]\n",
      " 30%|███       | 608/2000 [00:14<00:33, 41.24it/s]\n",
      " 31%|███       | 613/2000 [00:14<00:33, 41.23it/s]\n",
      " 31%|███       | 618/2000 [00:14<00:33, 41.20it/s]\n",
      " 31%|███       | 623/2000 [00:15<00:33, 41.20it/s]\n",
      " 31%|███▏      | 628/2000 [00:15<00:33, 41.22it/s]\n",
      " 32%|███▏      | 633/2000 [00:15<00:33, 41.21it/s]\n",
      " 32%|███▏      | 638/2000 [00:15<00:33, 41.21it/s]\n",
      " 32%|███▏      | 643/2000 [00:15<00:32, 41.20it/s]\n",
      " 32%|███▏      | 648/2000 [00:15<00:32, 41.19it/s]\n",
      " 33%|███▎      | 653/2000 [00:15<00:32, 41.19it/s]\n",
      " 33%|███▎      | 658/2000 [00:15<00:32, 41.20it/s]\n",
      " 33%|███▎      | 663/2000 [00:16<00:32, 41.20it/s]\n",
      " 33%|███▎      | 668/2000 [00:16<00:32, 41.19it/s]\n",
      " 34%|███▎      | 673/2000 [00:16<00:32, 41.18it/s]\n",
      " 34%|███▍      | 678/2000 [00:16<00:32, 41.19it/s]\n",
      " 34%|███▍      | 683/2000 [00:16<00:31, 41.19it/s]\n",
      " 34%|███▍      | 688/2000 [00:16<00:31, 41.18it/s]\n",
      " 35%|███▍      | 693/2000 [00:16<00:31, 41.18it/s]\n",
      " 35%|███▍      | 698/2000 [00:16<00:31, 41.15it/s]\n",
      " 35%|███▌      | 703/2000 [00:17<00:31, 41.15it/s]\n",
      " 35%|███▌      | 708/2000 [00:17<00:31, 40.88it/s]\n",
      " 36%|███▌      | 713/2000 [00:17<00:31, 41.15it/s]\n",
      " 36%|███▌      | 718/2000 [00:17<00:31, 41.25it/s]\n",
      " 36%|███▌      | 723/2000 [00:17<00:30, 41.30it/s]\n",
      " 36%|███▋      | 728/2000 [00:17<00:30, 41.35it/s]\n",
      " 37%|███▋      | 733/2000 [00:17<00:30, 41.38it/s]\n",
      " 37%|███▋      | 738/2000 [00:17<00:30, 41.38it/s]\n",
      " 37%|███▋      | 743/2000 [00:17<00:30, 41.39it/s]\n",
      " 37%|███▋      | 748/2000 [00:18<00:30, 41.39it/s]\n",
      " 38%|███▊      | 753/2000 [00:18<00:30, 41.40it/s]\n",
      " 38%|███▊      | 758/2000 [00:18<00:30, 40.84it/s]\n",
      " 38%|███▊      | 763/2000 [00:18<00:30, 40.76it/s]\n",
      " 38%|███▊      | 768/2000 [00:18<00:30, 40.28it/s]\n",
      " 39%|███▊      | 773/2000 [00:18<00:30, 40.55it/s]\n",
      " 39%|███▉      | 778/2000 [00:18<00:29, 40.84it/s]\n",
      " 39%|███▉      | 783/2000 [00:18<00:29, 41.06it/s]\n",
      " 39%|███▉      | 788/2000 [00:19<00:29, 41.20it/s]\n",
      " 40%|███▉      | 793/2000 [00:19<00:29, 41.31it/s]\n",
      " 40%|███▉      | 798/2000 [00:19<00:29, 41.39it/s]\n",
      " 40%|████      | 803/2000 [00:19<00:28, 41.46it/s]\n",
      " 40%|████      | 808/2000 [00:19<00:28, 41.51it/s]\n",
      " 41%|████      | 813/2000 [00:19<00:28, 41.55it/s]\n",
      " 41%|████      | 818/2000 [00:19<00:28, 41.56it/s]\n",
      " 41%|████      | 823/2000 [00:19<00:28, 41.56it/s]\n",
      " 41%|████▏     | 828/2000 [00:20<00:28, 41.53it/s]\n",
      " 42%|████▏     | 833/2000 [00:20<00:28, 41.53it/s]\n",
      " 42%|████▏     | 838/2000 [00:20<00:27, 41.53it/s]\n",
      " 42%|████▏     | 843/2000 [00:20<00:27, 41.53it/s]\n",
      " 42%|████▏     | 848/2000 [00:20<00:27, 41.53it/s]\n",
      " 43%|████▎     | 853/2000 [00:20<00:27, 41.44it/s]\n",
      " 43%|████▎     | 858/2000 [00:20<00:27, 41.41it/s]\n",
      " 43%|████▎     | 863/2000 [00:20<00:27, 41.39it/s]\n",
      " 43%|████▎     | 868/2000 [00:21<00:27, 41.37it/s]\n",
      " 44%|████▎     | 873/2000 [00:21<00:27, 41.36it/s]\n",
      " 44%|████▍     | 878/2000 [00:21<00:27, 41.36it/s]\n",
      " 44%|████▍     | 883/2000 [00:21<00:27, 41.35it/s]\n",
      " 44%|████▍     | 888/2000 [00:21<00:26, 41.34it/s]\n",
      " 45%|████▍     | 893/2000 [00:21<00:26, 41.33it/s]\n",
      " 45%|████▍     | 898/2000 [00:21<00:26, 41.33it/s]\n",
      " 45%|████▌     | 903/2000 [00:21<00:26, 41.33it/s]\n",
      " 45%|████▌     | 908/2000 [00:21<00:26, 41.33it/s]\n",
      " 46%|████▌     | 913/2000 [00:22<00:26, 41.33it/s]\n",
      " 46%|████▌     | 918/2000 [00:22<00:26, 41.32it/s]\n",
      " 46%|████▌     | 923/2000 [00:22<00:26, 41.33it/s]\n",
      " 46%|████▋     | 928/2000 [00:22<00:25, 41.33it/s]\n",
      " 47%|████▋     | 933/2000 [00:22<00:25, 41.32it/s]\n",
      " 47%|████▋     | 938/2000 [00:22<00:25, 41.34it/s]\n",
      " 47%|████▋     | 943/2000 [00:22<00:25, 41.35it/s]\n",
      " 47%|████▋     | 948/2000 [00:22<00:25, 41.37it/s]\n",
      " 48%|████▊     | 953/2000 [00:23<00:25, 41.42it/s]\n",
      " 48%|████▊     | 958/2000 [00:23<00:25, 41.46it/s]\n",
      " 48%|████▊     | 963/2000 [00:23<00:24, 41.48it/s]\n",
      " 48%|████▊     | 968/2000 [00:23<00:24, 41.49it/s]\n",
      " 49%|████▊     | 973/2000 [00:23<00:24, 41.52it/s]\n",
      " 49%|████▉     | 978/2000 [00:23<00:24, 41.52it/s]\n",
      " 49%|████▉     | 983/2000 [00:23<00:24, 41.53it/s]\n",
      " 49%|████▉     | 988/2000 [00:23<00:24, 41.54it/s]\n",
      " 50%|████▉     | 993/2000 [00:24<00:24, 41.54it/s]\n",
      " 50%|████▉     | 998/2000 [00:24<00:24, 41.54it/s]\n",
      " 50%|█████     | 1003/2000 [00:24<00:23, 41.55it/s]\n",
      " 50%|█████     | 1008/2000 [00:24<00:23, 41.54it/s]\n",
      " 51%|█████     | 1013/2000 [00:24<00:23, 41.45it/s]\n",
      " 51%|█████     | 1018/2000 [00:24<00:23, 41.48it/s]\n",
      " 51%|█████     | 1023/2000 [00:24<00:23, 41.50it/s]\n",
      " 51%|█████▏    | 1028/2000 [00:24<00:23, 41.51it/s]\n",
      " 52%|█████▏    | 1033/2000 [00:24<00:23, 41.53it/s]\n",
      " 52%|█████▏    | 1038/2000 [00:25<00:23, 41.54it/s]\n",
      " 52%|█████▏    | 1043/2000 [00:25<00:23, 41.55it/s]\n",
      " 52%|█████▏    | 1048/2000 [00:25<00:22, 41.55it/s]\n",
      " 53%|█████▎    | 1053/2000 [00:25<00:22, 41.55it/s]\n",
      " 53%|█████▎    | 1058/2000 [00:25<00:22, 41.55it/s]\n",
      " 53%|█████▎    | 1063/2000 [00:25<00:22, 41.56it/s]\n",
      " 53%|█████▎    | 1068/2000 [00:25<00:22, 41.56it/s]\n",
      " 54%|█████▎    | 1073/2000 [00:25<00:22, 41.52it/s]\n",
      " 54%|█████▍    | 1078/2000 [00:26<00:22, 41.47it/s]\n",
      " 54%|█████▍    | 1083/2000 [00:26<00:22, 41.45it/s]\n",
      " 54%|█████▍    | 1088/2000 [00:26<00:22, 41.42it/s]\n",
      " 55%|█████▍    | 1093/2000 [00:26<00:22, 41.11it/s]\n",
      " 55%|█████▍    | 1098/2000 [00:26<00:21, 41.18it/s]\n",
      " 55%|█████▌    | 1103/2000 [00:26<00:21, 41.25it/s]\n",
      " 55%|█████▌    | 1108/2000 [00:26<00:21, 41.29it/s]\n",
      " 56%|█████▌    | 1113/2000 [00:26<00:21, 41.32it/s]\n",
      " 56%|█████▌    | 1118/2000 [00:27<00:21, 41.34it/s]\n",
      " 56%|█████▌    | 1123/2000 [00:27<00:21, 41.35it/s]\n",
      " 56%|█████▋    | 1128/2000 [00:27<00:21, 41.37it/s]\n",
      " 57%|█████▋    | 1133/2000 [00:27<00:20, 41.37it/s]\n",
      " 57%|█████▋    | 1138/2000 [00:27<00:20, 41.38it/s]\n",
      " 57%|█████▋    | 1143/2000 [00:27<00:20, 41.37it/s]\n",
      " 57%|█████▋    | 1148/2000 [00:27<00:20, 40.82it/s]\n",
      " 58%|█████▊    | 1153/2000 [00:27<00:20, 41.00it/s]\n",
      " 58%|█████▊    | 1158/2000 [00:28<00:20, 41.13it/s]\n",
      " 58%|█████▊    | 1163/2000 [00:28<00:20, 41.22it/s]\n",
      " 58%|█████▊    | 1168/2000 [00:28<00:20, 41.22it/s]\n",
      " 59%|█████▊    | 1173/2000 [00:28<00:20, 41.24it/s]\n",
      " 59%|█████▉    | 1178/2000 [00:28<00:19, 41.28it/s]\n",
      " 59%|█████▉    | 1183/2000 [00:28<00:19, 41.32it/s]\n",
      " 59%|█████▉    | 1188/2000 [00:28<00:19, 41.34it/s]\n",
      " 60%|█████▉    | 1193/2000 [00:28<00:19, 41.37it/s]\n",
      " 60%|█████▉    | 1198/2000 [00:28<00:19, 41.37it/s]\n",
      " 60%|██████    | 1203/2000 [00:29<00:19, 41.38it/s]\n",
      " 60%|██████    | 1208/2000 [00:29<00:19, 41.37it/s]\n",
      " 61%|██████    | 1213/2000 [00:29<00:19, 41.37it/s]\n",
      " 61%|██████    | 1218/2000 [00:29<00:18, 41.39it/s]\n",
      " 61%|██████    | 1223/2000 [00:29<00:18, 41.38it/s]\n",
      " 61%|██████▏   | 1228/2000 [00:29<00:18, 41.38it/s]\n",
      " 62%|██████▏   | 1233/2000 [00:29<00:18, 41.39it/s]\n",
      " 62%|██████▏   | 1238/2000 [00:29<00:18, 41.39it/s]\n",
      " 62%|██████▏   | 1243/2000 [00:30<00:18, 41.38it/s]\n",
      " 62%|██████▏   | 1248/2000 [00:30<00:18, 41.39it/s]\n",
      " 63%|██████▎   | 1253/2000 [00:30<00:18, 41.39it/s]\n",
      " 63%|██████▎   | 1258/2000 [00:30<00:18, 40.85it/s]\n",
      " 63%|██████▎   | 1263/2000 [00:30<00:17, 40.95it/s]\n",
      " 63%|██████▎   | 1268/2000 [00:30<00:17, 41.10it/s]\n",
      " 64%|██████▎   | 1273/2000 [00:30<00:17, 41.21it/s]\n",
      " 64%|██████▍   | 1278/2000 [00:30<00:17, 41.30it/s]\n",
      " 64%|██████▍   | 1283/2000 [00:31<00:17, 41.37it/s]\n",
      " 64%|██████▍   | 1288/2000 [00:31<00:17, 41.42it/s]\n",
      " 65%|██████▍   | 1293/2000 [00:31<00:17, 41.45it/s]\n",
      " 65%|██████▍   | 1298/2000 [00:31<00:16, 41.47it/s]\n",
      " 65%|██████▌   | 1303/2000 [00:31<00:16, 41.46it/s]\n",
      " 65%|██████▌   | 1308/2000 [00:31<00:16, 41.47it/s]\n",
      " 66%|██████▌   | 1313/2000 [00:31<00:16, 41.45it/s]\n",
      " 66%|██████▌   | 1318/2000 [00:31<00:16, 41.41it/s]\n",
      " 66%|██████▌   | 1323/2000 [00:32<00:16, 40.80it/s]\n",
      " 66%|██████▋   | 1328/2000 [00:32<00:16, 41.01it/s]\n",
      " 67%|██████▋   | 1333/2000 [00:32<00:16, 41.15it/s]\n",
      " 67%|██████▋   | 1338/2000 [00:32<00:16, 41.26it/s]\n",
      " 67%|██████▋   | 1343/2000 [00:32<00:15, 41.34it/s]\n",
      " 67%|██████▋   | 1348/2000 [00:32<00:15, 41.40it/s]\n",
      " 68%|██████▊   | 1353/2000 [00:32<00:15, 41.44it/s]\n",
      " 68%|██████▊   | 1358/2000 [00:32<00:15, 41.42it/s]\n",
      " 68%|██████▊   | 1363/2000 [00:32<00:15, 41.44it/s]\n",
      " 68%|██████▊   | 1368/2000 [00:33<00:15, 41.46it/s]\n",
      " 69%|██████▊   | 1373/2000 [00:33<00:15, 41.49it/s]\n",
      " 69%|██████▉   | 1378/2000 [00:33<00:14, 41.49it/s]\n",
      " 69%|██████▉   | 1383/2000 [00:33<00:14, 41.45it/s]\n",
      " 69%|██████▉   | 1388/2000 [00:33<00:14, 41.36it/s]\n",
      " 70%|██████▉   | 1393/2000 [00:33<00:14, 41.28it/s]\n",
      " 70%|██████▉   | 1398/2000 [00:33<00:14, 41.23it/s]\n",
      " 70%|███████   | 1403/2000 [00:33<00:14, 41.19it/s]\n",
      " 70%|███████   | 1408/2000 [00:34<00:14, 41.16it/s]\n",
      " 71%|███████   | 1413/2000 [00:34<00:14, 41.15it/s]\n",
      " 71%|███████   | 1418/2000 [00:34<00:14, 41.14it/s]\n",
      " 71%|███████   | 1423/2000 [00:34<00:14, 41.12it/s]\n",
      " 71%|███████▏  | 1428/2000 [00:34<00:13, 41.11it/s]\n",
      " 72%|███████▏  | 1433/2000 [00:34<00:13, 41.13it/s]\n",
      " 72%|███████▏  | 1438/2000 [00:34<00:13, 41.14it/s]\n",
      " 72%|███████▏  | 1443/2000 [00:34<00:13, 41.14it/s]\n",
      " 72%|███████▏  | 1448/2000 [00:35<00:13, 41.16it/s]\n",
      " 73%|███████▎  | 1453/2000 [00:35<00:13, 41.07it/s]\n",
      " 73%|███████▎  | 1458/2000 [00:35<00:13, 41.10it/s]\n",
      " 73%|███████▎  | 1463/2000 [00:35<00:13, 41.12it/s]\n",
      " 73%|███████▎  | 1468/2000 [00:35<00:12, 41.12it/s]\n",
      " 74%|███████▎  | 1473/2000 [00:35<00:12, 41.09it/s]\n",
      " 74%|███████▍  | 1478/2000 [00:35<00:12, 41.09it/s]\n",
      " 74%|███████▍  | 1483/2000 [00:35<00:12, 41.11it/s]\n",
      " 74%|███████▍  | 1488/2000 [00:36<00:12, 41.10it/s]\n",
      " 75%|███████▍  | 1493/2000 [00:36<00:12, 41.09it/s]\n",
      " 75%|███████▍  | 1498/2000 [00:36<00:12, 41.10it/s]\n",
      " 75%|███████▌  | 1503/2000 [00:36<00:12, 41.10it/s]\n",
      " 75%|███████▌  | 1508/2000 [00:36<00:11, 41.08it/s]\n",
      " 76%|███████▌  | 1513/2000 [00:36<00:11, 41.09it/s]\n",
      " 76%|███████▌  | 1518/2000 [00:36<00:11, 41.10it/s]\n",
      " 76%|███████▌  | 1523/2000 [00:36<00:11, 41.11it/s]\n",
      " 76%|███████▋  | 1528/2000 [00:36<00:11, 41.11it/s]\n",
      " 77%|███████▋  | 1533/2000 [00:37<00:11, 41.11it/s]\n",
      " 77%|███████▋  | 1538/2000 [00:37<00:11, 41.11it/s]\n",
      " 77%|███████▋  | 1543/2000 [00:37<00:11, 41.11it/s]\n",
      " 77%|███████▋  | 1548/2000 [00:37<00:11, 41.08it/s]\n",
      " 78%|███████▊  | 1553/2000 [00:37<00:10, 41.09it/s]\n",
      " 78%|███████▊  | 1558/2000 [00:37<00:10, 41.09it/s]\n",
      " 78%|███████▊  | 1563/2000 [00:37<00:10, 41.08it/s]\n",
      " 78%|███████▊  | 1568/2000 [00:37<00:10, 41.08it/s]\n",
      " 79%|███████▊  | 1573/2000 [00:38<00:10, 41.09it/s]\n",
      " 79%|███████▉  | 1578/2000 [00:38<00:10, 41.09it/s]\n",
      " 79%|███████▉  | 1583/2000 [00:38<00:10, 41.09it/s]\n",
      " 79%|███████▉  | 1588/2000 [00:38<00:10, 41.09it/s]\n",
      " 80%|███████▉  | 1593/2000 [00:38<00:09, 41.13it/s]\n",
      " 80%|███████▉  | 1598/2000 [00:38<00:09, 41.15it/s]\n",
      " 80%|████████  | 1603/2000 [00:38<00:09, 41.17it/s]\n",
      " 80%|████████  | 1608/2000 [00:38<00:09, 41.18it/s]\n",
      " 81%|████████  | 1613/2000 [00:39<00:09, 41.20it/s]\n",
      " 81%|████████  | 1618/2000 [00:39<00:09, 41.21it/s]\n",
      " 81%|████████  | 1623/2000 [00:39<00:09, 41.22it/s]\n",
      " 81%|████████▏ | 1628/2000 [00:39<00:09, 41.21it/s]\n",
      " 82%|████████▏ | 1633/2000 [00:39<00:08, 41.20it/s]\n",
      " 82%|████████▏ | 1638/2000 [00:39<00:08, 41.21it/s]\n",
      " 82%|████████▏ | 1643/2000 [00:39<00:08, 41.18it/s]\n",
      " 82%|████████▏ | 1648/2000 [00:39<00:08, 41.18it/s]\n",
      " 83%|████████▎ | 1653/2000 [00:40<00:08, 41.18it/s]\n",
      " 83%|████████▎ | 1658/2000 [00:40<00:08, 41.19it/s]\n",
      " 83%|████████▎ | 1663/2000 [00:40<00:08, 41.13it/s]\n",
      " 83%|████████▎ | 1668/2000 [00:40<00:08, 41.24it/s]\n",
      " 84%|████████▎ | 1673/2000 [00:40<00:07, 41.29it/s]\n",
      " 84%|████████▍ | 1678/2000 [00:40<00:07, 41.34it/s]\n",
      " 84%|████████▍ | 1683/2000 [00:40<00:07, 41.37it/s]\n",
      " 84%|████████▍ | 1688/2000 [00:40<00:07, 41.33it/s]\n",
      " 85%|████████▍ | 1693/2000 [00:40<00:07, 41.29it/s]\n",
      " 85%|████████▍ | 1698/2000 [00:41<00:07, 41.26it/s]\n",
      " 85%|████████▌ | 1703/2000 [00:41<00:07, 41.25it/s]\n",
      " 85%|████████▌ | 1708/2000 [00:41<00:07, 41.23it/s]\n",
      " 86%|████████▌ | 1713/2000 [00:41<00:06, 41.22it/s]\n",
      " 86%|████████▌ | 1718/2000 [00:41<00:06, 41.22it/s]\n",
      " 86%|████████▌ | 1723/2000 [00:41<00:06, 41.23it/s]\n",
      " 86%|████████▋ | 1728/2000 [00:41<00:06, 41.22it/s]\n",
      " 87%|████████▋ | 1733/2000 [00:41<00:06, 41.22it/s]\n",
      " 87%|████████▋ | 1738/2000 [00:42<00:06, 41.07it/s]\n",
      " 87%|████████▋ | 1743/2000 [00:42<00:06, 41.09it/s]\n",
      " 87%|████████▋ | 1748/2000 [00:42<00:06, 41.11it/s]\n",
      " 88%|████████▊ | 1753/2000 [00:42<00:06, 41.07it/s]\n",
      " 88%|████████▊ | 1758/2000 [00:42<00:05, 41.04it/s]\n",
      " 88%|████████▊ | 1763/2000 [00:42<00:05, 41.08it/s]\n",
      " 88%|████████▊ | 1768/2000 [00:42<00:05, 41.11it/s]\n",
      " 89%|████████▊ | 1773/2000 [00:42<00:05, 41.14it/s]\n",
      " 89%|████████▉ | 1778/2000 [00:43<00:05, 41.17it/s]\n",
      " 89%|████████▉ | 1783/2000 [00:43<00:05, 41.16it/s]\n",
      " 89%|████████▉ | 1788/2000 [00:43<00:05, 41.10it/s]\n",
      " 90%|████████▉ | 1793/2000 [00:43<00:05, 41.08it/s]\n",
      " 90%|████████▉ | 1798/2000 [00:43<00:04, 41.06it/s]\n",
      " 90%|█████████ | 1803/2000 [00:43<00:04, 41.05it/s]\n",
      " 90%|█████████ | 1808/2000 [00:43<00:04, 41.03it/s]\n",
      " 91%|█████████ | 1813/2000 [00:43<00:04, 41.02it/s]\n",
      " 91%|█████████ | 1818/2000 [00:44<00:04, 41.01it/s]\n",
      " 91%|█████████ | 1823/2000 [00:44<00:04, 41.00it/s]\n",
      " 91%|█████████▏| 1828/2000 [00:44<00:04, 40.98it/s]\n",
      " 92%|█████████▏| 1833/2000 [00:44<00:04, 40.96it/s]\n",
      " 92%|█████████▏| 1838/2000 [00:44<00:03, 41.02it/s]\n",
      " 92%|█████████▏| 1843/2000 [00:44<00:03, 41.07it/s]\n",
      " 92%|█████████▏| 1848/2000 [00:44<00:03, 41.10it/s]\n",
      " 93%|█████████▎| 1853/2000 [00:44<00:03, 41.09it/s]\n",
      " 93%|█████████▎| 1858/2000 [00:45<00:03, 41.05it/s]\n",
      " 93%|█████████▎| 1863/2000 [00:45<00:03, 41.03it/s]\n",
      " 93%|█████████▎| 1868/2000 [00:45<00:03, 41.02it/s]\n",
      " 94%|█████████▎| 1873/2000 [00:45<00:03, 41.01it/s]\n",
      " 94%|█████████▍| 1878/2000 [00:45<00:02, 41.04it/s]\n",
      " 94%|█████████▍| 1883/2000 [00:45<00:02, 41.09it/s]\n",
      " 94%|█████████▍| 1888/2000 [00:45<00:02, 41.12it/s]\n",
      " 95%|█████████▍| 1893/2000 [00:45<00:02, 41.13it/s]\n",
      " 95%|█████████▍| 1898/2000 [00:45<00:02, 41.14it/s]\n",
      " 95%|█████████▌| 1903/2000 [00:46<00:02, 41.15it/s]\n",
      " 95%|█████████▌| 1908/2000 [00:46<00:02, 41.17it/s]\n",
      " 96%|█████████▌| 1913/2000 [00:46<00:02, 41.17it/s]\n",
      " 96%|█████████▌| 1918/2000 [00:46<00:01, 41.18it/s]\n",
      " 96%|█████████▌| 1923/2000 [00:46<00:01, 41.19it/s]\n",
      " 96%|█████████▋| 1928/2000 [00:46<00:01, 41.11it/s]\n",
      " 97%|█████████▋| 1933/2000 [00:46<00:01, 41.13it/s]\n",
      " 97%|█████████▋| 1938/2000 [00:46<00:01, 41.15it/s]\n",
      " 97%|█████████▋| 1943/2000 [00:47<00:01, 41.16it/s]\n",
      " 97%|█████████▋| 1948/2000 [00:47<00:01, 41.17it/s]\n",
      " 98%|█████████▊| 1953/2000 [00:47<00:01, 40.73it/s]\n",
      " 98%|█████████▊| 1958/2000 [00:47<00:01, 40.81it/s]\n",
      " 98%|█████████▊| 1963/2000 [00:47<00:00, 41.04it/s]\n",
      " 98%|█████████▊| 1968/2000 [00:47<00:00, 41.19it/s]\n",
      " 99%|█████████▊| 1973/2000 [00:47<00:00, 41.30it/s]\n",
      " 99%|█████████▉| 1978/2000 [00:47<00:00, 41.36it/s]\n",
      " 99%|█████████▉| 1983/2000 [00:48<00:00, 41.40it/s]\n",
      " 99%|█████████▉| 1988/2000 [00:48<00:00, 41.45it/s]\n",
      "100%|█████████▉| 1993/2000 [00:48<00:00, 41.47it/s]\n",
      "100%|█████████▉| 1998/2000 [00:48<00:00, 41.50it/s]\n",
      "100%|██████████| 2000/2000 [00:48<00:00, 41.28it/s]\n",
      "\n",
      "  0%|          | 0/2000 [00:00<?, ?it/s]\n",
      "  0%|          | 3/2000 [00:00<01:08, 28.94it/s]\n",
      "  0%|          | 6/2000 [00:00<01:08, 29.25it/s]\n",
      "  0%|          | 10/2000 [00:00<01:06, 30.01it/s]\n",
      "  1%|          | 14/2000 [00:00<01:04, 30.64it/s]\n",
      "  1%|          | 18/2000 [00:00<01:03, 31.08it/s]\n",
      "  1%|          | 22/2000 [00:00<01:02, 31.40it/s]\n",
      "  1%|▏         | 26/2000 [00:00<01:02, 31.62it/s]\n",
      "  2%|▏         | 30/2000 [00:00<01:02, 31.72it/s]\n",
      "  2%|▏         | 34/2000 [00:01<01:01, 31.79it/s]\n",
      "  2%|▏         | 38/2000 [00:01<01:01, 31.85it/s]\n",
      "  2%|▏         | 42/2000 [00:01<01:01, 31.90it/s]\n",
      "  2%|▏         | 46/2000 [00:01<01:01, 31.90it/s]\n",
      "  2%|▎         | 50/2000 [00:01<01:01, 31.92it/s]\n",
      "  3%|▎         | 54/2000 [00:01<01:00, 31.93it/s]\n",
      "  3%|▎         | 58/2000 [00:01<01:00, 31.93it/s]\n",
      "  3%|▎         | 62/2000 [00:01<01:00, 31.93it/s]\n",
      "  3%|▎         | 66/2000 [00:02<01:00, 31.94it/s]\n",
      "  4%|▎         | 70/2000 [00:02<01:00, 31.94it/s]\n",
      "  4%|▎         | 74/2000 [00:02<01:00, 31.83it/s]\n",
      "  4%|▍         | 78/2000 [00:02<01:00, 31.83it/s]\n",
      "  4%|▍         | 82/2000 [00:02<01:00, 31.86it/s]\n",
      "  4%|▍         | 86/2000 [00:02<01:00, 31.88it/s]\n",
      "  4%|▍         | 90/2000 [00:02<00:59, 31.89it/s]\n",
      "  5%|▍         | 94/2000 [00:02<00:59, 31.89it/s]\n",
      "  5%|▍         | 98/2000 [00:03<00:59, 31.90it/s]\n",
      "  5%|▌         | 102/2000 [00:03<00:59, 31.90it/s]\n",
      "  5%|▌         | 106/2000 [00:03<00:59, 31.91it/s]\n",
      "  6%|▌         | 110/2000 [00:03<00:59, 31.90it/s]\n",
      "  6%|▌         | 114/2000 [00:03<00:59, 31.88it/s]\n",
      "  6%|▌         | 118/2000 [00:03<00:59, 31.89it/s]\n",
      "  6%|▌         | 122/2000 [00:03<00:58, 31.90it/s]\n",
      "  6%|▋         | 126/2000 [00:03<01:00, 31.18it/s]\n",
      "  6%|▋         | 130/2000 [00:04<00:59, 31.34it/s]\n",
      "  7%|▋         | 134/2000 [00:04<00:59, 31.48it/s]\n",
      "  7%|▋         | 138/2000 [00:04<00:59, 31.56it/s]\n",
      "  7%|▋         | 142/2000 [00:04<00:58, 31.62it/s]\n",
      "  7%|▋         | 146/2000 [00:04<00:58, 31.67it/s]\n",
      "  8%|▊         | 150/2000 [00:04<00:58, 31.49it/s]\n",
      "  8%|▊         | 154/2000 [00:04<00:58, 31.39it/s]\n",
      "  8%|▊         | 158/2000 [00:04<00:59, 31.15it/s]\n",
      "  8%|▊         | 162/2000 [00:05<00:58, 31.43it/s]\n",
      "  8%|▊         | 166/2000 [00:05<00:57, 31.64it/s]\n",
      "  8%|▊         | 170/2000 [00:05<00:57, 31.78it/s]\n",
      "  9%|▊         | 174/2000 [00:05<00:57, 31.90it/s]\n",
      "  9%|▉         | 178/2000 [00:05<00:56, 32.13it/s]\n",
      "  9%|▉         | 182/2000 [00:05<00:56, 32.31it/s]\n",
      "  9%|▉         | 186/2000 [00:05<00:55, 32.45it/s]\n",
      " 10%|▉         | 190/2000 [00:05<00:55, 32.54it/s]\n",
      " 10%|▉         | 194/2000 [00:06<00:55, 32.62it/s]\n",
      " 10%|▉         | 198/2000 [00:06<00:55, 32.67it/s]\n",
      " 10%|█         | 202/2000 [00:06<00:54, 32.71it/s]\n",
      " 10%|█         | 206/2000 [00:06<00:54, 32.71it/s]\n",
      " 10%|█         | 210/2000 [00:06<00:54, 32.68it/s]\n",
      " 11%|█         | 214/2000 [00:06<00:54, 32.67it/s]\n",
      " 11%|█         | 218/2000 [00:06<00:56, 31.70it/s]\n",
      " 11%|█         | 222/2000 [00:06<00:55, 31.98it/s]\n",
      " 11%|█▏        | 226/2000 [00:07<00:55, 32.17it/s]\n",
      " 12%|█▏        | 230/2000 [00:07<00:54, 32.31it/s]\n",
      " 12%|█▏        | 234/2000 [00:07<00:54, 32.40it/s]\n",
      " 12%|█▏        | 238/2000 [00:07<00:54, 32.36it/s]\n",
      " 12%|█▏        | 242/2000 [00:07<00:54, 32.44it/s]\n",
      " 12%|█▏        | 246/2000 [00:07<00:53, 32.50it/s]\n",
      " 12%|█▎        | 250/2000 [00:07<00:53, 32.53it/s]\n",
      " 13%|█▎        | 254/2000 [00:07<00:53, 32.56it/s]\n",
      " 13%|█▎        | 258/2000 [00:08<00:53, 32.57it/s]\n",
      " 13%|█▎        | 262/2000 [00:08<00:53, 32.59it/s]\n",
      " 13%|█▎        | 266/2000 [00:08<00:53, 32.59it/s]\n",
      " 14%|█▎        | 270/2000 [00:08<00:53, 32.61it/s]\n",
      " 14%|█▎        | 274/2000 [00:08<00:52, 32.61it/s]\n",
      " 14%|█▍        | 278/2000 [00:08<00:52, 32.61it/s]\n",
      " 14%|█▍        | 282/2000 [00:08<00:52, 32.62it/s]\n",
      " 14%|█▍        | 286/2000 [00:08<00:52, 32.65it/s]\n",
      " 14%|█▍        | 290/2000 [00:09<00:53, 32.18it/s]\n",
      " 15%|█▍        | 294/2000 [00:09<00:52, 32.28it/s]\n",
      " 15%|█▍        | 298/2000 [00:09<00:52, 32.35it/s]\n",
      " 15%|█▌        | 302/2000 [00:09<00:52, 32.41it/s]\n",
      " 15%|█▌        | 306/2000 [00:09<00:52, 32.44it/s]\n",
      " 16%|█▌        | 310/2000 [00:09<00:52, 32.46it/s]\n",
      " 16%|█▌        | 314/2000 [00:09<00:51, 32.48it/s]\n",
      " 16%|█▌        | 318/2000 [00:09<00:51, 32.49it/s]\n",
      " 16%|█▌        | 322/2000 [00:10<00:51, 32.47it/s]\n",
      " 16%|█▋        | 326/2000 [00:10<00:52, 32.01it/s]\n",
      " 16%|█▋        | 330/2000 [00:10<00:51, 32.24it/s]\n",
      " 17%|█▋        | 334/2000 [00:10<00:51, 32.40it/s]\n",
      " 17%|█▋        | 338/2000 [00:10<00:52, 31.95it/s]\n",
      " 17%|█▋        | 342/2000 [00:10<00:51, 32.16it/s]\n",
      " 17%|█▋        | 346/2000 [00:10<00:51, 32.30it/s]\n",
      " 18%|█▊        | 350/2000 [00:10<00:51, 32.32it/s]\n",
      " 18%|█▊        | 354/2000 [00:11<00:50, 32.33it/s]\n",
      " 18%|█▊        | 358/2000 [00:11<00:50, 32.32it/s]\n",
      " 18%|█▊        | 362/2000 [00:11<00:50, 32.32it/s]\n",
      " 18%|█▊        | 366/2000 [00:11<00:50, 32.32it/s]\n",
      " 18%|█▊        | 370/2000 [00:11<00:50, 32.32it/s]\n",
      " 19%|█▊        | 374/2000 [00:11<00:50, 32.33it/s]\n",
      " 19%|█▉        | 378/2000 [00:11<00:50, 32.33it/s]\n",
      " 19%|█▉        | 382/2000 [00:11<00:50, 32.32it/s]\n",
      " 19%|█▉        | 386/2000 [00:12<00:49, 32.32it/s]\n",
      " 20%|█▉        | 390/2000 [00:12<00:49, 32.31it/s]\n",
      " 20%|█▉        | 394/2000 [00:12<00:49, 32.31it/s]\n",
      " 20%|█▉        | 398/2000 [00:12<00:49, 32.32it/s]\n",
      " 20%|██        | 402/2000 [00:12<00:49, 32.33it/s]\n",
      " 20%|██        | 406/2000 [00:12<00:49, 32.32it/s]\n",
      " 20%|██        | 410/2000 [00:12<00:49, 32.33it/s]\n",
      " 21%|██        | 414/2000 [00:12<00:49, 32.28it/s]\n",
      " 21%|██        | 418/2000 [00:13<00:49, 32.29it/s]\n",
      " 21%|██        | 422/2000 [00:13<00:48, 32.30it/s]\n",
      " 21%|██▏       | 426/2000 [00:13<00:48, 32.31it/s]\n",
      " 22%|██▏       | 430/2000 [00:13<00:48, 32.31it/s]\n",
      " 22%|██▏       | 434/2000 [00:13<00:48, 32.31it/s]\n",
      " 22%|██▏       | 438/2000 [00:13<00:48, 32.32it/s]\n",
      " 22%|██▏       | 442/2000 [00:13<00:48, 32.32it/s]\n",
      " 22%|██▏       | 446/2000 [00:13<00:49, 31.58it/s]\n",
      " 22%|██▎       | 450/2000 [00:14<00:48, 31.87it/s]\n",
      " 23%|██▎       | 454/2000 [00:14<00:48, 32.07it/s]\n",
      " 23%|██▎       | 458/2000 [00:14<00:47, 32.22it/s]\n",
      " 23%|██▎       | 462/2000 [00:14<00:47, 32.33it/s]\n",
      " 23%|██▎       | 466/2000 [00:14<00:47, 32.34it/s]\n",
      " 24%|██▎       | 470/2000 [00:14<00:47, 32.42it/s]\n",
      " 24%|██▎       | 474/2000 [00:14<00:47, 32.42it/s]\n",
      " 24%|██▍       | 478/2000 [00:14<00:46, 32.45it/s]\n",
      " 24%|██▍       | 482/2000 [00:14<00:46, 32.48it/s]\n",
      " 24%|██▍       | 486/2000 [00:15<00:46, 32.50it/s]\n",
      " 24%|██▍       | 490/2000 [00:15<00:46, 32.51it/s]\n",
      " 25%|██▍       | 494/2000 [00:15<00:46, 32.05it/s]\n",
      " 25%|██▍       | 498/2000 [00:15<00:46, 32.18it/s]\n",
      " 25%|██▌       | 502/2000 [00:15<00:46, 32.27it/s]\n",
      " 25%|██▌       | 506/2000 [00:15<00:46, 31.95it/s]\n",
      " 26%|██▌       | 510/2000 [00:15<00:46, 32.16it/s]\n",
      " 26%|██▌       | 514/2000 [00:15<00:46, 32.29it/s]\n",
      " 26%|██▌       | 518/2000 [00:16<00:45, 32.39it/s]\n",
      " 26%|██▌       | 522/2000 [00:16<00:45, 32.50it/s]\n",
      " 26%|██▋       | 526/2000 [00:16<00:45, 32.59it/s]\n",
      " 26%|██▋       | 530/2000 [00:16<00:45, 32.64it/s]\n",
      " 27%|██▋       | 534/2000 [00:16<00:44, 32.68it/s]\n",
      " 27%|██▋       | 538/2000 [00:16<00:44, 32.67it/s]\n",
      " 27%|██▋       | 542/2000 [00:16<00:44, 32.71it/s]\n",
      " 27%|██▋       | 546/2000 [00:16<00:44, 32.74it/s]\n",
      " 28%|██▊       | 550/2000 [00:17<00:44, 32.76it/s]\n",
      " 28%|██▊       | 554/2000 [00:17<00:44, 32.74it/s]\n",
      " 28%|██▊       | 558/2000 [00:17<00:44, 32.68it/s]\n",
      " 28%|██▊       | 562/2000 [00:17<00:44, 32.64it/s]\n",
      " 28%|██▊       | 566/2000 [00:17<00:43, 32.60it/s]\n",
      " 28%|██▊       | 570/2000 [00:17<00:44, 32.09it/s]\n",
      " 29%|██▊       | 574/2000 [00:17<00:44, 32.19it/s]\n",
      " 29%|██▉       | 578/2000 [00:17<00:44, 32.26it/s]\n",
      " 29%|██▉       | 582/2000 [00:18<00:43, 32.30it/s]\n",
      " 29%|██▉       | 586/2000 [00:18<00:43, 32.33it/s]\n",
      " 30%|██▉       | 590/2000 [00:18<00:43, 32.39it/s]\n",
      " 30%|██▉       | 594/2000 [00:18<00:43, 32.35it/s]\n",
      " 30%|██▉       | 598/2000 [00:18<00:43, 32.35it/s]\n",
      " 30%|███       | 602/2000 [00:18<00:43, 32.35it/s]\n",
      " 30%|███       | 606/2000 [00:18<00:43, 32.35it/s]\n",
      " 30%|███       | 610/2000 [00:18<00:42, 32.35it/s]\n",
      " 31%|███       | 614/2000 [00:19<00:42, 32.35it/s]\n",
      " 31%|███       | 618/2000 [00:19<00:42, 32.35it/s]\n",
      " 31%|███       | 622/2000 [00:19<00:42, 32.35it/s]\n",
      " 31%|███▏      | 626/2000 [00:19<00:42, 32.34it/s]\n",
      " 32%|███▏      | 630/2000 [00:19<00:42, 32.33it/s]\n",
      " 32%|███▏      | 634/2000 [00:19<00:42, 32.33it/s]\n",
      " 32%|███▏      | 638/2000 [00:19<00:42, 31.86it/s]\n",
      " 32%|███▏      | 642/2000 [00:19<00:42, 32.03it/s]\n",
      " 32%|███▏      | 646/2000 [00:20<00:41, 32.25it/s]\n",
      " 32%|███▎      | 650/2000 [00:20<00:41, 32.41it/s]\n",
      " 33%|███▎      | 654/2000 [00:20<00:41, 32.44it/s]\n",
      " 33%|███▎      | 658/2000 [00:20<00:42, 31.93it/s]\n",
      " 33%|███▎      | 662/2000 [00:20<00:42, 31.71it/s]\n",
      " 33%|███▎      | 666/2000 [00:20<00:41, 31.97it/s]\n",
      " 34%|███▎      | 670/2000 [00:20<00:41, 32.14it/s]\n",
      " 34%|███▎      | 674/2000 [00:20<00:41, 32.29it/s]\n",
      " 34%|███▍      | 678/2000 [00:21<00:40, 32.31it/s]\n",
      " 34%|███▍      | 682/2000 [00:21<00:40, 32.29it/s]\n",
      " 34%|███▍      | 686/2000 [00:21<00:40, 32.35it/s]\n",
      " 34%|███▍      | 690/2000 [00:21<00:40, 32.40it/s]\n",
      " 35%|███▍      | 694/2000 [00:21<00:40, 32.05it/s]\n",
      " 35%|███▍      | 698/2000 [00:21<00:40, 32.22it/s]\n",
      " 35%|███▌      | 702/2000 [00:21<00:40, 31.96it/s]\n",
      " 35%|███▌      | 706/2000 [00:21<00:40, 31.77it/s]\n",
      " 36%|███▌      | 710/2000 [00:22<00:40, 31.96it/s]\n",
      " 36%|███▌      | 714/2000 [00:22<00:40, 32.10it/s]\n",
      " 36%|███▌      | 718/2000 [00:22<00:39, 32.20it/s]\n",
      " 36%|███▌      | 722/2000 [00:22<00:39, 32.27it/s]\n",
      " 36%|███▋      | 726/2000 [00:22<00:39, 32.32it/s]\n",
      " 36%|███▋      | 730/2000 [00:22<00:39, 32.36it/s]\n",
      " 37%|███▋      | 734/2000 [00:22<00:39, 32.38it/s]\n",
      " 37%|███▋      | 738/2000 [00:22<00:38, 32.42it/s]\n",
      " 37%|███▋      | 742/2000 [00:23<00:38, 32.42it/s]\n",
      " 37%|███▋      | 746/2000 [00:23<00:38, 32.39it/s]\n",
      " 38%|███▊      | 750/2000 [00:23<00:38, 32.38it/s]\n",
      " 38%|███▊      | 754/2000 [00:23<00:38, 32.36it/s]\n",
      " 38%|███▊      | 758/2000 [00:23<00:38, 32.36it/s]\n",
      " 38%|███▊      | 762/2000 [00:23<00:38, 32.35it/s]\n",
      " 38%|███▊      | 766/2000 [00:23<00:38, 32.35it/s]\n",
      " 38%|███▊      | 770/2000 [00:23<00:38, 31.58it/s]\n",
      " 39%|███▊      | 774/2000 [00:24<00:38, 31.86it/s]\n",
      " 39%|███▉      | 778/2000 [00:24<00:38, 31.60it/s]\n",
      " 39%|███▉      | 782/2000 [00:24<00:38, 31.83it/s]\n",
      " 39%|███▉      | 786/2000 [00:24<00:37, 31.95it/s]\n",
      " 40%|███▉      | 790/2000 [00:24<00:37, 32.03it/s]\n",
      " 40%|███▉      | 794/2000 [00:24<00:37, 32.09it/s]\n",
      " 40%|███▉      | 798/2000 [00:24<00:37, 32.14it/s]\n",
      " 40%|████      | 802/2000 [00:24<00:37, 32.17it/s]\n",
      " 40%|████      | 806/2000 [00:25<00:37, 32.20it/s]\n",
      " 40%|████      | 810/2000 [00:25<00:36, 32.22it/s]\n",
      " 41%|████      | 814/2000 [00:25<00:36, 32.22it/s]\n",
      " 41%|████      | 818/2000 [00:25<00:36, 32.23it/s]\n",
      " 41%|████      | 822/2000 [00:25<00:36, 32.23it/s]\n",
      " 41%|████▏     | 826/2000 [00:25<00:36, 31.79it/s]\n",
      " 42%|████▏     | 830/2000 [00:25<00:36, 32.02it/s]\n",
      " 42%|████▏     | 834/2000 [00:25<00:36, 32.23it/s]\n",
      " 42%|████▏     | 838/2000 [00:26<00:35, 32.37it/s]\n",
      " 42%|████▏     | 842/2000 [00:26<00:35, 32.46it/s]\n",
      " 42%|████▏     | 846/2000 [00:26<00:35, 32.53it/s]\n",
      " 42%|████▎     | 850/2000 [00:26<00:35, 32.56it/s]\n",
      " 43%|████▎     | 854/2000 [00:26<00:35, 32.61it/s]\n",
      " 43%|████▎     | 858/2000 [00:26<00:35, 32.60it/s]\n",
      " 43%|████▎     | 862/2000 [00:26<00:34, 32.64it/s]\n",
      " 43%|████▎     | 866/2000 [00:26<00:34, 32.70it/s]\n",
      " 44%|████▎     | 870/2000 [00:27<00:34, 32.73it/s]\n",
      " 44%|████▎     | 874/2000 [00:27<00:34, 32.71it/s]\n",
      " 44%|████▍     | 878/2000 [00:27<00:34, 32.69it/s]\n",
      " 44%|████▍     | 882/2000 [00:27<00:34, 32.69it/s]\n",
      " 44%|████▍     | 886/2000 [00:27<00:34, 32.68it/s]\n",
      " 44%|████▍     | 890/2000 [00:27<00:33, 32.67it/s]\n",
      " 45%|████▍     | 894/2000 [00:27<00:33, 32.66it/s]\n",
      " 45%|████▍     | 898/2000 [00:27<00:33, 32.66it/s]\n",
      " 45%|████▌     | 902/2000 [00:27<00:33, 32.66it/s]\n",
      " 45%|████▌     | 906/2000 [00:28<00:33, 32.65it/s]\n",
      " 46%|████▌     | 910/2000 [00:28<00:33, 32.65it/s]\n",
      " 46%|████▌     | 914/2000 [00:28<00:33, 32.64it/s]\n",
      " 46%|████▌     | 918/2000 [00:28<00:33, 32.63it/s]\n",
      " 46%|████▌     | 922/2000 [00:28<00:33, 32.63it/s]\n",
      " 46%|████▋     | 926/2000 [00:28<00:32, 32.63it/s]\n",
      " 46%|████▋     | 930/2000 [00:28<00:32, 32.63it/s]\n",
      " 47%|████▋     | 934/2000 [00:28<00:32, 32.62it/s]\n",
      " 47%|████▋     | 938/2000 [00:29<00:32, 32.61it/s]\n",
      " 47%|████▋     | 942/2000 [00:29<00:32, 32.57it/s]\n",
      " 47%|████▋     | 946/2000 [00:29<00:32, 32.55it/s]\n",
      " 48%|████▊     | 950/2000 [00:29<00:32, 31.89it/s]\n",
      " 48%|████▊     | 954/2000 [00:29<00:32, 32.05it/s]\n",
      " 48%|████▊     | 958/2000 [00:29<00:32, 32.15it/s]\n",
      " 48%|████▊     | 962/2000 [00:29<00:32, 32.23it/s]\n",
      " 48%|████▊     | 966/2000 [00:29<00:32, 32.28it/s]\n",
      " 48%|████▊     | 970/2000 [00:30<00:31, 32.35it/s]\n",
      " 49%|████▊     | 974/2000 [00:30<00:31, 32.42it/s]\n",
      " 49%|████▉     | 978/2000 [00:30<00:31, 32.47it/s]\n",
      " 49%|████▉     | 982/2000 [00:30<00:31, 32.50it/s]\n",
      " 49%|████▉     | 986/2000 [00:30<00:31, 32.53it/s]\n",
      " 50%|████▉     | 990/2000 [00:30<00:31, 32.55it/s]\n",
      " 50%|████▉     | 994/2000 [00:30<00:30, 32.56it/s]\n",
      " 50%|████▉     | 998/2000 [00:30<00:30, 32.48it/s]\n",
      " 50%|█████     | 1002/2000 [00:31<00:30, 32.42it/s]\n",
      " 50%|█████     | 1006/2000 [00:31<00:30, 32.38it/s]\n",
      " 50%|█████     | 1010/2000 [00:31<00:30, 32.36it/s]\n",
      " 51%|█████     | 1014/2000 [00:31<00:30, 31.87it/s]\n",
      " 51%|█████     | 1018/2000 [00:31<00:30, 32.08it/s]\n",
      " 51%|█████     | 1022/2000 [00:31<00:30, 32.23it/s]\n",
      " 51%|█████▏    | 1026/2000 [00:31<00:30, 32.34it/s]\n",
      " 52%|█████▏    | 1030/2000 [00:31<00:29, 32.41it/s]\n",
      " 52%|█████▏    | 1034/2000 [00:32<00:29, 32.45it/s]\n",
      " 52%|█████▏    | 1038/2000 [00:32<00:29, 32.27it/s]\n",
      " 52%|█████▏    | 1042/2000 [00:32<00:29, 32.34it/s]\n",
      " 52%|█████▏    | 1046/2000 [00:32<00:29, 32.41it/s]\n",
      " 52%|█████▎    | 1050/2000 [00:32<00:29, 32.46it/s]\n",
      " 53%|█████▎    | 1054/2000 [00:32<00:29, 32.49it/s]\n",
      " 53%|█████▎    | 1058/2000 [00:32<00:28, 32.51it/s]\n",
      " 53%|█████▎    | 1062/2000 [00:32<00:28, 32.53it/s]\n",
      " 53%|█████▎    | 1066/2000 [00:33<00:28, 32.54it/s]\n",
      " 54%|█████▎    | 1070/2000 [00:33<00:28, 32.56it/s]\n",
      " 54%|█████▎    | 1074/2000 [00:33<00:28, 32.56it/s]\n",
      " 54%|█████▍    | 1078/2000 [00:33<00:28, 32.56it/s]\n",
      " 54%|█████▍    | 1082/2000 [00:33<00:28, 32.55it/s]\n",
      " 54%|█████▍    | 1086/2000 [00:33<00:28, 32.56it/s]\n",
      " 55%|█████▍    | 1090/2000 [00:33<00:27, 32.57it/s]\n",
      " 55%|█████▍    | 1094/2000 [00:33<00:27, 32.56it/s]\n",
      " 55%|█████▍    | 1098/2000 [00:34<00:27, 32.56it/s]\n",
      " 55%|█████▌    | 1102/2000 [00:34<00:27, 32.56it/s]\n",
      " 55%|█████▌    | 1106/2000 [00:34<00:27, 32.55it/s]\n",
      " 56%|█████▌    | 1110/2000 [00:34<00:27, 32.55it/s]\n",
      " 56%|█████▌    | 1114/2000 [00:34<00:27, 32.56it/s]\n",
      " 56%|█████▌    | 1118/2000 [00:34<00:27, 32.56it/s]\n",
      " 56%|█████▌    | 1122/2000 [00:34<00:26, 32.56it/s]\n",
      " 56%|█████▋    | 1126/2000 [00:34<00:27, 31.92it/s]\n",
      " 56%|█████▋    | 1130/2000 [00:35<00:27, 32.04it/s]\n",
      " 57%|█████▋    | 1134/2000 [00:35<00:26, 32.14it/s]\n",
      " 57%|█████▋    | 1138/2000 [00:35<00:26, 32.20it/s]\n",
      " 57%|█████▋    | 1142/2000 [00:35<00:26, 32.25it/s]\n",
      " 57%|█████▋    | 1146/2000 [00:35<00:26, 32.29it/s]\n",
      " 57%|█████▊    | 1150/2000 [00:35<00:26, 32.32it/s]\n",
      " 58%|█████▊    | 1154/2000 [00:35<00:26, 32.33it/s]\n",
      " 58%|█████▊    | 1158/2000 [00:35<00:26, 32.34it/s]\n",
      " 58%|█████▊    | 1162/2000 [00:36<00:25, 32.35it/s]\n",
      " 58%|█████▊    | 1166/2000 [00:36<00:26, 31.72it/s]\n",
      " 58%|█████▊    | 1170/2000 [00:36<00:26, 31.70it/s]\n",
      " 59%|█████▊    | 1174/2000 [00:36<00:26, 31.59it/s]\n",
      " 59%|█████▉    | 1178/2000 [00:36<00:25, 31.86it/s]\n",
      " 59%|█████▉    | 1182/2000 [00:36<00:25, 32.06it/s]\n",
      " 59%|█████▉    | 1186/2000 [00:36<00:25, 32.21it/s]\n",
      " 60%|█████▉    | 1190/2000 [00:36<00:25, 32.31it/s]\n",
      " 60%|█████▉    | 1194/2000 [00:37<00:24, 32.38it/s]\n",
      " 60%|█████▉    | 1198/2000 [00:37<00:24, 32.43it/s]\n",
      " 60%|██████    | 1202/2000 [00:37<00:24, 32.46it/s]\n",
      " 60%|██████    | 1206/2000 [00:37<00:24, 32.49it/s]\n",
      " 60%|██████    | 1210/2000 [00:37<00:24, 32.51it/s]\n",
      " 61%|██████    | 1214/2000 [00:37<00:24, 32.53it/s]\n",
      " 61%|██████    | 1218/2000 [00:37<00:24, 32.45it/s]\n",
      " 61%|██████    | 1222/2000 [00:37<00:23, 32.48it/s]\n",
      " 61%|██████▏   | 1226/2000 [00:37<00:23, 32.49it/s]\n",
      " 62%|██████▏   | 1230/2000 [00:38<00:23, 32.50it/s]\n",
      " 62%|██████▏   | 1234/2000 [00:38<00:23, 32.51it/s]\n",
      " 62%|██████▏   | 1238/2000 [00:38<00:23, 32.52it/s]\n",
      " 62%|██████▏   | 1242/2000 [00:38<00:23, 32.52it/s]\n",
      " 62%|██████▏   | 1246/2000 [00:38<00:23, 32.53it/s]\n",
      " 62%|██████▎   | 1250/2000 [00:38<00:23, 32.53it/s]\n",
      " 63%|██████▎   | 1254/2000 [00:38<00:22, 32.53it/s]\n",
      " 63%|██████▎   | 1258/2000 [00:38<00:23, 32.08it/s]\n",
      " 63%|██████▎   | 1262/2000 [00:39<00:22, 32.21it/s]\n",
      " 63%|██████▎   | 1266/2000 [00:39<00:22, 32.32it/s]\n",
      " 64%|██████▎   | 1270/2000 [00:39<00:22, 31.91it/s]\n",
      " 64%|██████▎   | 1274/2000 [00:39<00:22, 32.14it/s]\n",
      " 64%|██████▍   | 1278/2000 [00:39<00:22, 32.31it/s]\n",
      " 64%|██████▍   | 1282/2000 [00:39<00:22, 32.42it/s]\n",
      " 64%|██████▍   | 1286/2000 [00:39<00:21, 32.46it/s]\n",
      " 64%|██████▍   | 1290/2000 [00:39<00:21, 32.49it/s]\n",
      " 65%|██████▍   | 1294/2000 [00:40<00:21, 32.49it/s]\n",
      " 65%|██████▍   | 1298/2000 [00:40<00:21, 32.49it/s]\n",
      " 65%|██████▌   | 1302/2000 [00:40<00:21, 32.47it/s]\n",
      " 65%|██████▌   | 1306/2000 [00:40<00:21, 32.42it/s]\n",
      " 66%|██████▌   | 1310/2000 [00:40<00:21, 31.93it/s]\n",
      " 66%|██████▌   | 1314/2000 [00:40<00:21, 32.04it/s]\n",
      " 66%|██████▌   | 1318/2000 [00:40<00:21, 32.16it/s]\n",
      " 66%|██████▌   | 1322/2000 [00:40<00:21, 32.23it/s]\n",
      " 66%|██████▋   | 1326/2000 [00:41<00:20, 32.28it/s]\n",
      " 66%|██████▋   | 1330/2000 [00:41<00:20, 32.32it/s]\n",
      " 67%|██████▋   | 1334/2000 [00:41<00:20, 32.34it/s]\n",
      " 67%|██████▋   | 1338/2000 [00:41<00:20, 32.34it/s]\n",
      " 67%|██████▋   | 1342/2000 [00:41<00:20, 32.36it/s]\n",
      " 67%|██████▋   | 1346/2000 [00:41<00:20, 32.36it/s]\n",
      " 68%|██████▊   | 1350/2000 [00:41<00:20, 32.38it/s]\n",
      " 68%|██████▊   | 1354/2000 [00:41<00:19, 32.38it/s]\n",
      " 68%|██████▊   | 1358/2000 [00:42<00:19, 32.39it/s]\n",
      " 68%|██████▊   | 1362/2000 [00:42<00:19, 32.39it/s]\n",
      " 68%|██████▊   | 1366/2000 [00:42<00:19, 32.40it/s]\n",
      " 68%|██████▊   | 1370/2000 [00:42<00:19, 32.38it/s]\n",
      " 69%|██████▊   | 1374/2000 [00:42<00:19, 32.42it/s]\n",
      " 69%|██████▉   | 1378/2000 [00:42<00:19, 32.11it/s]\n",
      " 69%|██████▉   | 1382/2000 [00:42<00:19, 32.30it/s]\n",
      " 69%|██████▉   | 1386/2000 [00:42<00:18, 32.42it/s]\n",
      " 70%|██████▉   | 1390/2000 [00:43<00:18, 32.50it/s]\n",
      " 70%|██████▉   | 1394/2000 [00:43<00:18, 32.56it/s]\n",
      " 70%|██████▉   | 1398/2000 [00:43<00:18, 32.54it/s]\n",
      " 70%|███████   | 1402/2000 [00:43<00:18, 32.56it/s]\n",
      " 70%|███████   | 1406/2000 [00:43<00:18, 32.57it/s]\n",
      " 70%|███████   | 1410/2000 [00:43<00:18, 32.62it/s]\n",
      " 71%|███████   | 1414/2000 [00:43<00:17, 32.65it/s]\n",
      " 71%|███████   | 1418/2000 [00:43<00:17, 32.66it/s]\n",
      " 71%|███████   | 1422/2000 [00:44<00:17, 32.67it/s]\n",
      " 71%|███████▏  | 1426/2000 [00:44<00:17, 32.69it/s]\n",
      " 72%|███████▏  | 1430/2000 [00:44<00:17, 32.69it/s]\n",
      " 72%|███████▏  | 1434/2000 [00:44<00:17, 32.26it/s]\n",
      " 72%|███████▏  | 1438/2000 [00:44<00:17, 32.31it/s]\n",
      " 72%|███████▏  | 1442/2000 [00:44<00:17, 32.33it/s]\n",
      " 72%|███████▏  | 1446/2000 [00:44<00:17, 32.33it/s]\n",
      " 72%|███████▎  | 1450/2000 [00:44<00:17, 32.34it/s]\n",
      " 73%|███████▎  | 1454/2000 [00:45<00:17, 31.90it/s]\n",
      " 73%|███████▎  | 1458/2000 [00:45<00:16, 32.08it/s]\n",
      " 73%|███████▎  | 1462/2000 [00:45<00:16, 32.21it/s]\n",
      " 73%|███████▎  | 1466/2000 [00:45<00:16, 32.30it/s]\n",
      " 74%|███████▎  | 1470/2000 [00:45<00:16, 32.42it/s]\n",
      " 74%|███████▎  | 1474/2000 [00:45<00:16, 32.51it/s]\n",
      " 74%|███████▍  | 1478/2000 [00:45<00:16, 32.56it/s]\n",
      " 74%|███████▍  | 1482/2000 [00:45<00:15, 32.59it/s]\n",
      " 74%|███████▍  | 1486/2000 [00:46<00:15, 32.61it/s]\n",
      " 74%|███████▍  | 1490/2000 [00:46<00:15, 32.63it/s]\n",
      " 75%|███████▍  | 1494/2000 [00:46<00:15, 32.64it/s]\n",
      " 75%|███████▍  | 1498/2000 [00:46<00:15, 32.66it/s]\n",
      " 75%|███████▌  | 1502/2000 [00:46<00:15, 32.67it/s]\n",
      " 75%|███████▌  | 1506/2000 [00:46<00:15, 32.68it/s]\n",
      " 76%|███████▌  | 1510/2000 [00:46<00:15, 32.66it/s]\n",
      " 76%|███████▌  | 1514/2000 [00:46<00:14, 32.59it/s]\n",
      " 76%|███████▌  | 1518/2000 [00:47<00:14, 32.54it/s]\n",
      " 76%|███████▌  | 1522/2000 [00:47<00:14, 31.95it/s]\n",
      " 76%|███████▋  | 1526/2000 [00:47<00:14, 32.08it/s]\n",
      " 76%|███████▋  | 1530/2000 [00:47<00:14, 32.15it/s]\n",
      " 77%|███████▋  | 1534/2000 [00:47<00:14, 32.21it/s]\n",
      " 77%|███████▋  | 1538/2000 [00:47<00:14, 32.25it/s]\n",
      " 77%|███████▋  | 1542/2000 [00:47<00:14, 32.29it/s]\n",
      " 77%|███████▋  | 1546/2000 [00:47<00:14, 32.00it/s]\n",
      " 78%|███████▊  | 1550/2000 [00:48<00:13, 32.17it/s]\n",
      " 78%|███████▊  | 1554/2000 [00:48<00:13, 32.27it/s]\n",
      " 78%|███████▊  | 1558/2000 [00:48<00:13, 32.32it/s]\n",
      " 78%|███████▊  | 1562/2000 [00:48<00:13, 32.34it/s]\n",
      " 78%|███████▊  | 1566/2000 [00:48<00:13, 32.34it/s]\n",
      " 78%|███████▊  | 1570/2000 [00:48<00:13, 32.35it/s]\n",
      " 79%|███████▊  | 1574/2000 [00:48<00:13, 32.36it/s]\n",
      " 79%|███████▉  | 1578/2000 [00:48<00:13, 32.35it/s]\n",
      " 79%|███████▉  | 1582/2000 [00:49<00:13, 31.62it/s]\n",
      " 79%|███████▉  | 1586/2000 [00:49<00:12, 31.89it/s]\n",
      " 80%|███████▉  | 1590/2000 [00:49<00:12, 32.08it/s]\n",
      " 80%|███████▉  | 1594/2000 [00:49<00:12, 32.20it/s]\n",
      " 80%|███████▉  | 1598/2000 [00:49<00:12, 32.31it/s]\n",
      " 80%|████████  | 1602/2000 [00:49<00:12, 32.37it/s]\n",
      " 80%|████████  | 1606/2000 [00:49<00:12, 32.42it/s]\n",
      " 80%|████████  | 1610/2000 [00:49<00:12, 32.45it/s]\n",
      " 81%|████████  | 1614/2000 [00:49<00:11, 32.48it/s]\n",
      " 81%|████████  | 1618/2000 [00:50<00:11, 32.47it/s]\n",
      " 81%|████████  | 1622/2000 [00:50<00:11, 32.48it/s]\n",
      " 81%|████████▏ | 1626/2000 [00:50<00:11, 32.49it/s]\n",
      " 82%|████████▏ | 1630/2000 [00:50<00:11, 32.50it/s]\n",
      " 82%|████████▏ | 1634/2000 [00:50<00:11, 32.49it/s]\n",
      " 82%|████████▏ | 1638/2000 [00:50<00:11, 32.50it/s]\n",
      " 82%|████████▏ | 1642/2000 [00:50<00:11, 32.51it/s]\n",
      " 82%|████████▏ | 1646/2000 [00:50<00:11, 31.96it/s]\n",
      " 82%|████████▎ | 1650/2000 [00:51<00:11, 31.67it/s]\n",
      " 83%|████████▎ | 1654/2000 [00:51<00:10, 31.97it/s]\n",
      " 83%|████████▎ | 1658/2000 [00:51<00:10, 32.12it/s]\n",
      " 83%|████████▎ | 1662/2000 [00:51<00:10, 32.21it/s]\n",
      " 83%|████████▎ | 1666/2000 [00:51<00:10, 32.18it/s]\n",
      " 84%|████████▎ | 1670/2000 [00:51<00:10, 32.23it/s]\n",
      " 84%|████████▎ | 1674/2000 [00:51<00:10, 32.28it/s]\n",
      " 84%|████████▍ | 1678/2000 [00:51<00:09, 32.31it/s]\n",
      " 84%|████████▍ | 1682/2000 [00:52<00:09, 31.90it/s]\n",
      " 84%|████████▍ | 1686/2000 [00:52<00:09, 32.05it/s]\n",
      " 84%|████████▍ | 1690/2000 [00:52<00:09, 32.14it/s]\n",
      " 85%|████████▍ | 1694/2000 [00:52<00:09, 32.17it/s]\n",
      " 85%|████████▍ | 1698/2000 [00:52<00:09, 32.18it/s]\n",
      " 85%|████████▌ | 1702/2000 [00:52<00:09, 32.18it/s]\n",
      " 85%|████████▌ | 1706/2000 [00:52<00:09, 32.20it/s]\n",
      " 86%|████████▌ | 1710/2000 [00:52<00:09, 32.20it/s]\n",
      " 86%|████████▌ | 1714/2000 [00:53<00:08, 32.21it/s]\n",
      " 86%|████████▌ | 1718/2000 [00:53<00:08, 32.22it/s]\n",
      " 86%|████████▌ | 1722/2000 [00:53<00:08, 32.23it/s]\n",
      " 86%|████████▋ | 1726/2000 [00:53<00:08, 32.25it/s]\n",
      " 86%|████████▋ | 1730/2000 [00:53<00:08, 32.28it/s]\n",
      " 87%|████████▋ | 1734/2000 [00:53<00:08, 32.29it/s]\n",
      " 87%|████████▋ | 1738/2000 [00:53<00:08, 32.31it/s]\n",
      " 87%|████████▋ | 1742/2000 [00:53<00:07, 32.31it/s]\n",
      " 87%|████████▋ | 1746/2000 [00:54<00:07, 32.32it/s]\n",
      " 88%|████████▊ | 1750/2000 [00:54<00:07, 32.33it/s]\n",
      " 88%|████████▊ | 1754/2000 [00:54<00:07, 32.33it/s]\n",
      " 88%|████████▊ | 1758/2000 [00:54<00:07, 32.26it/s]\n",
      " 88%|████████▊ | 1762/2000 [00:54<00:07, 32.26it/s]\n",
      " 88%|████████▊ | 1766/2000 [00:54<00:07, 32.26it/s]\n",
      " 88%|████████▊ | 1770/2000 [00:54<00:07, 32.27it/s]\n",
      " 89%|████████▊ | 1774/2000 [00:54<00:07, 32.26it/s]\n",
      " 89%|████████▉ | 1778/2000 [00:55<00:06, 32.26it/s]\n",
      " 89%|████████▉ | 1782/2000 [00:55<00:06, 32.27it/s]\n",
      " 89%|████████▉ | 1786/2000 [00:55<00:06, 32.30it/s]\n",
      " 90%|████████▉ | 1790/2000 [00:55<00:06, 32.33it/s]\n",
      " 90%|████████▉ | 1794/2000 [00:55<00:06, 32.36it/s]\n",
      " 90%|████████▉ | 1798/2000 [00:55<00:06, 32.42it/s]\n",
      " 90%|█████████ | 1802/2000 [00:55<00:06, 32.47it/s]\n",
      " 90%|█████████ | 1806/2000 [00:55<00:05, 32.49it/s]\n",
      " 90%|█████████ | 1810/2000 [00:56<00:05, 32.09it/s]\n",
      " 91%|█████████ | 1814/2000 [00:56<00:05, 32.29it/s]\n",
      " 91%|█████████ | 1818/2000 [00:56<00:05, 32.40it/s]\n",
      " 91%|█████████ | 1822/2000 [00:56<00:05, 32.38it/s]\n",
      " 91%|█████████▏| 1826/2000 [00:56<00:05, 32.38it/s]\n",
      " 92%|█████████▏| 1830/2000 [00:56<00:05, 32.45it/s]\n",
      " 92%|█████████▏| 1834/2000 [00:56<00:05, 32.44it/s]\n",
      " 92%|█████████▏| 1838/2000 [00:56<00:05, 31.98it/s]\n",
      " 92%|█████████▏| 1842/2000 [00:57<00:04, 32.12it/s]\n",
      " 92%|█████████▏| 1846/2000 [00:57<00:04, 32.14it/s]\n",
      " 92%|█████████▎| 1850/2000 [00:57<00:04, 32.18it/s]\n",
      " 93%|█████████▎| 1854/2000 [00:57<00:04, 32.21it/s]\n",
      " 93%|█████████▎| 1858/2000 [00:57<00:04, 32.22it/s]\n",
      " 93%|█████████▎| 1862/2000 [00:57<00:04, 32.24it/s]\n",
      " 93%|█████████▎| 1866/2000 [00:57<00:04, 32.25it/s]\n",
      " 94%|█████████▎| 1870/2000 [00:57<00:04, 32.26it/s]\n",
      " 94%|█████████▎| 1874/2000 [00:58<00:03, 32.34it/s]\n",
      " 94%|█████████▍| 1878/2000 [00:58<00:03, 32.05it/s]\n",
      " 94%|█████████▍| 1882/2000 [00:58<00:03, 32.24it/s]\n",
      " 94%|█████████▍| 1886/2000 [00:58<00:03, 32.31it/s]\n",
      " 94%|█████████▍| 1890/2000 [00:58<00:03, 32.35it/s]\n",
      " 95%|█████████▍| 1894/2000 [00:58<00:03, 32.38it/s]\n",
      " 95%|█████████▍| 1898/2000 [00:58<00:03, 31.97it/s]\n",
      " 95%|█████████▌| 1902/2000 [00:58<00:03, 32.12it/s]\n",
      " 95%|█████████▌| 1906/2000 [00:59<00:02, 32.24it/s]\n",
      " 96%|█████████▌| 1910/2000 [00:59<00:02, 32.32it/s]\n",
      " 96%|█████████▌| 1914/2000 [00:59<00:02, 32.38it/s]\n",
      " 96%|█████████▌| 1918/2000 [00:59<00:02, 32.39it/s]\n",
      " 96%|█████████▌| 1922/2000 [00:59<00:02, 32.42it/s]\n",
      " 96%|█████████▋| 1926/2000 [00:59<00:02, 32.44it/s]\n",
      " 96%|█████████▋| 1930/2000 [00:59<00:02, 32.46it/s]\n",
      " 97%|█████████▋| 1934/2000 [00:59<00:02, 32.47it/s]\n",
      " 97%|█████████▋| 1938/2000 [01:00<00:01, 32.25it/s]\n",
      " 97%|█████████▋| 1942/2000 [01:00<00:01, 32.32it/s]\n",
      " 97%|█████████▋| 1946/2000 [01:00<00:01, 32.38it/s]\n",
      " 98%|█████████▊| 1950/2000 [01:00<00:01, 32.42it/s]\n",
      " 98%|█████████▊| 1954/2000 [01:00<00:01, 32.44it/s]\n",
      " 98%|█████████▊| 1958/2000 [01:00<00:01, 32.46it/s]\n",
      " 98%|█████████▊| 1962/2000 [01:00<00:01, 32.47it/s]\n",
      " 98%|█████████▊| 1966/2000 [01:00<00:01, 32.48it/s]\n",
      " 98%|█████████▊| 1970/2000 [01:01<00:00, 32.49it/s]\n",
      " 99%|█████████▊| 1974/2000 [01:01<00:00, 32.50it/s]\n",
      " 99%|█████████▉| 1978/2000 [01:01<00:00, 32.50it/s]\n",
      " 99%|█████████▉| 1982/2000 [01:01<00:00, 32.48it/s]\n",
      " 99%|█████████▉| 1986/2000 [01:01<00:00, 32.08it/s]\n",
      "100%|█████████▉| 1990/2000 [01:01<00:00, 32.29it/s]\n",
      "100%|█████████▉| 1994/2000 [01:01<00:00, 32.43it/s]\n",
      "100%|█████████▉| 1998/2000 [01:01<00:00, 32.54it/s]\n",
      "100%|██████████| 2000/2000 [01:01<00:00, 32.29it/s]\n",
      "\n",
      "  0%|          | 0/2000 [00:00<?, ?it/s]\n",
      "  0%|          | 3/2000 [00:00<01:21, 24.49it/s]\n",
      "  0%|          | 6/2000 [00:00<01:19, 25.13it/s]\n",
      "  0%|          | 9/2000 [00:00<01:17, 25.56it/s]\n",
      "  1%|          | 12/2000 [00:00<01:16, 25.87it/s]\n",
      "  1%|          | 15/2000 [00:00<01:16, 26.10it/s]\n",
      "  1%|          | 18/2000 [00:00<01:15, 26.25it/s]\n",
      "  1%|          | 21/2000 [00:00<01:15, 26.36it/s]\n",
      "  1%|          | 24/2000 [00:00<01:14, 26.43it/s]\n",
      "  1%|▏         | 27/2000 [00:01<01:14, 26.48it/s]\n",
      "  2%|▏         | 30/2000 [00:01<01:14, 26.51it/s]\n",
      "  2%|▏         | 33/2000 [00:01<01:14, 26.53it/s]\n",
      "  2%|▏         | 36/2000 [00:01<01:13, 26.55it/s]\n",
      "  2%|▏         | 39/2000 [00:01<01:13, 26.57it/s]\n",
      "  2%|▏         | 42/2000 [00:01<01:13, 26.58it/s]\n",
      "  2%|▏         | 45/2000 [00:01<01:13, 26.60it/s]\n",
      "  2%|▏         | 48/2000 [00:01<01:13, 26.62it/s]\n",
      "  3%|▎         | 51/2000 [00:01<01:13, 26.63it/s]\n",
      "  3%|▎         | 54/2000 [00:02<01:13, 26.63it/s]\n",
      "  3%|▎         | 57/2000 [00:02<01:13, 26.61it/s]\n",
      "  3%|▎         | 60/2000 [00:02<01:12, 26.61it/s]\n",
      "  3%|▎         | 63/2000 [00:02<01:14, 26.14it/s]\n",
      "  3%|▎         | 66/2000 [00:02<01:13, 26.39it/s]\n",
      "  3%|▎         | 69/2000 [00:02<01:12, 26.58it/s]\n",
      "  4%|▎         | 72/2000 [00:02<01:12, 26.72it/s]\n",
      "  4%|▍         | 75/2000 [00:02<01:11, 26.82it/s]\n",
      "  4%|▍         | 78/2000 [00:02<01:11, 26.82it/s]\n",
      "  4%|▍         | 81/2000 [00:03<01:11, 26.89it/s]\n",
      "  4%|▍         | 84/2000 [00:03<01:11, 26.93it/s]\n",
      "  4%|▍         | 87/2000 [00:03<01:11, 26.91it/s]\n",
      "  4%|▍         | 90/2000 [00:03<01:10, 26.91it/s]\n",
      "  5%|▍         | 93/2000 [00:03<01:10, 26.91it/s]\n",
      "  5%|▍         | 96/2000 [00:03<01:10, 26.89it/s]\n",
      "  5%|▍         | 99/2000 [00:03<01:10, 26.90it/s]\n",
      "  5%|▌         | 102/2000 [00:03<01:10, 26.90it/s]\n",
      "  5%|▌         | 105/2000 [00:03<01:10, 26.89it/s]\n",
      "  5%|▌         | 108/2000 [00:04<01:10, 26.90it/s]\n",
      "  6%|▌         | 111/2000 [00:04<01:10, 26.89it/s]\n",
      "  6%|▌         | 114/2000 [00:04<01:10, 26.88it/s]\n",
      "  6%|▌         | 117/2000 [00:04<01:10, 26.89it/s]\n",
      "  6%|▌         | 120/2000 [00:04<01:09, 26.89it/s]\n",
      "  6%|▌         | 123/2000 [00:04<01:09, 26.89it/s]\n",
      "  6%|▋         | 126/2000 [00:04<01:09, 26.88it/s]\n",
      "  6%|▋         | 129/2000 [00:04<01:09, 26.89it/s]\n",
      "  7%|▋         | 132/2000 [00:04<01:09, 26.80it/s]\n",
      "  7%|▋         | 135/2000 [00:05<01:09, 26.81it/s]\n",
      "  7%|▋         | 138/2000 [00:05<01:09, 26.81it/s]\n",
      "  7%|▋         | 141/2000 [00:05<01:09, 26.81it/s]\n",
      "  7%|▋         | 144/2000 [00:05<01:09, 26.81it/s]\n",
      "  7%|▋         | 147/2000 [00:05<01:09, 26.82it/s]\n",
      "  8%|▊         | 150/2000 [00:05<01:08, 26.83it/s]\n",
      "  8%|▊         | 153/2000 [00:05<01:10, 26.37it/s]\n",
      "  8%|▊         | 156/2000 [00:05<01:09, 26.48it/s]\n",
      "  8%|▊         | 159/2000 [00:05<01:09, 26.57it/s]\n",
      "  8%|▊         | 162/2000 [00:06<01:08, 26.67it/s]\n",
      "  8%|▊         | 165/2000 [00:06<01:08, 26.74it/s]\n",
      "  8%|▊         | 168/2000 [00:06<01:09, 26.47it/s]\n",
      "  9%|▊         | 171/2000 [00:06<01:08, 26.54it/s]\n",
      "  9%|▊         | 174/2000 [00:06<01:08, 26.61it/s]\n",
      "  9%|▉         | 177/2000 [00:06<01:08, 26.62it/s]\n",
      "  9%|▉         | 180/2000 [00:06<01:08, 26.64it/s]\n",
      "  9%|▉         | 183/2000 [00:06<01:08, 26.66it/s]\n",
      "  9%|▉         | 186/2000 [00:06<01:08, 26.67it/s]\n",
      "  9%|▉         | 189/2000 [00:07<01:07, 26.66it/s]\n",
      " 10%|▉         | 192/2000 [00:07<01:07, 26.67it/s]\n",
      " 10%|▉         | 195/2000 [00:07<01:07, 26.67it/s]\n",
      " 10%|▉         | 198/2000 [00:07<01:07, 26.67it/s]\n",
      " 10%|█         | 201/2000 [00:07<01:07, 26.67it/s]\n",
      " 10%|█         | 204/2000 [00:07<01:08, 26.38it/s]\n",
      " 10%|█         | 207/2000 [00:07<01:07, 26.59it/s]\n",
      " 10%|█         | 210/2000 [00:07<01:06, 26.75it/s]\n",
      " 11%|█         | 213/2000 [00:07<01:07, 26.52it/s]\n",
      " 11%|█         | 216/2000 [00:08<01:06, 26.67it/s]\n",
      " 11%|█         | 219/2000 [00:08<01:06, 26.74it/s]\n",
      " 11%|█         | 222/2000 [00:08<01:06, 26.80it/s]\n",
      " 11%|█▏        | 225/2000 [00:08<01:06, 26.85it/s]\n",
      " 11%|█▏        | 228/2000 [00:08<01:05, 26.87it/s]\n",
      " 12%|█▏        | 231/2000 [00:08<01:05, 26.86it/s]\n",
      " 12%|█▏        | 234/2000 [00:08<01:05, 26.89it/s]\n",
      " 12%|█▏        | 237/2000 [00:08<01:05, 26.91it/s]\n",
      " 12%|█▏        | 240/2000 [00:08<01:05, 26.91it/s]\n",
      " 12%|█▏        | 243/2000 [00:09<01:05, 26.94it/s]\n",
      " 12%|█▏        | 246/2000 [00:09<01:05, 26.96it/s]\n",
      " 12%|█▏        | 249/2000 [00:09<01:04, 26.94it/s]\n",
      " 13%|█▎        | 252/2000 [00:09<01:04, 26.94it/s]\n",
      " 13%|█▎        | 255/2000 [00:09<01:04, 26.95it/s]\n",
      " 13%|█▎        | 258/2000 [00:09<01:04, 26.84it/s]\n",
      " 13%|█▎        | 261/2000 [00:09<01:04, 26.82it/s]\n",
      " 13%|█▎        | 264/2000 [00:09<01:04, 26.80it/s]\n",
      " 13%|█▎        | 267/2000 [00:10<01:05, 26.45it/s]\n",
      " 14%|█▎        | 270/2000 [00:10<01:05, 26.60it/s]\n",
      " 14%|█▎        | 273/2000 [00:10<01:04, 26.71it/s]\n",
      " 14%|█▍        | 276/2000 [00:10<01:04, 26.79it/s]\n",
      " 14%|█▍        | 279/2000 [00:10<01:04, 26.84it/s]\n",
      " 14%|█▍        | 282/2000 [00:10<01:03, 26.87it/s]\n",
      " 14%|█▍        | 285/2000 [00:10<01:03, 26.89it/s]\n",
      " 14%|█▍        | 288/2000 [00:10<01:03, 26.91it/s]\n",
      " 15%|█▍        | 291/2000 [00:10<01:03, 26.91it/s]\n",
      " 15%|█▍        | 294/2000 [00:11<01:03, 26.90it/s]\n",
      " 15%|█▍        | 297/2000 [00:11<01:03, 26.91it/s]\n",
      " 15%|█▌        | 300/2000 [00:11<01:03, 26.92it/s]\n",
      " 15%|█▌        | 303/2000 [00:11<01:03, 26.93it/s]\n",
      " 15%|█▌        | 306/2000 [00:11<01:02, 26.93it/s]\n",
      " 15%|█▌        | 309/2000 [00:11<01:02, 26.93it/s]\n",
      " 16%|█▌        | 312/2000 [00:11<01:02, 26.93it/s]\n",
      " 16%|█▌        | 315/2000 [00:11<01:02, 26.93it/s]\n",
      " 16%|█▌        | 318/2000 [00:11<01:02, 26.94it/s]\n",
      " 16%|█▌        | 321/2000 [00:12<01:02, 26.93it/s]\n",
      " 16%|█▌        | 324/2000 [00:12<01:02, 26.94it/s]\n",
      " 16%|█▋        | 327/2000 [00:12<01:02, 26.94it/s]\n",
      " 16%|█▋        | 330/2000 [00:12<01:01, 26.97it/s]\n",
      " 17%|█▋        | 333/2000 [00:12<01:01, 27.00it/s]\n",
      " 17%|█▋        | 336/2000 [00:12<01:01, 27.01it/s]\n",
      " 17%|█▋        | 339/2000 [00:12<01:01, 27.02it/s]\n",
      " 17%|█▋        | 342/2000 [00:12<01:01, 26.98it/s]\n",
      " 17%|█▋        | 345/2000 [00:12<01:02, 26.46it/s]\n",
      " 17%|█▋        | 348/2000 [00:13<01:02, 26.55it/s]\n",
      " 18%|█▊        | 351/2000 [00:13<01:01, 26.61it/s]\n",
      " 18%|█▊        | 354/2000 [00:13<01:01, 26.65it/s]\n",
      " 18%|█▊        | 357/2000 [00:13<01:01, 26.67it/s]\n",
      " 18%|█▊        | 360/2000 [00:13<01:01, 26.68it/s]\n",
      " 18%|█▊        | 363/2000 [00:13<01:01, 26.70it/s]\n",
      " 18%|█▊        | 366/2000 [00:13<01:01, 26.71it/s]\n",
      " 18%|█▊        | 369/2000 [00:13<01:01, 26.72it/s]\n",
      " 19%|█▊        | 372/2000 [00:13<01:01, 26.44it/s]\n",
      " 19%|█▉        | 375/2000 [00:14<01:01, 26.58it/s]\n",
      " 19%|█▉        | 378/2000 [00:14<01:00, 26.69it/s]\n",
      " 19%|█▉        | 381/2000 [00:14<01:00, 26.72it/s]\n",
      " 19%|█▉        | 384/2000 [00:14<01:00, 26.79it/s]\n",
      " 19%|█▉        | 387/2000 [00:14<01:00, 26.83it/s]\n",
      " 20%|█▉        | 390/2000 [00:14<00:59, 26.87it/s]\n",
      " 20%|█▉        | 393/2000 [00:14<00:59, 26.89it/s]\n",
      " 20%|█▉        | 396/2000 [00:14<00:59, 26.92it/s]\n",
      " 20%|█▉        | 399/2000 [00:14<00:59, 26.94it/s]\n",
      " 20%|██        | 402/2000 [00:15<00:59, 26.94it/s]\n",
      " 20%|██        | 405/2000 [00:15<00:59, 26.95it/s]\n",
      " 20%|██        | 408/2000 [00:15<00:59, 26.96it/s]\n",
      " 21%|██        | 411/2000 [00:15<00:58, 26.97it/s]\n",
      " 21%|██        | 414/2000 [00:15<00:58, 26.98it/s]\n",
      " 21%|██        | 417/2000 [00:15<00:58, 26.99it/s]\n",
      " 21%|██        | 420/2000 [00:15<00:58, 26.99it/s]\n",
      " 21%|██        | 423/2000 [00:15<00:58, 26.99it/s]\n",
      " 21%|██▏       | 426/2000 [00:15<00:58, 26.99it/s]\n",
      " 21%|██▏       | 429/2000 [00:16<00:58, 26.99it/s]\n",
      " 22%|██▏       | 432/2000 [00:16<00:58, 27.00it/s]\n",
      " 22%|██▏       | 435/2000 [00:16<00:57, 26.99it/s]\n",
      " 22%|██▏       | 438/2000 [00:16<00:57, 26.99it/s]\n",
      " 22%|██▏       | 441/2000 [00:16<00:57, 26.98it/s]\n",
      " 22%|██▏       | 444/2000 [00:16<00:57, 26.97it/s]\n",
      " 22%|██▏       | 447/2000 [00:16<00:57, 26.98it/s]\n",
      " 22%|██▎       | 450/2000 [00:16<00:57, 26.99it/s]\n",
      " 23%|██▎       | 453/2000 [00:16<00:57, 26.98it/s]\n",
      " 23%|██▎       | 456/2000 [00:17<00:57, 26.97it/s]\n",
      " 23%|██▎       | 459/2000 [00:17<00:57, 26.97it/s]\n",
      " 23%|██▎       | 462/2000 [00:17<00:57, 26.97it/s]\n",
      " 23%|██▎       | 465/2000 [00:17<00:56, 26.97it/s]\n",
      " 23%|██▎       | 468/2000 [00:17<00:56, 26.97it/s]\n",
      " 24%|██▎       | 471/2000 [00:17<00:56, 26.97it/s]\n",
      " 24%|██▎       | 474/2000 [00:17<00:56, 26.97it/s]\n",
      " 24%|██▍       | 477/2000 [00:17<00:56, 26.96it/s]\n",
      " 24%|██▍       | 480/2000 [00:17<00:56, 26.96it/s]\n",
      " 24%|██▍       | 483/2000 [00:18<00:56, 26.97it/s]\n",
      " 24%|██▍       | 486/2000 [00:18<00:56, 27.00it/s]\n",
      " 24%|██▍       | 489/2000 [00:18<00:55, 26.99it/s]\n",
      " 25%|██▍       | 492/2000 [00:18<00:55, 27.03it/s]\n",
      " 25%|██▍       | 495/2000 [00:18<00:55, 27.07it/s]\n",
      " 25%|██▍       | 498/2000 [00:18<00:55, 27.08it/s]\n",
      " 25%|██▌       | 501/2000 [00:18<00:55, 27.06it/s]\n",
      " 25%|██▌       | 504/2000 [00:18<00:55, 27.01it/s]\n",
      " 25%|██▌       | 507/2000 [00:18<00:55, 26.97it/s]\n",
      " 26%|██▌       | 510/2000 [00:19<00:55, 26.95it/s]\n",
      " 26%|██▌       | 513/2000 [00:19<00:56, 26.48it/s]\n",
      " 26%|██▌       | 516/2000 [00:19<00:55, 26.60it/s]\n",
      " 26%|██▌       | 519/2000 [00:19<00:55, 26.68it/s]\n",
      " 26%|██▌       | 522/2000 [00:19<00:55, 26.74it/s]\n",
      " 26%|██▋       | 525/2000 [00:19<00:55, 26.78it/s]\n",
      " 26%|██▋       | 528/2000 [00:19<00:54, 26.81it/s]\n",
      " 27%|██▋       | 531/2000 [00:19<00:54, 26.80it/s]\n",
      " 27%|██▋       | 534/2000 [00:19<00:54, 26.82it/s]\n",
      " 27%|██▋       | 537/2000 [00:20<00:54, 26.81it/s]\n",
      " 27%|██▋       | 540/2000 [00:20<00:54, 26.82it/s]\n",
      " 27%|██▋       | 543/2000 [00:20<00:54, 26.82it/s]\n",
      " 27%|██▋       | 546/2000 [00:20<00:54, 26.84it/s]\n",
      " 27%|██▋       | 549/2000 [00:20<00:54, 26.57it/s]\n",
      " 28%|██▊       | 552/2000 [00:20<00:54, 26.68it/s]\n",
      " 28%|██▊       | 555/2000 [00:20<00:54, 26.76it/s]\n",
      " 28%|██▊       | 558/2000 [00:20<00:53, 26.81it/s]\n",
      " 28%|██▊       | 561/2000 [00:20<00:53, 26.79it/s]\n",
      " 28%|██▊       | 564/2000 [00:21<00:53, 26.81it/s]\n",
      " 28%|██▊       | 567/2000 [00:21<00:53, 26.82it/s]\n",
      " 28%|██▊       | 570/2000 [00:21<00:53, 26.84it/s]\n",
      " 29%|██▊       | 573/2000 [00:21<00:53, 26.86it/s]\n",
      " 29%|██▉       | 576/2000 [00:21<00:54, 26.37it/s]\n",
      " 29%|██▉       | 579/2000 [00:21<00:53, 26.55it/s]\n",
      " 29%|██▉       | 582/2000 [00:21<00:53, 26.67it/s]\n",
      " 29%|██▉       | 585/2000 [00:21<00:52, 26.76it/s]\n",
      " 29%|██▉       | 588/2000 [00:21<00:52, 26.83it/s]\n",
      " 30%|██▉       | 591/2000 [00:22<00:52, 26.87it/s]\n",
      " 30%|██▉       | 594/2000 [00:22<00:52, 26.89it/s]\n",
      " 30%|██▉       | 597/2000 [00:22<00:52, 26.91it/s]\n",
      " 30%|███       | 600/2000 [00:22<00:52, 26.92it/s]\n",
      " 30%|███       | 603/2000 [00:22<00:51, 26.93it/s]\n",
      " 30%|███       | 606/2000 [00:22<00:51, 26.83it/s]\n",
      " 30%|███       | 609/2000 [00:22<00:51, 26.86it/s]\n",
      " 31%|███       | 612/2000 [00:22<00:51, 26.87it/s]\n",
      " 31%|███       | 615/2000 [00:22<00:51, 26.86it/s]\n",
      " 31%|███       | 618/2000 [00:23<00:51, 26.83it/s]\n",
      " 31%|███       | 621/2000 [00:23<00:51, 26.82it/s]\n",
      " 31%|███       | 624/2000 [00:23<00:51, 26.80it/s]\n",
      " 31%|███▏      | 627/2000 [00:23<00:51, 26.80it/s]\n",
      " 32%|███▏      | 630/2000 [00:23<00:51, 26.79it/s]\n",
      " 32%|███▏      | 633/2000 [00:23<00:51, 26.79it/s]\n",
      " 32%|███▏      | 636/2000 [00:23<00:50, 26.79it/s]\n",
      " 32%|███▏      | 639/2000 [00:23<00:50, 26.78it/s]\n",
      " 32%|███▏      | 642/2000 [00:23<00:50, 26.78it/s]\n",
      " 32%|███▏      | 645/2000 [00:24<00:50, 26.79it/s]\n",
      " 32%|███▏      | 648/2000 [00:24<00:50, 26.79it/s]\n",
      " 33%|███▎      | 651/2000 [00:24<00:50, 26.79it/s]\n",
      " 33%|███▎      | 654/2000 [00:24<00:50, 26.79it/s]\n",
      " 33%|███▎      | 657/2000 [00:24<00:50, 26.78it/s]\n",
      " 33%|███▎      | 660/2000 [00:24<00:50, 26.78it/s]\n",
      " 33%|███▎      | 663/2000 [00:24<00:49, 26.78it/s]\n",
      " 33%|███▎      | 666/2000 [00:24<00:49, 26.79it/s]\n",
      " 33%|███▎      | 669/2000 [00:24<00:49, 26.78it/s]\n",
      " 34%|███▎      | 672/2000 [00:25<00:49, 26.78it/s]\n",
      " 34%|███▍      | 675/2000 [00:25<00:49, 26.78it/s]\n",
      " 34%|███▍      | 678/2000 [00:25<00:49, 26.76it/s]\n",
      " 34%|███▍      | 681/2000 [00:25<00:49, 26.65it/s]\n",
      " 34%|███▍      | 684/2000 [00:25<00:49, 26.65it/s]\n",
      " 34%|███▍      | 687/2000 [00:25<00:49, 26.64it/s]\n",
      " 34%|███▍      | 690/2000 [00:25<00:49, 26.63it/s]\n",
      " 35%|███▍      | 693/2000 [00:25<00:49, 26.29it/s]\n",
      " 35%|███▍      | 696/2000 [00:25<00:49, 26.50it/s]\n",
      " 35%|███▍      | 699/2000 [00:26<00:48, 26.66it/s]\n",
      " 35%|███▌      | 702/2000 [00:26<00:48, 26.74it/s]\n",
      " 35%|███▌      | 705/2000 [00:26<00:48, 26.76it/s]\n",
      " 35%|███▌      | 708/2000 [00:26<00:48, 26.76it/s]\n",
      " 36%|███▌      | 711/2000 [00:26<00:48, 26.77it/s]\n",
      " 36%|███▌      | 714/2000 [00:26<00:48, 26.77it/s]\n",
      " 36%|███▌      | 717/2000 [00:26<00:48, 26.23it/s]\n",
      " 36%|███▌      | 720/2000 [00:26<00:48, 26.38it/s]\n",
      " 36%|███▌      | 723/2000 [00:27<00:48, 26.48it/s]\n",
      " 36%|███▋      | 726/2000 [00:27<00:48, 26.53it/s]\n",
      " 36%|███▋      | 729/2000 [00:27<00:47, 26.58it/s]\n",
      " 37%|███▋      | 732/2000 [00:27<00:47, 26.61it/s]\n",
      " 37%|███▋      | 735/2000 [00:27<00:47, 26.64it/s]\n",
      " 37%|███▋      | 738/2000 [00:27<00:47, 26.31it/s]\n",
      " 37%|███▋      | 741/2000 [00:27<00:47, 26.53it/s]\n",
      " 37%|███▋      | 744/2000 [00:27<00:47, 26.69it/s]\n",
      " 37%|███▋      | 747/2000 [00:27<00:46, 26.81it/s]\n",
      " 38%|███▊      | 750/2000 [00:28<00:46, 26.90it/s]\n",
      " 38%|███▊      | 753/2000 [00:28<00:46, 26.89it/s]\n",
      " 38%|███▊      | 756/2000 [00:28<00:46, 26.93it/s]\n",
      " 38%|███▊      | 759/2000 [00:28<00:46, 26.93it/s]\n",
      " 38%|███▊      | 762/2000 [00:28<00:45, 26.96it/s]\n",
      " 38%|███▊      | 765/2000 [00:28<00:45, 26.96it/s]\n",
      " 38%|███▊      | 768/2000 [00:28<00:45, 26.98it/s]\n",
      " 39%|███▊      | 771/2000 [00:28<00:45, 27.00it/s]\n",
      " 39%|███▊      | 774/2000 [00:28<00:45, 27.01it/s]\n",
      " 39%|███▉      | 777/2000 [00:29<00:45, 27.03it/s]\n",
      " 39%|███▉      | 780/2000 [00:29<00:45, 27.05it/s]\n",
      " 39%|███▉      | 783/2000 [00:29<00:45, 26.69it/s]\n",
      " 39%|███▉      | 786/2000 [00:29<00:45, 26.69it/s]\n",
      " 39%|███▉      | 789/2000 [00:29<00:45, 26.70it/s]\n",
      " 40%|███▉      | 792/2000 [00:29<00:45, 26.70it/s]\n",
      " 40%|███▉      | 795/2000 [00:29<00:45, 26.70it/s]\n",
      " 40%|███▉      | 798/2000 [00:29<00:44, 26.71it/s]\n",
      " 40%|████      | 801/2000 [00:29<00:44, 26.72it/s]\n",
      " 40%|████      | 804/2000 [00:30<00:44, 26.70it/s]\n",
      " 40%|████      | 807/2000 [00:30<00:45, 26.32it/s]\n",
      " 40%|████      | 810/2000 [00:30<00:44, 26.55it/s]\n",
      " 41%|████      | 813/2000 [00:30<00:44, 26.70it/s]\n",
      " 41%|████      | 816/2000 [00:30<00:44, 26.82it/s]\n",
      " 41%|████      | 819/2000 [00:30<00:43, 26.88it/s]\n",
      " 41%|████      | 822/2000 [00:30<00:43, 26.92it/s]\n",
      " 41%|████▏     | 825/2000 [00:30<00:43, 26.94it/s]\n",
      " 41%|████▏     | 828/2000 [00:30<00:43, 26.89it/s]\n",
      " 42%|████▏     | 831/2000 [00:31<00:43, 26.92it/s]\n",
      " 42%|████▏     | 834/2000 [00:31<00:43, 26.95it/s]\n",
      " 42%|████▏     | 837/2000 [00:31<00:43, 26.96it/s]\n",
      " 42%|████▏     | 840/2000 [00:31<00:42, 26.98it/s]\n",
      " 42%|████▏     | 843/2000 [00:31<00:42, 26.99it/s]\n",
      " 42%|████▏     | 846/2000 [00:31<00:42, 27.01it/s]\n",
      " 42%|████▏     | 849/2000 [00:31<00:42, 27.04it/s]\n",
      " 43%|████▎     | 852/2000 [00:31<00:42, 27.05it/s]\n",
      " 43%|████▎     | 855/2000 [00:31<00:42, 27.06it/s]\n",
      " 43%|████▎     | 858/2000 [00:32<00:42, 27.02it/s]\n",
      " 43%|████▎     | 861/2000 [00:32<00:42, 27.01it/s]\n",
      " 43%|████▎     | 864/2000 [00:32<00:42, 27.02it/s]\n",
      " 43%|████▎     | 867/2000 [00:32<00:41, 27.05it/s]\n",
      " 44%|████▎     | 870/2000 [00:32<00:41, 27.06it/s]\n",
      " 44%|████▎     | 873/2000 [00:32<00:41, 27.07it/s]\n",
      " 44%|████▍     | 876/2000 [00:32<00:41, 27.09it/s]\n",
      " 44%|████▍     | 879/2000 [00:32<00:41, 27.07it/s]\n",
      " 44%|████▍     | 882/2000 [00:32<00:41, 27.07it/s]\n",
      " 44%|████▍     | 885/2000 [00:33<00:41, 27.07it/s]\n",
      " 44%|████▍     | 888/2000 [00:33<00:41, 27.08it/s]\n",
      " 45%|████▍     | 891/2000 [00:33<00:40, 27.08it/s]\n",
      " 45%|████▍     | 894/2000 [00:33<00:40, 27.08it/s]\n",
      " 45%|████▍     | 897/2000 [00:33<00:40, 27.08it/s]\n",
      " 45%|████▌     | 900/2000 [00:33<00:40, 27.09it/s]\n",
      " 45%|████▌     | 903/2000 [00:33<00:40, 26.99it/s]\n",
      " 45%|████▌     | 906/2000 [00:33<00:40, 27.02it/s]\n",
      " 45%|████▌     | 909/2000 [00:33<00:40, 27.04it/s]\n",
      " 46%|████▌     | 912/2000 [00:34<00:40, 27.06it/s]\n",
      " 46%|████▌     | 915/2000 [00:34<00:40, 27.07it/s]\n",
      " 46%|████▌     | 918/2000 [00:34<00:39, 27.07it/s]\n",
      " 46%|████▌     | 921/2000 [00:34<00:39, 27.08it/s]\n",
      " 46%|████▌     | 924/2000 [00:34<00:39, 27.09it/s]\n",
      " 46%|████▋     | 927/2000 [00:34<00:39, 27.10it/s]\n",
      " 46%|████▋     | 930/2000 [00:34<00:40, 26.74it/s]\n",
      " 47%|████▋     | 933/2000 [00:34<00:39, 26.74it/s]\n",
      " 47%|████▋     | 936/2000 [00:34<00:39, 26.77it/s]\n",
      " 47%|████▋     | 939/2000 [00:35<00:39, 26.79it/s]\n",
      " 47%|████▋     | 942/2000 [00:35<00:39, 26.80it/s]\n",
      " 47%|████▋     | 945/2000 [00:35<00:39, 26.80it/s]\n",
      " 47%|████▋     | 948/2000 [00:35<00:39, 26.81it/s]\n",
      " 48%|████▊     | 951/2000 [00:35<00:39, 26.82it/s]\n",
      " 48%|████▊     | 954/2000 [00:35<00:38, 26.82it/s]\n",
      " 48%|████▊     | 957/2000 [00:35<00:38, 26.83it/s]\n",
      " 48%|████▊     | 960/2000 [00:35<00:38, 26.83it/s]\n",
      " 48%|████▊     | 963/2000 [00:35<00:38, 26.82it/s]\n",
      " 48%|████▊     | 966/2000 [00:36<00:38, 26.81it/s]\n",
      " 48%|████▊     | 969/2000 [00:36<00:38, 26.81it/s]\n",
      " 49%|████▊     | 972/2000 [00:36<00:38, 26.81it/s]\n",
      " 49%|████▉     | 975/2000 [00:36<00:38, 26.83it/s]\n",
      " 49%|████▉     | 978/2000 [00:36<00:38, 26.69it/s]\n",
      " 49%|████▉     | 981/2000 [00:36<00:38, 26.76it/s]\n",
      " 49%|████▉     | 984/2000 [00:36<00:37, 26.83it/s]\n",
      " 49%|████▉     | 987/2000 [00:36<00:37, 26.87it/s]\n",
      " 50%|████▉     | 990/2000 [00:36<00:37, 26.89it/s]\n",
      " 50%|████▉     | 993/2000 [00:37<00:37, 26.92it/s]\n",
      " 50%|████▉     | 996/2000 [00:37<00:37, 26.95it/s]\n",
      " 50%|████▉     | 999/2000 [00:37<00:37, 26.95it/s]\n",
      " 50%|█████     | 1002/2000 [00:37<00:37, 26.93it/s]\n",
      " 50%|█████     | 1005/2000 [00:37<00:37, 26.88it/s]\n",
      " 50%|█████     | 1008/2000 [00:37<00:36, 26.83it/s]\n",
      " 51%|█████     | 1011/2000 [00:37<00:36, 26.79it/s]\n",
      " 51%|█████     | 1014/2000 [00:37<00:36, 26.78it/s]\n",
      " 51%|█████     | 1017/2000 [00:37<00:36, 26.76it/s]\n",
      " 51%|█████     | 1020/2000 [00:38<00:37, 26.41it/s]\n",
      " 51%|█████     | 1023/2000 [00:38<00:36, 26.61it/s]\n",
      " 51%|█████▏    | 1026/2000 [00:38<00:36, 26.76it/s]\n",
      " 51%|█████▏    | 1029/2000 [00:38<00:36, 26.86it/s]\n",
      " 52%|█████▏    | 1032/2000 [00:38<00:35, 26.93it/s]\n",
      " 52%|█████▏    | 1035/2000 [00:38<00:35, 26.88it/s]\n",
      " 52%|█████▏    | 1038/2000 [00:38<00:35, 26.91it/s]\n",
      " 52%|█████▏    | 1041/2000 [00:38<00:35, 26.90it/s]\n",
      " 52%|█████▏    | 1044/2000 [00:38<00:35, 26.89it/s]\n",
      " 52%|█████▏    | 1047/2000 [00:39<00:35, 26.87it/s]\n",
      " 52%|█████▎    | 1050/2000 [00:39<00:35, 26.87it/s]\n",
      " 53%|█████▎    | 1053/2000 [00:39<00:35, 26.77it/s]\n",
      " 53%|█████▎    | 1056/2000 [00:39<00:35, 26.80it/s]\n",
      " 53%|█████▎    | 1059/2000 [00:39<00:35, 26.81it/s]\n",
      " 53%|█████▎    | 1062/2000 [00:39<00:34, 26.83it/s]\n",
      " 53%|█████▎    | 1065/2000 [00:39<00:34, 26.84it/s]\n",
      " 53%|█████▎    | 1068/2000 [00:39<00:34, 26.84it/s]\n",
      " 54%|█████▎    | 1071/2000 [00:39<00:34, 26.85it/s]\n",
      " 54%|█████▎    | 1074/2000 [00:40<00:34, 26.83it/s]\n",
      " 54%|█████▍    | 1077/2000 [00:40<00:34, 26.84it/s]\n",
      " 54%|█████▍    | 1080/2000 [00:40<00:34, 26.36it/s]\n",
      " 54%|█████▍    | 1083/2000 [00:40<00:34, 26.52it/s]\n",
      " 54%|█████▍    | 1086/2000 [00:40<00:34, 26.62it/s]\n",
      " 54%|█████▍    | 1089/2000 [00:40<00:34, 26.68it/s]\n",
      " 55%|█████▍    | 1092/2000 [00:40<00:33, 26.73it/s]\n",
      " 55%|█████▍    | 1095/2000 [00:40<00:33, 26.76it/s]\n",
      " 55%|█████▍    | 1098/2000 [00:40<00:33, 26.78it/s]\n",
      " 55%|█████▌    | 1101/2000 [00:41<00:33, 26.81it/s]\n",
      " 55%|█████▌    | 1104/2000 [00:41<00:33, 26.83it/s]\n",
      " 55%|█████▌    | 1107/2000 [00:41<00:33, 26.87it/s]\n",
      " 56%|█████▌    | 1110/2000 [00:41<00:33, 26.90it/s]\n",
      " 56%|█████▌    | 1113/2000 [00:41<00:32, 26.90it/s]\n",
      " 56%|█████▌    | 1116/2000 [00:41<00:32, 26.92it/s]\n",
      " 56%|█████▌    | 1119/2000 [00:41<00:32, 26.94it/s]\n",
      " 56%|█████▌    | 1122/2000 [00:41<00:32, 26.95it/s]\n",
      " 56%|█████▋    | 1125/2000 [00:41<00:32, 26.95it/s]\n",
      " 56%|█████▋    | 1128/2000 [00:42<00:32, 26.93it/s]\n",
      " 57%|█████▋    | 1131/2000 [00:42<00:32, 26.95it/s]\n",
      " 57%|█████▋    | 1134/2000 [00:42<00:32, 26.95it/s]\n",
      " 57%|█████▋    | 1137/2000 [00:42<00:32, 26.96it/s]\n",
      " 57%|█████▋    | 1140/2000 [00:42<00:32, 26.48it/s]\n",
      " 57%|█████▋    | 1143/2000 [00:42<00:32, 26.62it/s]\n",
      " 57%|█████▋    | 1146/2000 [00:42<00:31, 26.71it/s]\n",
      " 57%|█████▋    | 1149/2000 [00:42<00:31, 26.71it/s]\n",
      " 58%|█████▊    | 1152/2000 [00:42<00:31, 26.71it/s]\n",
      " 58%|█████▊    | 1155/2000 [00:43<00:31, 26.73it/s]\n",
      " 58%|█████▊    | 1158/2000 [00:43<00:31, 26.74it/s]\n",
      " 58%|█████▊    | 1161/2000 [00:43<00:31, 26.38it/s]\n",
      " 58%|█████▊    | 1164/2000 [00:43<00:31, 26.59it/s]\n",
      " 58%|█████▊    | 1167/2000 [00:43<00:31, 26.68it/s]\n",
      " 58%|█████▊    | 1170/2000 [00:43<00:31, 26.73it/s]\n",
      " 59%|█████▊    | 1173/2000 [00:43<00:30, 26.76it/s]\n",
      " 59%|█████▉    | 1176/2000 [00:43<00:30, 26.80it/s]\n",
      " 59%|█████▉    | 1179/2000 [00:43<00:30, 26.83it/s]\n",
      " 59%|█████▉    | 1182/2000 [00:44<00:30, 26.85it/s]\n",
      " 59%|█████▉    | 1185/2000 [00:44<00:30, 26.85it/s]\n",
      " 59%|█████▉    | 1188/2000 [00:44<00:30, 26.85it/s]\n",
      " 60%|█████▉    | 1191/2000 [00:44<00:30, 26.85it/s]\n",
      " 60%|█████▉    | 1194/2000 [00:44<00:30, 26.84it/s]\n",
      " 60%|█████▉    | 1197/2000 [00:44<00:29, 26.84it/s]\n",
      " 60%|██████    | 1200/2000 [00:44<00:29, 26.85it/s]\n",
      " 60%|██████    | 1203/2000 [00:44<00:29, 26.81it/s]\n",
      " 60%|██████    | 1206/2000 [00:44<00:29, 26.81it/s]\n",
      " 60%|██████    | 1209/2000 [00:45<00:29, 26.82it/s]\n",
      " 61%|██████    | 1212/2000 [00:45<00:29, 26.84it/s]\n",
      " 61%|██████    | 1215/2000 [00:45<00:29, 26.84it/s]\n",
      " 61%|██████    | 1218/2000 [00:45<00:29, 26.84it/s]\n",
      " 61%|██████    | 1221/2000 [00:45<00:29, 26.84it/s]\n",
      " 61%|██████    | 1224/2000 [00:45<00:28, 26.85it/s]\n",
      " 61%|██████▏   | 1227/2000 [00:45<00:28, 26.85it/s]\n",
      " 62%|██████▏   | 1230/2000 [00:45<00:28, 26.86it/s]\n",
      " 62%|██████▏   | 1233/2000 [00:45<00:28, 26.86it/s]\n",
      " 62%|██████▏   | 1236/2000 [00:46<00:28, 26.86it/s]\n",
      " 62%|██████▏   | 1239/2000 [00:46<00:28, 26.87it/s]\n",
      " 62%|██████▏   | 1242/2000 [00:46<00:28, 26.86it/s]\n",
      " 62%|██████▏   | 1245/2000 [00:46<00:28, 26.86it/s]\n",
      " 62%|██████▏   | 1248/2000 [00:46<00:28, 26.85it/s]\n",
      " 63%|██████▎   | 1251/2000 [00:46<00:27, 26.86it/s]\n",
      " 63%|██████▎   | 1254/2000 [00:46<00:27, 26.85it/s]\n",
      " 63%|██████▎   | 1257/2000 [00:46<00:27, 26.85it/s]\n",
      " 63%|██████▎   | 1260/2000 [00:47<00:27, 26.83it/s]\n",
      " 63%|██████▎   | 1263/2000 [00:47<00:27, 26.82it/s]\n",
      " 63%|██████▎   | 1266/2000 [00:47<00:27, 26.82it/s]\n",
      " 63%|██████▎   | 1269/2000 [00:47<00:27, 26.82it/s]\n",
      " 64%|██████▎   | 1272/2000 [00:47<00:27, 26.82it/s]\n",
      " 64%|██████▍   | 1275/2000 [00:47<00:27, 26.83it/s]\n",
      " 64%|██████▍   | 1278/2000 [00:47<00:26, 26.83it/s]\n",
      " 64%|██████▍   | 1281/2000 [00:47<00:26, 26.81it/s]\n",
      " 64%|██████▍   | 1284/2000 [00:47<00:26, 26.83it/s]\n",
      " 64%|██████▍   | 1287/2000 [00:48<00:26, 26.83it/s]\n",
      " 64%|██████▍   | 1290/2000 [00:48<00:26, 26.83it/s]\n",
      " 65%|██████▍   | 1293/2000 [00:48<00:26, 26.84it/s]\n",
      " 65%|██████▍   | 1296/2000 [00:48<00:26, 26.85it/s]\n",
      " 65%|██████▍   | 1299/2000 [00:48<00:26, 26.85it/s]\n",
      " 65%|██████▌   | 1302/2000 [00:48<00:25, 26.85it/s]\n",
      " 65%|██████▌   | 1305/2000 [00:48<00:26, 26.54it/s]\n",
      " 65%|██████▌   | 1308/2000 [00:48<00:25, 26.66it/s]\n",
      " 66%|██████▌   | 1311/2000 [00:48<00:25, 26.71it/s]\n",
      " 66%|██████▌   | 1314/2000 [00:49<00:25, 26.72it/s]\n",
      " 66%|██████▌   | 1317/2000 [00:49<00:25, 26.72it/s]\n",
      " 66%|██████▌   | 1320/2000 [00:49<00:25, 26.71it/s]\n",
      " 66%|██████▌   | 1323/2000 [00:49<00:25, 26.71it/s]\n",
      " 66%|██████▋   | 1326/2000 [00:49<00:25, 26.72it/s]\n",
      " 66%|██████▋   | 1329/2000 [00:49<00:25, 26.71it/s]\n",
      " 67%|██████▋   | 1332/2000 [00:49<00:25, 26.71it/s]\n",
      " 67%|██████▋   | 1335/2000 [00:49<00:24, 26.72it/s]\n",
      " 67%|██████▋   | 1338/2000 [00:49<00:24, 26.73it/s]\n",
      " 67%|██████▋   | 1341/2000 [00:50<00:24, 26.72it/s]\n",
      " 67%|██████▋   | 1344/2000 [00:50<00:24, 26.73it/s]\n",
      " 67%|██████▋   | 1347/2000 [00:50<00:24, 26.74it/s]\n",
      " 68%|██████▊   | 1350/2000 [00:50<00:24, 26.72it/s]\n",
      " 68%|██████▊   | 1353/2000 [00:50<00:24, 26.72it/s]\n",
      " 68%|██████▊   | 1356/2000 [00:50<00:24, 26.71it/s]\n",
      " 68%|██████▊   | 1359/2000 [00:50<00:23, 26.72it/s]\n",
      " 68%|██████▊   | 1362/2000 [00:50<00:23, 26.71it/s]\n",
      " 68%|██████▊   | 1365/2000 [00:50<00:23, 26.71it/s]\n",
      " 68%|██████▊   | 1368/2000 [00:51<00:23, 26.70it/s]\n",
      " 69%|██████▊   | 1371/2000 [00:51<00:23, 26.70it/s]\n",
      " 69%|██████▊   | 1374/2000 [00:51<00:23, 26.70it/s]\n",
      " 69%|██████▉   | 1377/2000 [00:51<00:23, 26.71it/s]\n",
      " 69%|██████▉   | 1380/2000 [00:51<00:23, 26.72it/s]\n",
      " 69%|██████▉   | 1383/2000 [00:51<00:23, 26.73it/s]\n",
      " 69%|██████▉   | 1386/2000 [00:51<00:22, 26.72it/s]\n",
      " 69%|██████▉   | 1389/2000 [00:51<00:22, 26.72it/s]\n",
      " 70%|██████▉   | 1392/2000 [00:51<00:22, 26.72it/s]\n",
      " 70%|██████▉   | 1395/2000 [00:52<00:22, 26.72it/s]\n",
      " 70%|██████▉   | 1398/2000 [00:52<00:22, 26.71it/s]\n",
      " 70%|███████   | 1401/2000 [00:52<00:22, 26.70it/s]\n",
      " 70%|███████   | 1404/2000 [00:52<00:22, 26.71it/s]\n",
      " 70%|███████   | 1407/2000 [00:52<00:22, 26.71it/s]\n",
      " 70%|███████   | 1410/2000 [00:52<00:22, 26.71it/s]\n",
      " 71%|███████   | 1413/2000 [00:52<00:21, 26.71it/s]\n",
      " 71%|███████   | 1416/2000 [00:52<00:21, 26.72it/s]\n",
      " 71%|███████   | 1419/2000 [00:52<00:21, 26.72it/s]\n",
      " 71%|███████   | 1422/2000 [00:53<00:21, 26.72it/s]\n",
      " 71%|███████▏  | 1425/2000 [00:53<00:21, 26.61it/s]\n",
      " 71%|███████▏  | 1428/2000 [00:53<00:21, 26.61it/s]\n",
      " 72%|███████▏  | 1431/2000 [00:53<00:21, 26.61it/s]\n",
      " 72%|███████▏  | 1434/2000 [00:53<00:21, 26.60it/s]\n",
      " 72%|███████▏  | 1437/2000 [00:53<00:21, 26.61it/s]\n",
      " 72%|███████▏  | 1440/2000 [00:53<00:21, 26.65it/s]\n",
      " 72%|███████▏  | 1443/2000 [00:53<00:20, 26.67it/s]\n",
      " 72%|███████▏  | 1446/2000 [00:53<00:20, 26.68it/s]\n",
      " 72%|███████▏  | 1449/2000 [00:54<00:20, 26.69it/s]\n",
      " 73%|███████▎  | 1452/2000 [00:54<00:20, 26.70it/s]\n",
      " 73%|███████▎  | 1455/2000 [00:54<00:20, 26.70it/s]\n",
      " 73%|███████▎  | 1458/2000 [00:54<00:20, 26.71it/s]\n",
      " 73%|███████▎  | 1461/2000 [00:54<00:20, 26.70it/s]\n",
      " 73%|███████▎  | 1464/2000 [00:54<00:20, 26.69it/s]\n",
      " 73%|███████▎  | 1467/2000 [00:54<00:19, 26.71it/s]\n",
      " 74%|███████▎  | 1470/2000 [00:54<00:19, 26.72it/s]\n",
      " 74%|███████▎  | 1473/2000 [00:54<00:19, 26.71it/s]\n",
      " 74%|███████▍  | 1476/2000 [00:55<00:19, 26.33it/s]\n",
      " 74%|███████▍  | 1479/2000 [00:55<00:19, 26.50it/s]\n",
      " 74%|███████▍  | 1482/2000 [00:55<00:19, 26.63it/s]\n",
      " 74%|███████▍  | 1485/2000 [00:55<00:19, 26.72it/s]\n",
      " 74%|███████▍  | 1488/2000 [00:55<00:19, 26.79it/s]\n",
      " 75%|███████▍  | 1491/2000 [00:55<00:18, 26.84it/s]\n",
      " 75%|███████▍  | 1494/2000 [00:55<00:18, 26.88it/s]\n",
      " 75%|███████▍  | 1497/2000 [00:55<00:18, 26.88it/s]\n",
      " 75%|███████▌  | 1500/2000 [00:55<00:18, 26.81it/s]\n",
      " 75%|███████▌  | 1503/2000 [00:56<00:18, 26.80it/s]\n",
      " 75%|███████▌  | 1506/2000 [00:56<00:18, 26.80it/s]\n",
      " 75%|███████▌  | 1509/2000 [00:56<00:18, 26.80it/s]\n",
      " 76%|███████▌  | 1512/2000 [00:56<00:18, 26.37it/s]\n",
      " 76%|███████▌  | 1515/2000 [00:56<00:18, 26.53it/s]\n",
      " 76%|███████▌  | 1518/2000 [00:56<00:18, 26.63it/s]\n",
      " 76%|███████▌  | 1521/2000 [00:56<00:17, 26.72it/s]\n",
      " 76%|███████▌  | 1524/2000 [00:56<00:17, 26.80it/s]\n",
      " 76%|███████▋  | 1527/2000 [00:56<00:17, 26.84it/s]\n",
      " 76%|███████▋  | 1530/2000 [00:57<00:17, 26.87it/s]\n",
      " 77%|███████▋  | 1533/2000 [00:57<00:17, 26.85it/s]\n",
      " 77%|███████▋  | 1536/2000 [00:57<00:17, 26.85it/s]\n",
      " 77%|███████▋  | 1539/2000 [00:57<00:17, 26.88it/s]\n",
      " 77%|███████▋  | 1542/2000 [00:57<00:17, 26.89it/s]\n",
      " 77%|███████▋  | 1545/2000 [00:57<00:16, 26.89it/s]\n",
      " 77%|███████▋  | 1548/2000 [00:57<00:16, 26.90it/s]\n",
      " 78%|███████▊  | 1551/2000 [00:57<00:16, 26.90it/s]\n",
      " 78%|███████▊  | 1554/2000 [00:58<00:16, 26.91it/s]\n",
      " 78%|███████▊  | 1557/2000 [00:58<00:16, 26.91it/s]\n",
      " 78%|███████▊  | 1560/2000 [00:58<00:16, 26.92it/s]\n",
      " 78%|███████▊  | 1563/2000 [00:58<00:16, 26.92it/s]\n",
      " 78%|███████▊  | 1566/2000 [00:58<00:16, 26.91it/s]\n",
      " 78%|███████▊  | 1569/2000 [00:58<00:16, 26.90it/s]\n",
      " 79%|███████▊  | 1572/2000 [00:58<00:15, 26.91it/s]\n",
      " 79%|███████▉  | 1575/2000 [00:58<00:15, 26.86it/s]\n",
      " 79%|███████▉  | 1578/2000 [00:58<00:15, 26.87it/s]\n",
      " 79%|███████▉  | 1581/2000 [00:59<00:15, 26.88it/s]\n",
      " 79%|███████▉  | 1584/2000 [00:59<00:15, 26.89it/s]\n",
      " 79%|███████▉  | 1587/2000 [00:59<00:15, 26.89it/s]\n",
      " 80%|███████▉  | 1590/2000 [00:59<00:15, 26.90it/s]\n",
      " 80%|███████▉  | 1593/2000 [00:59<00:15, 26.91it/s]\n",
      " 80%|███████▉  | 1596/2000 [00:59<00:15, 26.63it/s]\n",
      " 80%|███████▉  | 1599/2000 [00:59<00:15, 26.44it/s]\n",
      " 80%|████████  | 1602/2000 [00:59<00:14, 26.55it/s]\n",
      " 80%|████████  | 1605/2000 [00:59<00:14, 26.62it/s]\n",
      " 80%|████████  | 1608/2000 [01:00<00:14, 26.66it/s]\n",
      " 81%|████████  | 1611/2000 [01:00<00:14, 26.70it/s]\n",
      " 81%|████████  | 1614/2000 [01:00<00:14, 26.72it/s]\n",
      " 81%|████████  | 1617/2000 [01:00<00:14, 26.75it/s]\n",
      " 81%|████████  | 1620/2000 [01:00<00:14, 26.76it/s]\n",
      " 81%|████████  | 1623/2000 [01:00<00:14, 26.76it/s]\n",
      " 81%|████████▏ | 1626/2000 [01:00<00:13, 26.78it/s]\n",
      " 81%|████████▏ | 1629/2000 [01:00<00:13, 26.79it/s]\n",
      " 82%|████████▏ | 1632/2000 [01:00<00:13, 26.78it/s]\n",
      " 82%|████████▏ | 1635/2000 [01:01<00:13, 26.83it/s]\n",
      " 82%|████████▏ | 1638/2000 [01:01<00:13, 26.87it/s]\n",
      " 82%|████████▏ | 1641/2000 [01:01<00:13, 26.90it/s]\n",
      " 82%|████████▏ | 1644/2000 [01:01<00:13, 26.90it/s]\n",
      " 82%|████████▏ | 1647/2000 [01:01<00:13, 26.15it/s]\n",
      " 82%|████████▎ | 1650/2000 [01:01<00:13, 26.27it/s]\n",
      " 83%|████████▎ | 1653/2000 [01:01<00:13, 26.51it/s]\n",
      " 83%|████████▎ | 1656/2000 [01:01<00:12, 26.69it/s]\n",
      " 83%|████████▎ | 1659/2000 [01:01<00:12, 26.81it/s]\n",
      " 83%|████████▎ | 1662/2000 [01:02<00:12, 26.90it/s]\n",
      " 83%|████████▎ | 1665/2000 [01:02<00:12, 26.96it/s]\n",
      " 83%|████████▎ | 1668/2000 [01:02<00:12, 27.00it/s]\n",
      " 84%|████████▎ | 1671/2000 [01:02<00:12, 27.01it/s]\n",
      " 84%|████████▎ | 1674/2000 [01:02<00:12, 27.04it/s]\n",
      " 84%|████████▍ | 1677/2000 [01:02<00:11, 27.06it/s]\n",
      " 84%|████████▍ | 1680/2000 [01:02<00:11, 26.72it/s]\n",
      " 84%|████████▍ | 1683/2000 [01:02<00:11, 26.78it/s]\n",
      " 84%|████████▍ | 1686/2000 [01:02<00:11, 26.83it/s]\n",
      " 84%|████████▍ | 1689/2000 [01:03<00:11, 26.86it/s]\n",
      " 85%|████████▍ | 1692/2000 [01:03<00:11, 26.82it/s]\n",
      " 85%|████████▍ | 1695/2000 [01:03<00:11, 26.78it/s]\n",
      " 85%|████████▍ | 1698/2000 [01:03<00:11, 26.75it/s]\n",
      " 85%|████████▌ | 1701/2000 [01:03<00:11, 26.72it/s]\n",
      " 85%|████████▌ | 1704/2000 [01:03<00:11, 26.71it/s]\n",
      " 85%|████████▌ | 1707/2000 [01:03<00:10, 26.71it/s]\n",
      " 86%|████████▌ | 1710/2000 [01:03<00:10, 26.71it/s]\n",
      " 86%|████████▌ | 1713/2000 [01:03<00:10, 26.69it/s]\n",
      " 86%|████████▌ | 1716/2000 [01:04<00:10, 26.66it/s]\n",
      " 86%|████████▌ | 1719/2000 [01:04<00:10, 26.65it/s]\n",
      " 86%|████████▌ | 1722/2000 [01:04<00:10, 26.11it/s]\n",
      " 86%|████████▋ | 1725/2000 [01:04<00:10, 26.33it/s]\n",
      " 86%|████████▋ | 1728/2000 [01:04<00:10, 26.49it/s]\n",
      " 87%|████████▋ | 1731/2000 [01:04<00:10, 26.61it/s]\n",
      " 87%|████████▋ | 1734/2000 [01:04<00:09, 26.70it/s]\n",
      " 87%|████████▋ | 1737/2000 [01:04<00:09, 26.77it/s]\n",
      " 87%|████████▋ | 1740/2000 [01:04<00:09, 26.81it/s]\n",
      " 87%|████████▋ | 1743/2000 [01:05<00:09, 26.84it/s]\n",
      " 87%|████████▋ | 1746/2000 [01:05<00:09, 26.86it/s]\n",
      " 87%|████████▋ | 1749/2000 [01:05<00:09, 26.87it/s]\n",
      " 88%|████████▊ | 1752/2000 [01:05<00:09, 26.86it/s]\n",
      " 88%|████████▊ | 1755/2000 [01:05<00:09, 26.88it/s]\n",
      " 88%|████████▊ | 1758/2000 [01:05<00:09, 26.88it/s]\n",
      " 88%|████████▊ | 1761/2000 [01:05<00:08, 26.89it/s]\n",
      " 88%|████████▊ | 1764/2000 [01:05<00:08, 26.90it/s]\n",
      " 88%|████████▊ | 1767/2000 [01:05<00:08, 26.91it/s]\n",
      " 88%|████████▊ | 1770/2000 [01:06<00:08, 26.90it/s]\n",
      " 89%|████████▊ | 1773/2000 [01:06<00:08, 26.92it/s]\n",
      " 89%|████████▉ | 1776/2000 [01:06<00:08, 26.46it/s]\n",
      " 89%|████████▉ | 1779/2000 [01:06<00:08, 26.30it/s]\n",
      " 89%|████████▉ | 1782/2000 [01:06<00:08, 26.51it/s]\n",
      " 89%|████████▉ | 1785/2000 [01:06<00:08, 26.66it/s]\n",
      " 89%|████████▉ | 1788/2000 [01:06<00:07, 26.77it/s]\n",
      " 90%|████████▉ | 1791/2000 [01:06<00:07, 26.83it/s]\n",
      " 90%|████████▉ | 1794/2000 [01:06<00:07, 26.89it/s]\n",
      " 90%|████████▉ | 1797/2000 [01:07<00:07, 26.84it/s]\n",
      " 90%|█████████ | 1800/2000 [01:07<00:07, 26.85it/s]\n",
      " 90%|█████████ | 1803/2000 [01:07<00:07, 26.44it/s]\n",
      " 90%|█████████ | 1806/2000 [01:07<00:07, 26.53it/s]\n",
      " 90%|█████████ | 1809/2000 [01:07<00:07, 26.60it/s]\n",
      " 91%|█████████ | 1812/2000 [01:07<00:07, 26.65it/s]\n",
      " 91%|█████████ | 1815/2000 [01:07<00:06, 26.66it/s]\n",
      " 91%|█████████ | 1818/2000 [01:07<00:06, 26.70it/s]\n",
      " 91%|█████████ | 1821/2000 [01:07<00:06, 26.72it/s]\n",
      " 91%|█████████ | 1824/2000 [01:08<00:06, 26.75it/s]\n",
      " 91%|█████████▏| 1827/2000 [01:08<00:06, 26.79it/s]\n",
      " 92%|█████████▏| 1830/2000 [01:08<00:06, 26.80it/s]\n",
      " 92%|█████████▏| 1833/2000 [01:08<00:06, 26.82it/s]\n",
      " 92%|█████████▏| 1836/2000 [01:08<00:06, 26.83it/s]\n",
      " 92%|█████████▏| 1839/2000 [01:08<00:06, 26.83it/s]\n",
      " 92%|█████████▏| 1842/2000 [01:08<00:05, 26.84it/s]\n",
      " 92%|█████████▏| 1845/2000 [01:08<00:05, 26.84it/s]\n",
      " 92%|█████████▏| 1848/2000 [01:08<00:05, 26.83it/s]\n",
      " 93%|█████████▎| 1851/2000 [01:09<00:05, 26.84it/s]\n",
      " 93%|█████████▎| 1854/2000 [01:09<00:05, 26.84it/s]\n",
      " 93%|█████████▎| 1857/2000 [01:09<00:05, 26.81it/s]\n",
      " 93%|█████████▎| 1860/2000 [01:09<00:05, 26.79it/s]\n",
      " 93%|█████████▎| 1863/2000 [01:09<00:05, 26.79it/s]\n",
      " 93%|█████████▎| 1866/2000 [01:09<00:05, 26.78it/s]\n",
      " 93%|█████████▎| 1869/2000 [01:09<00:04, 26.76it/s]\n",
      " 94%|█████████▎| 1872/2000 [01:09<00:04, 26.72it/s]\n",
      " 94%|█████████▍| 1875/2000 [01:09<00:04, 26.73it/s]\n",
      " 94%|█████████▍| 1878/2000 [01:10<00:04, 26.72it/s]\n",
      " 94%|█████████▍| 1881/2000 [01:10<00:04, 26.74it/s]\n",
      " 94%|█████████▍| 1884/2000 [01:10<00:04, 26.76it/s]\n",
      " 94%|█████████▍| 1887/2000 [01:10<00:04, 26.77it/s]\n",
      " 94%|█████████▍| 1890/2000 [01:10<00:04, 26.77it/s]\n",
      " 95%|█████████▍| 1893/2000 [01:10<00:03, 26.77it/s]\n",
      " 95%|█████████▍| 1896/2000 [01:10<00:03, 26.33it/s]\n",
      " 95%|█████████▍| 1899/2000 [01:10<00:03, 26.45it/s]\n",
      " 95%|█████████▌| 1902/2000 [01:11<00:03, 26.54it/s]\n",
      " 95%|█████████▌| 1905/2000 [01:11<00:03, 26.61it/s]\n",
      " 95%|█████████▌| 1908/2000 [01:11<00:03, 26.67it/s]\n",
      " 96%|█████████▌| 1911/2000 [01:11<00:03, 26.70it/s]\n",
      " 96%|█████████▌| 1914/2000 [01:11<00:03, 26.72it/s]\n",
      " 96%|█████████▌| 1917/2000 [01:11<00:03, 26.74it/s]\n",
      " 96%|█████████▌| 1920/2000 [01:11<00:02, 26.75it/s]\n",
      " 96%|█████████▌| 1923/2000 [01:11<00:02, 26.75it/s]\n",
      " 96%|█████████▋| 1926/2000 [01:11<00:02, 26.76it/s]\n",
      " 96%|█████████▋| 1929/2000 [01:12<00:02, 26.77it/s]\n",
      " 97%|█████████▋| 1932/2000 [01:12<00:02, 26.76it/s]\n",
      " 97%|█████████▋| 1935/2000 [01:12<00:02, 26.77it/s]\n",
      " 97%|█████████▋| 1938/2000 [01:12<00:02, 26.78it/s]\n",
      " 97%|█████████▋| 1941/2000 [01:12<00:02, 26.77it/s]\n",
      " 97%|█████████▋| 1944/2000 [01:12<00:02, 26.77it/s]\n",
      " 97%|█████████▋| 1947/2000 [01:12<00:01, 26.73it/s]\n",
      " 98%|█████████▊| 1950/2000 [01:12<00:01, 26.75it/s]\n",
      " 98%|█████████▊| 1953/2000 [01:12<00:01, 26.76it/s]\n",
      " 98%|█████████▊| 1956/2000 [01:13<00:01, 26.81it/s]\n",
      " 98%|█████████▊| 1959/2000 [01:13<00:01, 26.84it/s]\n",
      " 98%|█████████▊| 1962/2000 [01:13<00:01, 26.81it/s]\n",
      " 98%|█████████▊| 1965/2000 [01:13<00:01, 26.67it/s]\n",
      " 98%|█████████▊| 1968/2000 [01:13<00:01, 26.55it/s]\n",
      " 99%|█████████▊| 1971/2000 [01:13<00:01, 26.46it/s]\n",
      " 99%|█████████▊| 1974/2000 [01:13<00:00, 26.26it/s]\n",
      " 99%|█████████▉| 1977/2000 [01:13<00:00, 26.19it/s]\n",
      " 99%|█████████▉| 1980/2000 [01:13<00:00, 26.14it/s]\n",
      " 99%|█████████▉| 1983/2000 [01:14<00:00, 25.73it/s]\n",
      " 99%|█████████▉| 1986/2000 [01:14<00:00, 25.89it/s]\n",
      " 99%|█████████▉| 1989/2000 [01:14<00:00, 26.00it/s]\n",
      "100%|█████████▉| 1992/2000 [01:14<00:00, 26.07it/s]\n",
      "100%|█████████▉| 1995/2000 [01:14<00:00, 26.13it/s]\n",
      "100%|█████████▉| 1998/2000 [01:14<00:00, 26.16it/s]\n",
      "100%|██████████| 2000/2000 [01:14<00:00, 26.77it/s]\n",
      "\n",
      "  0%|          | 0/2000 [00:00<?, ?it/s]\n",
      "  0%|          | 3/2000 [00:00<01:35, 20.88it/s]\n",
      "  0%|          | 6/2000 [00:00<01:33, 21.30it/s]\n",
      "  0%|          | 9/2000 [00:00<01:32, 21.62it/s]\n",
      "  1%|          | 12/2000 [00:00<01:30, 21.85it/s]\n",
      "  1%|          | 15/2000 [00:00<01:30, 22.00it/s]\n",
      "  1%|          | 18/2000 [00:00<01:29, 22.11it/s]\n",
      "  1%|          | 21/2000 [00:00<01:29, 22.18it/s]\n",
      "  1%|          | 24/2000 [00:01<01:28, 22.23it/s]\n",
      "  1%|▏         | 27/2000 [00:01<01:28, 22.27it/s]\n",
      "  2%|▏         | 30/2000 [00:01<01:28, 22.29it/s]\n",
      "  2%|▏         | 33/2000 [00:01<01:28, 22.31it/s]\n",
      "  2%|▏         | 36/2000 [00:01<01:29, 22.03it/s]\n",
      "  2%|▏         | 39/2000 [00:01<01:28, 22.09it/s]\n",
      "  2%|▏         | 42/2000 [00:01<01:28, 22.14it/s]\n",
      "  2%|▏         | 45/2000 [00:02<01:28, 22.18it/s]\n",
      "  2%|▏         | 48/2000 [00:02<01:27, 22.21it/s]\n",
      "  3%|▎         | 51/2000 [00:02<01:27, 22.21it/s]\n",
      "  3%|▎         | 54/2000 [00:02<01:27, 22.22it/s]\n",
      "  3%|▎         | 57/2000 [00:02<01:27, 22.18it/s]\n",
      "  3%|▎         | 60/2000 [00:02<01:27, 22.16it/s]\n",
      "  3%|▎         | 63/2000 [00:02<01:27, 22.14it/s]\n",
      "  3%|▎         | 66/2000 [00:02<01:27, 22.13it/s]\n",
      "  3%|▎         | 69/2000 [00:03<01:27, 22.13it/s]\n",
      "  4%|▎         | 72/2000 [00:03<01:27, 22.12it/s]\n",
      "  4%|▍         | 75/2000 [00:03<01:26, 22.13it/s]\n",
      "  4%|▍         | 78/2000 [00:03<01:26, 22.13it/s]\n",
      "  4%|▍         | 81/2000 [00:03<01:26, 22.13it/s]\n",
      "  4%|▍         | 84/2000 [00:03<01:27, 21.94it/s]\n",
      "  4%|▍         | 87/2000 [00:03<01:26, 22.12it/s]\n",
      "  4%|▍         | 90/2000 [00:04<01:25, 22.31it/s]\n",
      "  5%|▍         | 93/2000 [00:04<01:24, 22.46it/s]\n",
      "  5%|▍         | 96/2000 [00:04<01:24, 22.57it/s]\n",
      "  5%|▍         | 99/2000 [00:04<01:23, 22.65it/s]\n",
      "  5%|▌         | 102/2000 [00:04<01:23, 22.71it/s]\n",
      "  5%|▌         | 105/2000 [00:04<01:23, 22.75it/s]\n",
      "  5%|▌         | 108/2000 [00:04<01:23, 22.78it/s]\n",
      "  6%|▌         | 111/2000 [00:04<01:22, 22.78it/s]\n",
      "  6%|▌         | 114/2000 [00:05<01:22, 22.80it/s]\n",
      "  6%|▌         | 117/2000 [00:05<01:22, 22.81it/s]\n",
      "  6%|▌         | 120/2000 [00:05<01:22, 22.82it/s]\n",
      "  6%|▌         | 123/2000 [00:05<01:22, 22.83it/s]\n",
      "  6%|▋         | 126/2000 [00:05<01:22, 22.83it/s]\n",
      "  6%|▋         | 129/2000 [00:05<01:22, 22.82it/s]\n",
      "  7%|▋         | 132/2000 [00:05<01:21, 22.82it/s]\n",
      "  7%|▋         | 135/2000 [00:06<01:21, 22.82it/s]\n",
      "  7%|▋         | 138/2000 [00:06<01:22, 22.61it/s]\n",
      "  7%|▋         | 141/2000 [00:06<01:22, 22.60it/s]\n",
      "  7%|▋         | 144/2000 [00:06<01:22, 22.37it/s]\n",
      "  7%|▋         | 147/2000 [00:06<01:22, 22.37it/s]\n",
      "  8%|▊         | 150/2000 [00:06<01:22, 22.48it/s]\n",
      "  8%|▊         | 153/2000 [00:06<01:21, 22.56it/s]\n",
      "  8%|▊         | 156/2000 [00:06<01:21, 22.62it/s]\n",
      "  8%|▊         | 159/2000 [00:07<01:21, 22.66it/s]\n",
      "  8%|▊         | 162/2000 [00:07<01:21, 22.68it/s]\n",
      "  8%|▊         | 165/2000 [00:07<01:20, 22.70it/s]\n",
      "  8%|▊         | 168/2000 [00:07<01:20, 22.72it/s]\n",
      "  9%|▊         | 171/2000 [00:07<01:20, 22.72it/s]\n",
      "  9%|▊         | 174/2000 [00:07<01:20, 22.73it/s]\n",
      "  9%|▉         | 177/2000 [00:07<01:20, 22.73it/s]\n",
      "  9%|▉         | 180/2000 [00:08<01:20, 22.73it/s]\n",
      "  9%|▉         | 183/2000 [00:08<01:19, 22.74it/s]\n",
      "  9%|▉         | 186/2000 [00:08<01:19, 22.74it/s]\n",
      "  9%|▉         | 189/2000 [00:08<01:19, 22.73it/s]\n",
      " 10%|▉         | 192/2000 [00:08<01:19, 22.65it/s]\n",
      " 10%|▉         | 195/2000 [00:08<01:19, 22.66it/s]\n",
      " 10%|▉         | 198/2000 [00:08<01:19, 22.66it/s]\n",
      " 10%|█         | 201/2000 [00:08<01:19, 22.68it/s]\n",
      " 10%|█         | 204/2000 [00:09<01:19, 22.69it/s]\n",
      " 10%|█         | 207/2000 [00:09<01:19, 22.45it/s]\n",
      " 10%|█         | 210/2000 [00:09<01:19, 22.55it/s]\n",
      " 11%|█         | 213/2000 [00:09<01:18, 22.63it/s]\n",
      " 11%|█         | 216/2000 [00:09<01:18, 22.69it/s]\n",
      " 11%|█         | 219/2000 [00:09<01:18, 22.57it/s]\n",
      " 11%|█         | 222/2000 [00:09<01:18, 22.59it/s]\n",
      " 11%|█▏        | 225/2000 [00:10<01:18, 22.60it/s]\n",
      " 11%|█▏        | 228/2000 [00:10<01:18, 22.61it/s]\n",
      " 12%|█▏        | 231/2000 [00:10<01:18, 22.61it/s]\n",
      " 12%|█▏        | 234/2000 [00:10<01:18, 22.61it/s]\n",
      " 12%|█▏        | 237/2000 [00:10<01:17, 22.60it/s]\n",
      " 12%|█▏        | 240/2000 [00:10<01:17, 22.60it/s]\n",
      " 12%|█▏        | 243/2000 [00:10<01:17, 22.60it/s]\n",
      " 12%|█▏        | 246/2000 [00:10<01:17, 22.60it/s]\n",
      " 12%|█▏        | 249/2000 [00:11<01:17, 22.60it/s]\n",
      " 13%|█▎        | 252/2000 [00:11<01:17, 22.59it/s]\n",
      " 13%|█▎        | 255/2000 [00:11<01:17, 22.54it/s]\n",
      " 13%|█▎        | 258/2000 [00:11<01:17, 22.54it/s]\n",
      " 13%|█▎        | 261/2000 [00:11<01:17, 22.54it/s]\n",
      " 13%|█▎        | 264/2000 [00:11<01:16, 22.55it/s]\n",
      " 13%|█▎        | 267/2000 [00:11<01:17, 22.26it/s]\n",
      " 14%|█▎        | 270/2000 [00:12<01:17, 22.39it/s]\n",
      " 14%|█▎        | 273/2000 [00:12<01:16, 22.50it/s]\n",
      " 14%|█▍        | 276/2000 [00:12<01:16, 22.58it/s]\n",
      " 14%|█▍        | 279/2000 [00:12<01:16, 22.62it/s]\n",
      " 14%|█▍        | 282/2000 [00:12<01:15, 22.64it/s]\n",
      " 14%|█▍        | 285/2000 [00:12<01:15, 22.65it/s]\n",
      " 14%|█▍        | 288/2000 [00:12<01:16, 22.40it/s]\n",
      " 15%|█▍        | 291/2000 [00:12<01:15, 22.51it/s]\n",
      " 15%|█▍        | 294/2000 [00:13<01:15, 22.58it/s]\n",
      " 15%|█▍        | 297/2000 [00:13<01:15, 22.64it/s]\n",
      " 15%|█▌        | 300/2000 [00:13<01:14, 22.68it/s]\n",
      " 15%|█▌        | 303/2000 [00:13<01:14, 22.70it/s]\n",
      " 15%|█▌        | 306/2000 [00:13<01:14, 22.72it/s]\n",
      " 15%|█▌        | 309/2000 [00:13<01:14, 22.73it/s]\n",
      " 16%|█▌        | 312/2000 [00:13<01:14, 22.72it/s]\n",
      " 16%|█▌        | 315/2000 [00:13<01:14, 22.73it/s]\n",
      " 16%|█▌        | 318/2000 [00:14<01:14, 22.71it/s]\n",
      " 16%|█▌        | 321/2000 [00:14<01:13, 22.72it/s]\n",
      " 16%|█▌        | 324/2000 [00:14<01:13, 22.72it/s]\n",
      " 16%|█▋        | 327/2000 [00:14<01:13, 22.71it/s]\n",
      " 16%|█▋        | 330/2000 [00:14<01:13, 22.67it/s]\n",
      " 17%|█▋        | 333/2000 [00:14<01:14, 22.39it/s]\n",
      " 17%|█▋        | 336/2000 [00:14<01:13, 22.54it/s]\n",
      " 17%|█▋        | 339/2000 [00:15<01:13, 22.63it/s]\n",
      " 17%|█▋        | 342/2000 [00:15<01:13, 22.69it/s]\n",
      " 17%|█▋        | 345/2000 [00:15<01:12, 22.74it/s]\n",
      " 17%|█▋        | 348/2000 [00:15<01:13, 22.48it/s]\n",
      " 18%|█▊        | 351/2000 [00:15<01:13, 22.53it/s]\n",
      " 18%|█▊        | 354/2000 [00:15<01:12, 22.57it/s]\n",
      " 18%|█▊        | 357/2000 [00:15<01:12, 22.60it/s]\n",
      " 18%|█▊        | 360/2000 [00:15<01:12, 22.61it/s]\n",
      " 18%|█▊        | 363/2000 [00:16<01:12, 22.60it/s]\n",
      " 18%|█▊        | 366/2000 [00:16<01:12, 22.60it/s]\n",
      " 18%|█▊        | 369/2000 [00:16<01:12, 22.60it/s]\n",
      " 19%|█▊        | 372/2000 [00:16<01:12, 22.60it/s]\n",
      " 19%|█▉        | 375/2000 [00:16<01:12, 22.34it/s]\n",
      " 19%|█▉        | 378/2000 [00:16<01:12, 22.51it/s]\n",
      " 19%|█▉        | 381/2000 [00:16<01:11, 22.58it/s]\n",
      " 19%|█▉        | 384/2000 [00:17<01:11, 22.58it/s]\n",
      " 19%|█▉        | 387/2000 [00:17<01:11, 22.62it/s]\n",
      " 20%|█▉        | 390/2000 [00:17<01:11, 22.65it/s]\n",
      " 20%|█▉        | 393/2000 [00:17<01:10, 22.66it/s]\n",
      " 20%|█▉        | 396/2000 [00:17<01:10, 22.67it/s]\n",
      " 20%|█▉        | 399/2000 [00:17<01:10, 22.68it/s]\n",
      " 20%|██        | 402/2000 [00:17<01:10, 22.69it/s]\n",
      " 20%|██        | 405/2000 [00:17<01:10, 22.69it/s]\n",
      " 20%|██        | 408/2000 [00:18<01:10, 22.69it/s]\n",
      " 21%|██        | 411/2000 [00:18<01:10, 22.70it/s]\n",
      " 21%|██        | 414/2000 [00:18<01:09, 22.70it/s]\n",
      " 21%|██        | 417/2000 [00:18<01:09, 22.69it/s]\n",
      " 21%|██        | 420/2000 [00:18<01:10, 22.34it/s]\n",
      " 21%|██        | 423/2000 [00:18<01:10, 22.42it/s]\n",
      " 21%|██▏       | 426/2000 [00:18<01:10, 22.48it/s]\n",
      " 21%|██▏       | 429/2000 [00:19<01:09, 22.53it/s]\n",
      " 22%|██▏       | 432/2000 [00:19<01:09, 22.57it/s]\n",
      " 22%|██▏       | 435/2000 [00:19<01:09, 22.58it/s]\n",
      " 22%|██▏       | 438/2000 [00:19<01:09, 22.59it/s]\n",
      " 22%|██▏       | 441/2000 [00:19<01:08, 22.61it/s]\n",
      " 22%|██▏       | 444/2000 [00:19<01:08, 22.55it/s]\n",
      " 22%|██▏       | 447/2000 [00:19<01:08, 22.52it/s]\n",
      " 22%|██▎       | 450/2000 [00:19<01:08, 22.54it/s]\n",
      " 23%|██▎       | 453/2000 [00:20<01:08, 22.55it/s]\n",
      " 23%|██▎       | 456/2000 [00:20<01:08, 22.55it/s]\n",
      " 23%|██▎       | 459/2000 [00:20<01:08, 22.55it/s]\n",
      " 23%|██▎       | 462/2000 [00:20<01:08, 22.56it/s]\n",
      " 23%|██▎       | 465/2000 [00:20<01:08, 22.57it/s]\n",
      " 23%|██▎       | 468/2000 [00:20<01:07, 22.55it/s]\n",
      " 24%|██▎       | 471/2000 [00:20<01:07, 22.55it/s]\n",
      " 24%|██▎       | 474/2000 [00:21<01:07, 22.55it/s]\n",
      " 24%|██▍       | 477/2000 [00:21<01:07, 22.55it/s]\n",
      " 24%|██▍       | 480/2000 [00:21<01:07, 22.55it/s]\n",
      " 24%|██▍       | 483/2000 [00:21<01:07, 22.55it/s]\n",
      " 24%|██▍       | 486/2000 [00:21<01:07, 22.55it/s]\n",
      " 24%|██▍       | 489/2000 [00:21<01:07, 22.55it/s]\n",
      " 25%|██▍       | 492/2000 [00:21<01:06, 22.54it/s]\n",
      " 25%|██▍       | 495/2000 [00:21<01:06, 22.54it/s]\n",
      " 25%|██▍       | 498/2000 [00:22<01:06, 22.54it/s]\n",
      " 25%|██▌       | 501/2000 [00:22<01:06, 22.55it/s]\n",
      " 25%|██▌       | 504/2000 [00:22<01:06, 22.54it/s]\n",
      " 25%|██▌       | 507/2000 [00:22<01:06, 22.40it/s]\n",
      " 26%|██▌       | 510/2000 [00:22<01:06, 22.44it/s]\n",
      " 26%|██▌       | 513/2000 [00:22<01:06, 22.48it/s]\n",
      " 26%|██▌       | 516/2000 [00:22<01:05, 22.50it/s]\n",
      " 26%|██▌       | 519/2000 [00:23<01:05, 22.51it/s]\n",
      " 26%|██▌       | 522/2000 [00:23<01:05, 22.53it/s]\n",
      " 26%|██▋       | 525/2000 [00:23<01:05, 22.55it/s]\n",
      " 26%|██▋       | 528/2000 [00:23<01:05, 22.56it/s]\n",
      " 27%|██▋       | 531/2000 [00:23<01:05, 22.57it/s]\n",
      " 27%|██▋       | 534/2000 [00:23<01:04, 22.57it/s]\n",
      " 27%|██▋       | 537/2000 [00:23<01:04, 22.58it/s]\n",
      " 27%|██▋       | 540/2000 [00:23<01:04, 22.58it/s]\n",
      " 27%|██▋       | 543/2000 [00:24<01:04, 22.55it/s]\n",
      " 27%|██▋       | 546/2000 [00:24<01:04, 22.56it/s]\n",
      " 27%|██▋       | 549/2000 [00:24<01:04, 22.56it/s]\n",
      " 28%|██▊       | 552/2000 [00:24<01:04, 22.57it/s]\n",
      " 28%|██▊       | 555/2000 [00:24<01:03, 22.58it/s]\n",
      " 28%|██▊       | 558/2000 [00:24<01:03, 22.64it/s]\n",
      " 28%|██▊       | 561/2000 [00:24<01:03, 22.68it/s]\n",
      " 28%|██▊       | 564/2000 [00:25<01:03, 22.70it/s]\n",
      " 28%|██▊       | 567/2000 [00:25<01:03, 22.73it/s]\n",
      " 28%|██▊       | 570/2000 [00:25<01:03, 22.65it/s]\n",
      " 29%|██▊       | 573/2000 [00:25<01:03, 22.63it/s]\n",
      " 29%|██▉       | 576/2000 [00:25<01:02, 22.61it/s]\n",
      " 29%|██▉       | 579/2000 [00:25<01:02, 22.60it/s]\n",
      " 29%|██▉       | 582/2000 [00:25<01:02, 22.59it/s]\n",
      " 29%|██▉       | 585/2000 [00:25<01:02, 22.58it/s]\n",
      " 29%|██▉       | 588/2000 [00:26<01:02, 22.58it/s]\n",
      " 30%|██▉       | 591/2000 [00:26<01:02, 22.58it/s]\n",
      " 30%|██▉       | 594/2000 [00:26<01:02, 22.57it/s]\n",
      " 30%|██▉       | 597/2000 [00:26<01:02, 22.57it/s]\n",
      " 30%|███       | 600/2000 [00:26<01:02, 22.57it/s]\n",
      " 30%|███       | 603/2000 [00:26<01:01, 22.57it/s]\n",
      " 30%|███       | 606/2000 [00:26<01:01, 22.57it/s]\n",
      " 30%|███       | 609/2000 [00:27<01:01, 22.56it/s]\n",
      " 31%|███       | 612/2000 [00:27<01:01, 22.56it/s]\n",
      " 31%|███       | 615/2000 [00:27<01:01, 22.57it/s]\n",
      " 31%|███       | 618/2000 [00:27<01:01, 22.57it/s]\n",
      " 31%|███       | 621/2000 [00:27<01:01, 22.57it/s]\n",
      " 31%|███       | 624/2000 [00:27<01:00, 22.57it/s]\n",
      " 31%|███▏      | 627/2000 [00:27<01:01, 22.19it/s]\n",
      " 32%|███▏      | 630/2000 [00:27<01:01, 22.35it/s]\n",
      " 32%|███▏      | 633/2000 [00:28<01:00, 22.47it/s]\n",
      " 32%|███▏      | 636/2000 [00:28<01:00, 22.56it/s]\n",
      " 32%|███▏      | 639/2000 [00:28<01:00, 22.62it/s]\n",
      " 32%|███▏      | 642/2000 [00:28<00:59, 22.66it/s]\n",
      " 32%|███▏      | 645/2000 [00:28<00:59, 22.69it/s]\n",
      " 32%|███▏      | 648/2000 [00:28<00:59, 22.71it/s]\n",
      " 33%|███▎      | 651/2000 [00:28<00:59, 22.72it/s]\n",
      " 33%|███▎      | 654/2000 [00:29<00:59, 22.73it/s]\n",
      " 33%|███▎      | 657/2000 [00:29<00:59, 22.73it/s]\n",
      " 33%|███▎      | 660/2000 [00:29<00:58, 22.74it/s]\n",
      " 33%|███▎      | 663/2000 [00:29<00:58, 22.73it/s]\n",
      " 33%|███▎      | 666/2000 [00:29<00:58, 22.74it/s]\n",
      " 33%|███▎      | 669/2000 [00:29<00:58, 22.75it/s]\n",
      " 34%|███▎      | 672/2000 [00:29<00:59, 22.42it/s]\n",
      " 34%|███▍      | 675/2000 [00:29<00:58, 22.49it/s]\n",
      " 34%|███▍      | 678/2000 [00:30<00:58, 22.53it/s]\n",
      " 34%|███▍      | 681/2000 [00:30<00:58, 22.56it/s]\n",
      " 34%|███▍      | 684/2000 [00:30<00:58, 22.59it/s]\n",
      " 34%|███▍      | 687/2000 [00:30<00:58, 22.59it/s]\n",
      " 34%|███▍      | 690/2000 [00:30<00:57, 22.60it/s]\n",
      " 35%|███▍      | 693/2000 [00:30<00:57, 22.61it/s]\n",
      " 35%|███▍      | 696/2000 [00:30<00:57, 22.63it/s]\n",
      " 35%|███▍      | 699/2000 [00:30<00:57, 22.63it/s]\n",
      " 35%|███▌      | 702/2000 [00:31<00:57, 22.64it/s]\n",
      " 35%|███▌      | 705/2000 [00:31<00:57, 22.65it/s]\n",
      " 35%|███▌      | 708/2000 [00:31<00:57, 22.65it/s]\n",
      " 36%|███▌      | 711/2000 [00:31<00:56, 22.66it/s]\n",
      " 36%|███▌      | 714/2000 [00:31<00:56, 22.68it/s]\n",
      " 36%|███▌      | 717/2000 [00:31<00:56, 22.71it/s]\n",
      " 36%|███▌      | 720/2000 [00:31<00:56, 22.73it/s]\n",
      " 36%|███▌      | 723/2000 [00:32<00:56, 22.71it/s]\n",
      " 36%|███▋      | 726/2000 [00:32<00:56, 22.68it/s]\n",
      " 36%|███▋      | 729/2000 [00:32<00:56, 22.65it/s]\n",
      " 37%|███▋      | 732/2000 [00:32<00:56, 22.63it/s]\n",
      " 37%|███▋      | 735/2000 [00:32<00:55, 22.62it/s]\n",
      " 37%|███▋      | 738/2000 [00:32<00:55, 22.60it/s]\n",
      " 37%|███▋      | 741/2000 [00:32<00:55, 22.59it/s]\n",
      " 37%|███▋      | 744/2000 [00:32<00:55, 22.59it/s]\n",
      " 37%|███▋      | 747/2000 [00:33<00:56, 22.35it/s]\n",
      " 38%|███▊      | 750/2000 [00:33<00:55, 22.51it/s]\n",
      " 38%|███▊      | 753/2000 [00:33<00:55, 22.63it/s]\n",
      " 38%|███▊      | 756/2000 [00:33<00:54, 22.71it/s]\n",
      " 38%|███▊      | 759/2000 [00:33<00:54, 22.77it/s]\n",
      " 38%|███▊      | 762/2000 [00:33<00:54, 22.78it/s]\n",
      " 38%|███▊      | 765/2000 [00:33<00:54, 22.76it/s]\n",
      " 38%|███▊      | 768/2000 [00:34<00:54, 22.75it/s]\n",
      " 39%|███▊      | 771/2000 [00:34<00:54, 22.74it/s]\n",
      " 39%|███▊      | 774/2000 [00:34<00:54, 22.39it/s]\n",
      " 39%|███▉      | 777/2000 [00:34<00:54, 22.48it/s]\n",
      " 39%|███▉      | 780/2000 [00:34<00:54, 22.25it/s]\n",
      " 39%|███▉      | 783/2000 [00:34<00:54, 22.41it/s]\n",
      " 39%|███▉      | 786/2000 [00:34<00:53, 22.52it/s]\n",
      " 39%|███▉      | 789/2000 [00:34<00:53, 22.60it/s]\n",
      " 40%|███▉      | 792/2000 [00:35<00:53, 22.65it/s]\n",
      " 40%|███▉      | 795/2000 [00:35<00:53, 22.69it/s]\n",
      " 40%|███▉      | 798/2000 [00:35<00:52, 22.73it/s]\n",
      " 40%|████      | 801/2000 [00:35<00:52, 22.74it/s]\n",
      " 40%|████      | 804/2000 [00:35<00:52, 22.75it/s]\n",
      " 40%|████      | 807/2000 [00:35<00:52, 22.75it/s]\n",
      " 40%|████      | 810/2000 [00:35<00:52, 22.76it/s]\n",
      " 41%|████      | 813/2000 [00:36<00:52, 22.77it/s]\n",
      " 41%|████      | 816/2000 [00:36<00:52, 22.43it/s]\n",
      " 41%|████      | 819/2000 [00:36<00:52, 22.50it/s]\n",
      " 41%|████      | 822/2000 [00:36<00:52, 22.55it/s]\n",
      " 41%|████▏     | 825/2000 [00:36<00:52, 22.59it/s]\n",
      " 41%|████▏     | 828/2000 [00:36<00:51, 22.61it/s]\n",
      " 42%|████▏     | 831/2000 [00:36<00:51, 22.64it/s]\n",
      " 42%|████▏     | 834/2000 [00:36<00:51, 22.65it/s]\n",
      " 42%|████▏     | 837/2000 [00:37<00:51, 22.66it/s]\n",
      " 42%|████▏     | 840/2000 [00:37<00:51, 22.67it/s]\n",
      " 42%|████▏     | 843/2000 [00:37<00:51, 22.67it/s]\n",
      " 42%|████▏     | 846/2000 [00:37<00:50, 22.71it/s]\n",
      " 42%|████▏     | 849/2000 [00:37<00:50, 22.75it/s]\n",
      " 43%|████▎     | 852/2000 [00:37<00:50, 22.77it/s]\n",
      " 43%|████▎     | 855/2000 [00:37<00:50, 22.78it/s]\n",
      " 43%|████▎     | 858/2000 [00:38<00:50, 22.80it/s]\n",
      " 43%|████▎     | 861/2000 [00:38<00:50, 22.40it/s]\n",
      " 43%|████▎     | 864/2000 [00:38<00:50, 22.53it/s]\n",
      " 43%|████▎     | 867/2000 [00:38<00:50, 22.63it/s]\n",
      " 44%|████▎     | 870/2000 [00:38<00:49, 22.68it/s]\n",
      " 44%|████▎     | 873/2000 [00:38<00:49, 22.72it/s]\n",
      " 44%|████▍     | 876/2000 [00:38<00:49, 22.76it/s]\n",
      " 44%|████▍     | 879/2000 [00:38<00:49, 22.81it/s]\n",
      " 44%|████▍     | 882/2000 [00:39<00:49, 22.80it/s]\n",
      " 44%|████▍     | 885/2000 [00:39<00:48, 22.77it/s]\n",
      " 44%|████▍     | 888/2000 [00:39<00:48, 22.75it/s]\n",
      " 45%|████▍     | 891/2000 [00:39<00:48, 22.75it/s]\n",
      " 45%|████▍     | 894/2000 [00:39<00:48, 22.74it/s]\n",
      " 45%|████▍     | 897/2000 [00:39<00:48, 22.74it/s]\n",
      " 45%|████▌     | 900/2000 [00:39<00:48, 22.74it/s]\n",
      " 45%|████▌     | 903/2000 [00:40<00:48, 22.73it/s]\n",
      " 45%|████▌     | 906/2000 [00:40<00:48, 22.73it/s]\n",
      " 45%|████▌     | 909/2000 [00:40<00:48, 22.73it/s]\n",
      " 46%|████▌     | 912/2000 [00:40<00:47, 22.73it/s]\n",
      " 46%|████▌     | 915/2000 [00:40<00:47, 22.73it/s]\n",
      " 46%|████▌     | 918/2000 [00:40<00:47, 22.73it/s]\n",
      " 46%|████▌     | 921/2000 [00:40<00:47, 22.73it/s]\n",
      " 46%|████▌     | 924/2000 [00:40<00:47, 22.73it/s]\n",
      " 46%|████▋     | 927/2000 [00:41<00:47, 22.67it/s]\n",
      " 46%|████▋     | 930/2000 [00:41<00:47, 22.68it/s]\n",
      " 47%|████▋     | 933/2000 [00:41<00:47, 22.69it/s]\n",
      " 47%|████▋     | 936/2000 [00:41<00:46, 22.68it/s]\n",
      " 47%|████▋     | 939/2000 [00:41<00:47, 22.42it/s]\n",
      " 47%|████▋     | 942/2000 [00:41<00:46, 22.53it/s]\n",
      " 47%|████▋     | 945/2000 [00:41<00:46, 22.61it/s]\n",
      " 47%|████▋     | 948/2000 [00:41<00:46, 22.66it/s]\n",
      " 48%|████▊     | 951/2000 [00:42<00:46, 22.69it/s]\n",
      " 48%|████▊     | 954/2000 [00:42<00:46, 22.72it/s]\n",
      " 48%|████▊     | 957/2000 [00:42<00:45, 22.74it/s]\n",
      " 48%|████▊     | 960/2000 [00:42<00:45, 22.76it/s]\n",
      " 48%|████▊     | 963/2000 [00:42<00:45, 22.76it/s]\n",
      " 48%|████▊     | 966/2000 [00:42<00:45, 22.72it/s]\n",
      " 48%|████▊     | 969/2000 [00:42<00:45, 22.69it/s]\n",
      " 49%|████▊     | 972/2000 [00:43<00:45, 22.66it/s]\n",
      " 49%|████▉     | 975/2000 [00:43<00:45, 22.64it/s]\n",
      " 49%|████▉     | 978/2000 [00:43<00:45, 22.63it/s]\n",
      " 49%|████▉     | 981/2000 [00:43<00:45, 22.45it/s]\n",
      " 49%|████▉     | 984/2000 [00:43<00:45, 22.55it/s]\n",
      " 49%|████▉     | 987/2000 [00:43<00:44, 22.56it/s]\n",
      " 50%|████▉     | 990/2000 [00:43<00:44, 22.62it/s]\n",
      " 50%|████▉     | 993/2000 [00:43<00:44, 22.65it/s]\n",
      " 50%|████▉     | 996/2000 [00:44<00:44, 22.68it/s]\n",
      " 50%|████▉     | 999/2000 [00:44<00:44, 22.70it/s]\n",
      " 50%|█████     | 1002/2000 [00:44<00:43, 22.72it/s]\n",
      " 50%|█████     | 1005/2000 [00:44<00:43, 22.72it/s]\n",
      " 50%|█████     | 1008/2000 [00:44<00:43, 22.73it/s]\n",
      " 51%|█████     | 1011/2000 [00:44<00:43, 22.72it/s]\n",
      " 51%|█████     | 1014/2000 [00:44<00:43, 22.73it/s]\n",
      " 51%|█████     | 1017/2000 [00:45<00:43, 22.73it/s]\n",
      " 51%|█████     | 1020/2000 [00:45<00:43, 22.73it/s]\n",
      " 51%|█████     | 1023/2000 [00:45<00:42, 22.73it/s]\n",
      " 51%|█████▏    | 1026/2000 [00:45<00:42, 22.77it/s]\n",
      " 51%|█████▏    | 1029/2000 [00:45<00:42, 22.80it/s]\n",
      " 52%|█████▏    | 1032/2000 [00:45<00:42, 22.81it/s]\n",
      " 52%|█████▏    | 1035/2000 [00:45<00:42, 22.82it/s]\n",
      " 52%|█████▏    | 1038/2000 [00:45<00:42, 22.84it/s]\n",
      " 52%|█████▏    | 1041/2000 [00:46<00:41, 22.84it/s]\n",
      " 52%|█████▏    | 1044/2000 [00:46<00:41, 22.81it/s]\n",
      " 52%|█████▏    | 1047/2000 [00:46<00:42, 22.65it/s]\n",
      " 52%|█████▎    | 1050/2000 [00:46<00:41, 22.64it/s]\n",
      " 53%|█████▎    | 1053/2000 [00:46<00:41, 22.64it/s]\n",
      " 53%|█████▎    | 1056/2000 [00:46<00:41, 22.64it/s]\n",
      " 53%|█████▎    | 1059/2000 [00:46<00:41, 22.64it/s]\n",
      " 53%|█████▎    | 1062/2000 [00:47<00:41, 22.65it/s]\n",
      " 53%|█████▎    | 1065/2000 [00:47<00:41, 22.64it/s]\n",
      " 53%|█████▎    | 1068/2000 [00:47<00:41, 22.65it/s]\n",
      " 54%|█████▎    | 1071/2000 [00:47<00:41, 22.65it/s]\n",
      " 54%|█████▎    | 1074/2000 [00:47<00:40, 22.65it/s]\n",
      " 54%|█████▍    | 1077/2000 [00:47<00:40, 22.64it/s]\n",
      " 54%|█████▍    | 1080/2000 [00:47<00:40, 22.65it/s]\n",
      " 54%|█████▍    | 1083/2000 [00:47<00:40, 22.67it/s]\n",
      " 54%|█████▍    | 1086/2000 [00:48<00:40, 22.68it/s]\n",
      " 54%|█████▍    | 1089/2000 [00:48<00:40, 22.70it/s]\n",
      " 55%|█████▍    | 1092/2000 [00:48<00:40, 22.70it/s]\n",
      " 55%|█████▍    | 1095/2000 [00:48<00:39, 22.71it/s]\n",
      " 55%|█████▍    | 1098/2000 [00:48<00:39, 22.71it/s]\n",
      " 55%|█████▌    | 1101/2000 [00:48<00:40, 22.33it/s]\n",
      " 55%|█████▌    | 1104/2000 [00:48<00:39, 22.43it/s]\n",
      " 55%|█████▌    | 1107/2000 [00:49<00:39, 22.45it/s]\n",
      " 56%|█████▌    | 1110/2000 [00:49<00:39, 22.45it/s]\n",
      " 56%|█████▌    | 1113/2000 [00:49<00:39, 22.51it/s]\n",
      " 56%|█████▌    | 1116/2000 [00:49<00:39, 22.56it/s]\n",
      " 56%|█████▌    | 1119/2000 [00:49<00:39, 22.58it/s]\n",
      " 56%|█████▌    | 1122/2000 [00:49<00:38, 22.61it/s]\n",
      " 56%|█████▋    | 1125/2000 [00:49<00:38, 22.64it/s]\n",
      " 56%|█████▋    | 1128/2000 [00:49<00:38, 22.64it/s]\n",
      " 57%|█████▋    | 1131/2000 [00:50<00:38, 22.64it/s]\n",
      " 57%|█████▋    | 1134/2000 [00:50<00:38, 22.65it/s]\n",
      " 57%|█████▋    | 1137/2000 [00:50<00:38, 22.65it/s]\n",
      " 57%|█████▋    | 1140/2000 [00:50<00:37, 22.65it/s]\n",
      " 57%|█████▋    | 1143/2000 [00:50<00:37, 22.66it/s]\n",
      " 57%|█████▋    | 1146/2000 [00:50<00:37, 22.66it/s]\n",
      " 57%|█████▋    | 1149/2000 [00:50<00:37, 22.66it/s]\n",
      " 58%|█████▊    | 1152/2000 [00:50<00:37, 22.67it/s]\n",
      " 58%|█████▊    | 1155/2000 [00:51<00:37, 22.67it/s]\n",
      " 58%|█████▊    | 1158/2000 [00:51<00:37, 22.67it/s]\n",
      " 58%|█████▊    | 1161/2000 [00:51<00:36, 22.69it/s]\n",
      " 58%|█████▊    | 1164/2000 [00:51<00:36, 22.68it/s]\n",
      " 58%|█████▊    | 1167/2000 [00:51<00:36, 22.67it/s]\n",
      " 58%|█████▊    | 1170/2000 [00:51<00:36, 22.58it/s]\n",
      " 59%|█████▊    | 1173/2000 [00:51<00:36, 22.60it/s]\n",
      " 59%|█████▉    | 1176/2000 [00:52<00:36, 22.62it/s]\n",
      " 59%|█████▉    | 1179/2000 [00:52<00:36, 22.66it/s]\n",
      " 59%|█████▉    | 1182/2000 [00:52<00:36, 22.70it/s]\n",
      " 59%|█████▉    | 1185/2000 [00:52<00:35, 22.73it/s]\n",
      " 59%|█████▉    | 1188/2000 [00:52<00:35, 22.75it/s]\n",
      " 60%|█████▉    | 1191/2000 [00:52<00:35, 22.76it/s]\n",
      " 60%|█████▉    | 1194/2000 [00:52<00:35, 22.78it/s]\n",
      " 60%|█████▉    | 1197/2000 [00:52<00:35, 22.58it/s]\n",
      " 60%|██████    | 1200/2000 [00:53<00:35, 22.63it/s]\n",
      " 60%|██████    | 1203/2000 [00:53<00:35, 22.67it/s]\n",
      " 60%|██████    | 1206/2000 [00:53<00:34, 22.71it/s]\n",
      " 60%|██████    | 1209/2000 [00:53<00:34, 22.74it/s]\n",
      " 61%|██████    | 1212/2000 [00:53<00:34, 22.75it/s]\n",
      " 61%|██████    | 1215/2000 [00:53<00:34, 22.79it/s]\n",
      " 61%|██████    | 1218/2000 [00:53<00:34, 22.83it/s]\n",
      " 61%|██████    | 1221/2000 [00:54<00:34, 22.86it/s]\n",
      " 61%|██████    | 1224/2000 [00:54<00:33, 22.85it/s]\n",
      " 61%|██████▏   | 1227/2000 [00:54<00:33, 22.82it/s]\n",
      " 62%|██████▏   | 1230/2000 [00:54<00:33, 22.73it/s]\n",
      " 62%|██████▏   | 1233/2000 [00:54<00:33, 22.73it/s]\n",
      " 62%|██████▏   | 1236/2000 [00:54<00:33, 22.72it/s]\n",
      " 62%|██████▏   | 1239/2000 [00:54<00:33, 22.71it/s]\n",
      " 62%|██████▏   | 1242/2000 [00:54<00:33, 22.71it/s]\n",
      " 62%|██████▏   | 1245/2000 [00:55<00:33, 22.71it/s]\n",
      " 62%|██████▏   | 1248/2000 [00:55<00:33, 22.71it/s]\n",
      " 63%|██████▎   | 1251/2000 [00:55<00:32, 22.72it/s]\n",
      " 63%|██████▎   | 1254/2000 [00:55<00:32, 22.71it/s]\n",
      " 63%|██████▎   | 1257/2000 [00:55<00:32, 22.71it/s]\n",
      " 63%|██████▎   | 1260/2000 [00:55<00:32, 22.71it/s]\n",
      " 63%|██████▎   | 1263/2000 [00:55<00:32, 22.71it/s]\n",
      " 63%|██████▎   | 1266/2000 [00:56<00:32, 22.71it/s]\n",
      " 63%|██████▎   | 1269/2000 [00:56<00:32, 22.71it/s]\n",
      " 64%|██████▎   | 1272/2000 [00:56<00:32, 22.71it/s]\n",
      " 64%|██████▍   | 1275/2000 [00:56<00:31, 22.71it/s]\n",
      " 64%|██████▍   | 1278/2000 [00:56<00:31, 22.71it/s]\n",
      " 64%|██████▍   | 1281/2000 [00:56<00:31, 22.71it/s]\n",
      " 64%|██████▍   | 1284/2000 [00:56<00:31, 22.72it/s]\n",
      " 64%|██████▍   | 1287/2000 [00:56<00:31, 22.72it/s]\n",
      " 64%|██████▍   | 1290/2000 [00:57<00:31, 22.69it/s]\n",
      " 65%|██████▍   | 1293/2000 [00:57<00:31, 22.70it/s]\n",
      " 65%|██████▍   | 1296/2000 [00:57<00:31, 22.70it/s]\n",
      " 65%|██████▍   | 1299/2000 [00:57<00:30, 22.71it/s]\n",
      " 65%|██████▌   | 1302/2000 [00:57<00:30, 22.70it/s]\n",
      " 65%|██████▌   | 1305/2000 [00:57<00:30, 22.70it/s]\n",
      " 65%|██████▌   | 1308/2000 [00:57<00:30, 22.71it/s]\n",
      " 66%|██████▌   | 1311/2000 [00:57<00:30, 22.71it/s]\n",
      " 66%|██████▌   | 1314/2000 [00:58<00:30, 22.70it/s]\n",
      " 66%|██████▌   | 1317/2000 [00:58<00:30, 22.71it/s]\n",
      " 66%|██████▌   | 1320/2000 [00:58<00:29, 22.71it/s]\n",
      " 66%|██████▌   | 1323/2000 [00:58<00:29, 22.71it/s]\n",
      " 66%|██████▋   | 1326/2000 [00:58<00:29, 22.71it/s]\n",
      " 66%|██████▋   | 1329/2000 [00:58<00:29, 22.71it/s]\n",
      " 67%|██████▋   | 1332/2000 [00:58<00:29, 22.70it/s]\n",
      " 67%|██████▋   | 1335/2000 [00:59<00:29, 22.70it/s]\n",
      " 67%|██████▋   | 1338/2000 [00:59<00:29, 22.72it/s]\n",
      " 67%|██████▋   | 1341/2000 [00:59<00:29, 22.72it/s]\n",
      " 67%|██████▋   | 1344/2000 [00:59<00:28, 22.71it/s]\n",
      " 67%|██████▋   | 1347/2000 [00:59<00:28, 22.70it/s]\n",
      " 68%|██████▊   | 1350/2000 [00:59<00:28, 22.66it/s]\n",
      " 68%|██████▊   | 1353/2000 [00:59<00:28, 22.67it/s]\n",
      " 68%|██████▊   | 1356/2000 [00:59<00:28, 22.68it/s]\n",
      " 68%|██████▊   | 1359/2000 [01:00<00:28, 22.30it/s]\n",
      " 68%|██████▊   | 1362/2000 [01:00<00:28, 22.40it/s]\n",
      " 68%|██████▊   | 1365/2000 [01:00<00:28, 22.47it/s]\n",
      " 68%|██████▊   | 1368/2000 [01:00<00:28, 22.51it/s]\n",
      " 69%|██████▊   | 1371/2000 [01:00<00:27, 22.55it/s]\n",
      " 69%|██████▊   | 1374/2000 [01:00<00:27, 22.59it/s]\n",
      " 69%|██████▉   | 1377/2000 [01:00<00:27, 22.61it/s]\n",
      " 69%|██████▉   | 1380/2000 [01:01<00:27, 22.62it/s]\n",
      " 69%|██████▉   | 1383/2000 [01:01<00:27, 22.62it/s]\n",
      " 69%|██████▉   | 1386/2000 [01:01<00:27, 22.62it/s]\n",
      " 69%|██████▉   | 1389/2000 [01:01<00:27, 22.61it/s]\n",
      " 70%|██████▉   | 1392/2000 [01:01<00:26, 22.60it/s]\n",
      " 70%|██████▉   | 1395/2000 [01:01<00:26, 22.44it/s]\n",
      " 70%|██████▉   | 1398/2000 [01:01<00:26, 22.57it/s]\n",
      " 70%|███████   | 1401/2000 [01:01<00:26, 22.67it/s]\n",
      " 70%|███████   | 1404/2000 [01:02<00:26, 22.74it/s]\n",
      " 70%|███████   | 1407/2000 [01:02<00:26, 22.79it/s]\n",
      " 70%|███████   | 1410/2000 [01:02<00:25, 22.69it/s]\n",
      " 71%|███████   | 1413/2000 [01:02<00:25, 22.70it/s]\n",
      " 71%|███████   | 1416/2000 [01:02<00:25, 22.70it/s]\n",
      " 71%|███████   | 1419/2000 [01:02<00:25, 22.70it/s]\n",
      " 71%|███████   | 1422/2000 [01:02<00:25, 22.71it/s]\n",
      " 71%|███████▏  | 1425/2000 [01:03<00:25, 22.70it/s]\n",
      " 71%|███████▏  | 1428/2000 [01:03<00:25, 22.69it/s]\n",
      " 72%|███████▏  | 1431/2000 [01:03<00:25, 22.69it/s]\n",
      " 72%|███████▏  | 1434/2000 [01:03<00:24, 22.69it/s]\n",
      " 72%|███████▏  | 1437/2000 [01:03<00:24, 22.70it/s]\n",
      " 72%|███████▏  | 1440/2000 [01:03<00:24, 22.70it/s]\n",
      " 72%|███████▏  | 1443/2000 [01:03<00:24, 22.70it/s]\n",
      " 72%|███████▏  | 1446/2000 [01:03<00:24, 22.39it/s]\n",
      " 72%|███████▏  | 1449/2000 [01:04<00:24, 22.47it/s]\n",
      " 73%|███████▎  | 1452/2000 [01:04<00:24, 22.52it/s]\n",
      " 73%|███████▎  | 1455/2000 [01:04<00:24, 22.57it/s]\n",
      " 73%|███████▎  | 1458/2000 [01:04<00:23, 22.60it/s]\n",
      " 73%|███████▎  | 1461/2000 [01:04<00:23, 22.65it/s]\n",
      " 73%|███████▎  | 1464/2000 [01:04<00:23, 22.69it/s]\n",
      " 73%|███████▎  | 1467/2000 [01:04<00:23, 22.71it/s]\n",
      " 74%|███████▎  | 1470/2000 [01:05<00:23, 22.72it/s]\n",
      " 74%|███████▎  | 1473/2000 [01:05<00:23, 22.75it/s]\n",
      " 74%|███████▍  | 1476/2000 [01:05<00:23, 22.76it/s]\n",
      " 74%|███████▍  | 1479/2000 [01:05<00:22, 22.77it/s]\n",
      " 74%|███████▍  | 1482/2000 [01:05<00:22, 22.80it/s]\n",
      " 74%|███████▍  | 1485/2000 [01:05<00:22, 22.80it/s]\n",
      " 74%|███████▍  | 1488/2000 [01:05<00:22, 22.81it/s]\n",
      " 75%|███████▍  | 1491/2000 [01:05<00:22, 22.82it/s]\n",
      " 75%|███████▍  | 1494/2000 [01:06<00:22, 22.82it/s]\n",
      " 75%|███████▍  | 1497/2000 [01:06<00:22, 22.82it/s]\n",
      " 75%|███████▌  | 1500/2000 [01:06<00:21, 22.82it/s]\n",
      " 75%|███████▌  | 1503/2000 [01:06<00:21, 22.80it/s]\n",
      " 75%|███████▌  | 1506/2000 [01:06<00:21, 22.49it/s]\n",
      " 75%|███████▌  | 1509/2000 [01:06<00:21, 22.61it/s]\n",
      " 76%|███████▌  | 1512/2000 [01:06<00:21, 22.69it/s]\n",
      " 76%|███████▌  | 1515/2000 [01:06<00:21, 22.74it/s]\n",
      " 76%|███████▌  | 1518/2000 [01:07<00:21, 22.55it/s]\n",
      " 76%|███████▌  | 1521/2000 [01:07<00:21, 22.62it/s]\n",
      " 76%|███████▌  | 1524/2000 [01:07<00:21, 22.66it/s]\n",
      " 76%|███████▋  | 1527/2000 [01:07<00:20, 22.65it/s]\n",
      " 76%|███████▋  | 1530/2000 [01:07<00:20, 22.58it/s]\n",
      " 77%|███████▋  | 1533/2000 [01:07<00:20, 22.56it/s]\n",
      " 77%|███████▋  | 1536/2000 [01:07<00:20, 22.56it/s]\n",
      " 77%|███████▋  | 1539/2000 [01:08<00:20, 22.26it/s]\n",
      " 77%|███████▋  | 1542/2000 [01:08<00:20, 22.46it/s]\n",
      " 77%|███████▋  | 1545/2000 [01:08<00:20, 22.57it/s]\n",
      " 77%|███████▋  | 1548/2000 [01:08<00:19, 22.65it/s]\n",
      " 78%|███████▊  | 1551/2000 [01:08<00:19, 22.73it/s]\n",
      " 78%|███████▊  | 1554/2000 [01:08<00:19, 22.73it/s]\n",
      " 78%|███████▊  | 1557/2000 [01:08<00:19, 22.71it/s]\n",
      " 78%|███████▊  | 1560/2000 [01:08<00:19, 22.70it/s]\n",
      " 78%|███████▊  | 1563/2000 [01:09<00:19, 22.69it/s]\n",
      " 78%|███████▊  | 1566/2000 [01:09<00:19, 22.69it/s]\n",
      " 78%|███████▊  | 1569/2000 [01:09<00:19, 22.68it/s]\n",
      " 79%|███████▊  | 1572/2000 [01:09<00:18, 22.68it/s]\n",
      " 79%|███████▉  | 1575/2000 [01:09<00:18, 22.67it/s]\n",
      " 79%|███████▉  | 1578/2000 [01:09<00:18, 22.67it/s]\n",
      " 79%|███████▉  | 1581/2000 [01:09<00:18, 22.67it/s]\n",
      " 79%|███████▉  | 1584/2000 [01:10<00:18, 22.67it/s]\n",
      " 79%|███████▉  | 1587/2000 [01:10<00:18, 22.68it/s]\n",
      " 80%|███████▉  | 1590/2000 [01:10<00:18, 22.64it/s]\n",
      " 80%|███████▉  | 1593/2000 [01:10<00:17, 22.62it/s]\n",
      " 80%|███████▉  | 1596/2000 [01:10<00:17, 22.62it/s]\n",
      " 80%|███████▉  | 1599/2000 [01:10<00:17, 22.61it/s]\n",
      " 80%|████████  | 1602/2000 [01:10<00:17, 22.63it/s]\n",
      " 80%|████████  | 1605/2000 [01:10<00:17, 22.69it/s]\n",
      " 80%|████████  | 1608/2000 [01:11<00:17, 22.71it/s]\n",
      " 81%|████████  | 1611/2000 [01:11<00:17, 22.67it/s]\n",
      " 81%|████████  | 1614/2000 [01:11<00:17, 22.66it/s]\n",
      " 81%|████████  | 1617/2000 [01:11<00:16, 22.69it/s]\n",
      " 81%|████████  | 1620/2000 [01:11<00:16, 22.72it/s]\n",
      " 81%|████████  | 1623/2000 [01:11<00:16, 22.68it/s]\n",
      " 81%|████████▏ | 1626/2000 [01:11<00:16, 22.66it/s]\n",
      " 81%|████████▏ | 1629/2000 [01:12<00:16, 22.64it/s]\n",
      " 82%|████████▏ | 1632/2000 [01:12<00:16, 22.63it/s]\n",
      " 82%|████████▏ | 1635/2000 [01:12<00:16, 22.61it/s]\n",
      " 82%|████████▏ | 1638/2000 [01:12<00:16, 22.61it/s]\n",
      " 82%|████████▏ | 1641/2000 [01:12<00:15, 22.62it/s]\n",
      " 82%|████████▏ | 1644/2000 [01:12<00:15, 22.30it/s]\n",
      " 82%|████████▏ | 1647/2000 [01:12<00:15, 22.36it/s]\n",
      " 82%|████████▎ | 1650/2000 [01:12<00:15, 22.37it/s]\n",
      " 83%|████████▎ | 1653/2000 [01:13<00:15, 22.40it/s]\n",
      " 83%|████████▎ | 1656/2000 [01:13<00:15, 22.42it/s]\n",
      " 83%|████████▎ | 1659/2000 [01:13<00:15, 22.12it/s]\n",
      " 83%|████████▎ | 1662/2000 [01:13<00:15, 22.27it/s]\n",
      " 83%|████████▎ | 1665/2000 [01:13<00:14, 22.40it/s]\n",
      " 83%|████████▎ | 1668/2000 [01:13<00:14, 22.48it/s]\n",
      " 84%|████████▎ | 1671/2000 [01:13<00:14, 22.53it/s]\n",
      " 84%|████████▎ | 1674/2000 [01:14<00:14, 22.57it/s]\n",
      " 84%|████████▍ | 1677/2000 [01:14<00:14, 22.58it/s]\n",
      " 84%|████████▍ | 1680/2000 [01:14<00:14, 22.62it/s]\n",
      " 84%|████████▍ | 1683/2000 [01:14<00:14, 22.62it/s]\n",
      " 84%|████████▍ | 1686/2000 [01:14<00:13, 22.65it/s]\n",
      " 84%|████████▍ | 1689/2000 [01:14<00:13, 22.66it/s]\n",
      " 85%|████████▍ | 1692/2000 [01:14<00:13, 22.67it/s]\n",
      " 85%|████████▍ | 1695/2000 [01:14<00:13, 22.69it/s]\n",
      " 85%|████████▍ | 1698/2000 [01:15<00:13, 22.68it/s]\n",
      " 85%|████████▌ | 1701/2000 [01:15<00:13, 22.65it/s]\n",
      " 85%|████████▌ | 1704/2000 [01:15<00:13, 22.64it/s]\n",
      " 85%|████████▌ | 1707/2000 [01:15<00:12, 22.64it/s]\n",
      " 86%|████████▌ | 1710/2000 [01:15<00:12, 22.58it/s]\n",
      " 86%|████████▌ | 1713/2000 [01:15<00:12, 22.55it/s]\n",
      " 86%|████████▌ | 1716/2000 [01:15<00:12, 22.57it/s]\n",
      " 86%|████████▌ | 1719/2000 [01:16<00:12, 22.59it/s]\n",
      " 86%|████████▌ | 1722/2000 [01:16<00:12, 22.61it/s]\n",
      " 86%|████████▋ | 1725/2000 [01:16<00:12, 22.62it/s]\n",
      " 86%|████████▋ | 1728/2000 [01:16<00:12, 22.60it/s]\n",
      " 87%|████████▋ | 1731/2000 [01:16<00:12, 22.25it/s]\n",
      " 87%|████████▋ | 1734/2000 [01:16<00:11, 22.35it/s]\n",
      " 87%|████████▋ | 1737/2000 [01:16<00:11, 22.41it/s]\n",
      " 87%|████████▋ | 1740/2000 [01:16<00:11, 22.46it/s]\n",
      " 87%|████████▋ | 1743/2000 [01:17<00:11, 22.49it/s]\n",
      " 87%|████████▋ | 1746/2000 [01:17<00:11, 22.51it/s]\n",
      " 87%|████████▋ | 1749/2000 [01:17<00:11, 22.53it/s]\n",
      " 88%|████████▊ | 1752/2000 [01:17<00:11, 22.54it/s]\n",
      " 88%|████████▊ | 1755/2000 [01:17<00:10, 22.55it/s]\n",
      " 88%|████████▊ | 1758/2000 [01:17<00:10, 22.55it/s]\n",
      " 88%|████████▊ | 1761/2000 [01:17<00:10, 22.56it/s]\n",
      " 88%|████████▊ | 1764/2000 [01:18<00:10, 22.56it/s]\n",
      " 88%|████████▊ | 1767/2000 [01:18<00:10, 22.57it/s]\n",
      " 88%|████████▊ | 1770/2000 [01:18<00:10, 22.52it/s]\n",
      " 89%|████████▊ | 1773/2000 [01:18<00:10, 22.49it/s]\n",
      " 89%|████████▉ | 1776/2000 [01:18<00:09, 22.52it/s]\n",
      " 89%|████████▉ | 1779/2000 [01:18<00:09, 22.53it/s]\n",
      " 89%|████████▉ | 1782/2000 [01:18<00:09, 22.53it/s]\n",
      " 89%|████████▉ | 1785/2000 [01:18<00:09, 22.55it/s]\n",
      " 89%|████████▉ | 1788/2000 [01:19<00:09, 22.55it/s]\n",
      " 90%|████████▉ | 1791/2000 [01:19<00:09, 22.55it/s]\n",
      " 90%|████████▉ | 1794/2000 [01:19<00:09, 22.55it/s]\n",
      " 90%|████████▉ | 1797/2000 [01:19<00:09, 22.55it/s]\n",
      " 90%|█████████ | 1800/2000 [01:19<00:08, 22.55it/s]\n",
      " 90%|█████████ | 1803/2000 [01:19<00:08, 22.26it/s]\n",
      " 90%|█████████ | 1806/2000 [01:19<00:08, 22.40it/s]\n",
      " 90%|█████████ | 1809/2000 [01:20<00:08, 22.48it/s]\n",
      " 91%|█████████ | 1812/2000 [01:20<00:08, 22.54it/s]\n",
      " 91%|█████████ | 1815/2000 [01:20<00:08, 22.59it/s]\n",
      " 91%|█████████ | 1818/2000 [01:20<00:08, 22.61it/s]\n",
      " 91%|█████████ | 1821/2000 [01:20<00:07, 22.64it/s]\n",
      " 91%|█████████ | 1824/2000 [01:20<00:07, 22.65it/s]\n",
      " 91%|█████████▏| 1827/2000 [01:20<00:07, 22.66it/s]\n",
      " 92%|█████████▏| 1830/2000 [01:20<00:07, 22.64it/s]\n",
      " 92%|█████████▏| 1833/2000 [01:21<00:07, 22.62it/s]\n",
      " 92%|█████████▏| 1836/2000 [01:21<00:07, 22.64it/s]\n",
      " 92%|█████████▏| 1839/2000 [01:21<00:07, 22.66it/s]\n",
      " 92%|█████████▏| 1842/2000 [01:21<00:06, 22.66it/s]\n",
      " 92%|█████████▏| 1845/2000 [01:21<00:06, 22.66it/s]\n",
      " 92%|█████████▏| 1848/2000 [01:21<00:06, 22.68it/s]\n",
      " 93%|█████████▎| 1851/2000 [01:21<00:06, 22.67it/s]\n",
      " 93%|█████████▎| 1854/2000 [01:22<00:06, 22.67it/s]\n",
      " 93%|█████████▎| 1857/2000 [01:22<00:06, 22.66it/s]\n",
      " 93%|█████████▎| 1860/2000 [01:22<00:06, 22.64it/s]\n",
      " 93%|█████████▎| 1863/2000 [01:22<00:06, 22.34it/s]\n",
      " 93%|█████████▎| 1866/2000 [01:22<00:05, 22.42it/s]\n",
      " 93%|█████████▎| 1869/2000 [01:22<00:05, 22.43it/s]\n",
      " 94%|█████████▎| 1872/2000 [01:22<00:05, 22.45it/s]\n",
      " 94%|█████████▍| 1875/2000 [01:22<00:05, 22.45it/s]\n",
      " 94%|█████████▍| 1878/2000 [01:23<00:05, 22.44it/s]\n",
      " 94%|█████████▍| 1881/2000 [01:23<00:05, 22.45it/s]\n",
      " 94%|█████████▍| 1884/2000 [01:23<00:05, 22.47it/s]\n",
      " 94%|█████████▍| 1887/2000 [01:23<00:05, 22.47it/s]\n",
      " 94%|█████████▍| 1890/2000 [01:23<00:04, 22.47it/s]\n",
      " 95%|█████████▍| 1893/2000 [01:23<00:04, 22.47it/s]\n",
      " 95%|█████████▍| 1896/2000 [01:23<00:04, 22.47it/s]\n",
      " 95%|█████████▍| 1899/2000 [01:24<00:04, 22.47it/s]\n",
      " 95%|█████████▌| 1902/2000 [01:24<00:04, 22.48it/s]\n",
      " 95%|█████████▌| 1905/2000 [01:24<00:04, 22.48it/s]\n",
      " 95%|█████████▌| 1908/2000 [01:24<00:04, 22.47it/s]\n",
      " 96%|█████████▌| 1911/2000 [01:24<00:03, 22.49it/s]\n",
      " 96%|█████████▌| 1914/2000 [01:24<00:03, 22.49it/s]\n",
      " 96%|█████████▌| 1917/2000 [01:24<00:03, 22.48it/s]\n",
      " 96%|█████████▌| 1920/2000 [01:24<00:03, 22.47it/s]\n",
      " 96%|█████████▌| 1923/2000 [01:25<00:03, 22.47it/s]\n",
      " 96%|█████████▋| 1926/2000 [01:25<00:03, 22.48it/s]\n",
      " 96%|█████████▋| 1929/2000 [01:25<00:03, 22.48it/s]\n",
      " 97%|█████████▋| 1932/2000 [01:25<00:03, 22.48it/s]\n",
      " 97%|█████████▋| 1935/2000 [01:25<00:02, 22.49it/s]\n",
      " 97%|█████████▋| 1938/2000 [01:25<00:02, 22.21it/s]\n",
      " 97%|█████████▋| 1941/2000 [01:25<00:02, 22.34it/s]\n",
      " 97%|█████████▋| 1944/2000 [01:26<00:02, 22.44it/s]\n",
      " 97%|█████████▋| 1947/2000 [01:26<00:02, 22.51it/s]\n",
      " 98%|█████████▊| 1950/2000 [01:26<00:02, 22.56it/s]\n",
      " 98%|█████████▊| 1953/2000 [01:26<00:02, 22.59it/s]\n",
      " 98%|█████████▊| 1956/2000 [01:26<00:01, 22.60it/s]\n",
      " 98%|█████████▊| 1959/2000 [01:26<00:01, 22.60it/s]\n",
      " 98%|█████████▊| 1962/2000 [01:26<00:01, 22.59it/s]\n",
      " 98%|█████████▊| 1965/2000 [01:26<00:01, 22.60it/s]\n",
      " 98%|█████████▊| 1968/2000 [01:27<00:01, 22.60it/s]\n",
      " 99%|█████████▊| 1971/2000 [01:27<00:01, 22.65it/s]\n",
      " 99%|█████████▊| 1974/2000 [01:27<00:01, 22.70it/s]\n",
      " 99%|█████████▉| 1977/2000 [01:27<00:01, 22.71it/s]\n",
      " 99%|█████████▉| 1980/2000 [01:27<00:00, 22.72it/s]\n",
      " 99%|█████████▉| 1983/2000 [01:27<00:00, 22.73it/s]\n",
      " 99%|█████████▉| 1986/2000 [01:27<00:00, 22.74it/s]\n",
      " 99%|█████████▉| 1989/2000 [01:28<00:00, 22.75it/s]\n",
      "100%|█████████▉| 1992/2000 [01:28<00:00, 22.75it/s]\n",
      "100%|█████████▉| 1995/2000 [01:28<00:00, 22.46it/s]\n",
      "100%|█████████▉| 1998/2000 [01:28<00:00, 22.52it/s]\n",
      "100%|██████████| 2000/2000 [01:28<00:00, 22.60it/s]\n",
      "\n",
      "sfashionmnist, seed 6\n",
      "Learning task 1\n",
      "Mean accuracy (test): 0.9885\n",
      "Accuracies (test): [0.9885]\n",
      "\n",
      "Learning task 2\n",
      "Mean accuracy (test): 0.9763\n",
      "Accuracies (test): [0.981, 0.9715]\n",
      "\n",
      "Learning task 3\n",
      "Mean accuracy (test): 0.9757\n",
      "Accuracies (test): [0.973, 0.954, 1.0]\n",
      "\n",
      "Learning task 4\n",
      "Mean accuracy (test): 0.9792\n",
      "Accuracies (test): [0.9735, 0.944, 0.9995, 1.0]\n",
      "\n",
      "Learning task 5\n",
      "Mean accuracy (test): 0.9729\n",
      "Accuracies (test): [0.967, 0.9205, 0.983, 0.9965, 0.9975]\n",
      "\n",
      "\n",
      "Test accuracy: mean = 0.9729, std = 0.0000\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9728999999999999\n"
     ]
    }
   ],
   "source": [
    "logdir = read_config_and_run(\"frcl_with_coreset.pkl\", task_sequence, \"frcl\")\n",
    "exp = lutils.read_exp(logdir)\n",
    "show_final_average_accuracy(exp, runner=\"frcl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FROMP (with lambda-descend coreset) <a name=\"res5\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading experiments: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 5623.80it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading from cache:\n",
      "Running on oat18.cs.ox.ac.uk\n",
      "Running with: python /auto/users/timner/qixuan/function-space-variational-inference/fsvi_cl/baselines/fromp/run_fromp.py --dataset sfashionmnist --n_tasks 5 --batch_size 128 --hidden_size 256 --n_layers 2 --lr 0.0001 --n_epochs 15 --seed 6 --n_seeds 1 --n_points 40 --select_method lambda_descend --tau 10.0 --n_permuted_tasks 10 --smnist_eps 1e-06 --logroot ablation --subdir reproduce_main_results_3 --save_alt --n_coreset_inputs_per_task 40 --n_steps not_specified\n",
      "\n",
      "sfashionmnist, seed 6\n",
      "start working on task 0\n",
      "Test accuracies, task 1: mean = 0.9910, all = [0.991]\n",
      "nb classes 2\n",
      "memorable points appended!\n",
      "updated fisher!\n",
      "start working on task 1\n",
      "Test accuracies, task 2: mean = 0.9798, all = [0.987  0.9725]\n",
      "nb classes 2\n",
      "memorable points appended!\n",
      "updated fisher!\n",
      "start working on task 2\n",
      "Test accuracies, task 3: mean = 0.9878, all = [0.9905 0.973  1.    ]\n",
      "nb classes 2\n",
      "memorable points appended!\n",
      "updated fisher!\n",
      "start working on task 3\n",
      "Test accuracies, task 4: mean = 0.9898, all = [0.987 0.973 0.999 1.   ]\n",
      "nb classes 2\n",
      "memorable points appended!\n",
      "updated fisher!\n",
      "start working on task 4\n",
      "Test accuracies, task 5: mean = 0.9908, all = [0.9855 0.9715 0.999  1.     0.998 ]\n",
      "nb classes 2\n",
      "memorable points appended!\n",
      "updated fisher!\n",
      "\n",
      "Test accuracy: mean = 0.9908, std = 0.0000\n",
      "\n",
      "0.9907999999999999\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "logdir = read_config_and_run(\"fromp_with_coreset.pkl\", task_sequence, runner=\"fromp\")\n",
    "exp = lutils.read_exp(logdir)\n",
    "show_final_average_accuracy(exp, runner=\"fromp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VCL (with random-choice coreset) <a name=\"res6\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading experiments: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 5731.39it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading from cache:\n",
      "Running on oat2.cs.ox.ac.uk\n",
      "Running with: python /auto/users/timner/qixuan/function-space-variational-inference/fsvi_cl/baselines/vcl/run_vcl.py --dataset sfashionmnist --n_epochs 100 --batch_size 256 --hidden_size 256 --n_layers 2 --seed 4 --select_method random_choice --n_permuted_tasks 10 --logroot ablation --subdir reproduce_main_results_3 --n_coreset_inputs_per_task 40\n",
      "----------------------------------------------------------------------------------------------------\n",
      "('Epoch:', '0001', 'cost=', '0.202752212')\n",
      "('Epoch:', '0006', 'cost=', '0.015417452')\n",
      "('Epoch:', '0011', 'cost=', '0.004580267')\n",
      "('Epoch:', '0016', 'cost=', '0.014853463')\n",
      "('Epoch:', '0021', 'cost=', '0.001920307')\n",
      "('Epoch:', '0026', 'cost=', '0.000228825')\n",
      "('Epoch:', '0031', 'cost=', '0.000083474')\n",
      "('Epoch:', '0036', 'cost=', '0.000043977')\n",
      "('Epoch:', '0041', 'cost=', '0.000025435')\n",
      "('Epoch:', '0046', 'cost=', '0.000015340')\n",
      "('Epoch:', '0051', 'cost=', '0.000010124')\n",
      "('Epoch:', '0056', 'cost=', '0.000006785')\n",
      "('Epoch:', '0061', 'cost=', '0.000004772')\n",
      "('Epoch:', '0066', 'cost=', '0.000003573')\n",
      "('Epoch:', '0071', 'cost=', '0.000002731')\n",
      "('Epoch:', '0076', 'cost=', '0.000002091')\n",
      "('Epoch:', '0081', 'cost=', '0.000001642')\n",
      "('Epoch:', '0086', 'cost=', '0.000001326')\n",
      "('Epoch:', '0091', 'cost=', '0.000001088')\n",
      "('Epoch:', '0096', 'cost=', '0.000000882')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '56.184821839')\n",
      "('Epoch:', '0006', 'cost=', '53.534708794')\n",
      "('Epoch:', '0011', 'cost=', '50.911155944')\n",
      "('Epoch:', '0016', 'cost=', '48.294008864')\n",
      "('Epoch:', '0021', 'cost=', '45.681344945')\n",
      "('Epoch:', '0026', 'cost=', '43.076598553')\n",
      "('Epoch:', '0031', 'cost=', '40.483142772')\n",
      "('Epoch:', '0036', 'cost=', '37.903589127')\n",
      "('Epoch:', '0041', 'cost=', '35.335034310')\n",
      "('Epoch:', '0046', 'cost=', '32.789699067')\n",
      "('Epoch:', '0051', 'cost=', '30.270914362')\n",
      "('Epoch:', '0056', 'cost=', '27.784961822')\n",
      "('Epoch:', '0061', 'cost=', '25.339220818')\n",
      "('Epoch:', '0066', 'cost=', '22.943319361')\n",
      "('Epoch:', '0071', 'cost=', '20.613127242')\n",
      "('Epoch:', '0076', 'cost=', '18.349826285')\n",
      "('Epoch:', '0081', 'cost=', '16.171614383')\n",
      "('Epoch:', '0086', 'cost=', '14.100884539')\n",
      "('Epoch:', '0091', 'cost=', '12.147916611')\n",
      "('Epoch:', '0096', 'cost=', '10.326407027')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '160.954055786')\n",
      "('Epoch:', '0006', 'cost=', '160.795974731')\n",
      "('Epoch:', '0011', 'cost=', '160.592514038')\n",
      "('Epoch:', '0016', 'cost=', '160.434600830')\n",
      "('Epoch:', '0021', 'cost=', '160.236328125')\n",
      "('Epoch:', '0026', 'cost=', '160.072418213')\n",
      "('Epoch:', '0031', 'cost=', '159.889160156')\n",
      "('Epoch:', '0036', 'cost=', '159.734375000')\n",
      "('Epoch:', '0041', 'cost=', '159.541778564')\n",
      "('Epoch:', '0046', 'cost=', '159.406585693')\n",
      "('Epoch:', '0051', 'cost=', '159.210327148')\n",
      "('Epoch:', '0056', 'cost=', '159.043090820')\n",
      "('Epoch:', '0061', 'cost=', '158.895919800')\n",
      "('Epoch:', '0066', 'cost=', '158.699966431')\n",
      "('Epoch:', '0071', 'cost=', '158.536346436')\n",
      "('Epoch:', '0076', 'cost=', '158.363204956')\n",
      "('Epoch:', '0081', 'cost=', '158.200073242')\n",
      "('Epoch:', '0086', 'cost=', '158.050643921')\n",
      "('Epoch:', '0091', 'cost=', '157.929290771')\n",
      "('Epoch:', '0096', 'cost=', '157.718612671')\n",
      "Optimization Finished!\n",
      "Accuracy at task 0: [0.9945]\n",
      "('Epoch:', '0001', 'cost=', '2.072543370')\n",
      "('Epoch:', '0006', 'cost=', '0.733051646')\n",
      "('Epoch:', '0011', 'cost=', '0.686631209')\n",
      "('Epoch:', '0016', 'cost=', '0.659649386')\n",
      "('Epoch:', '0021', 'cost=', '0.641772989')\n",
      "('Epoch:', '0026', 'cost=', '0.621017116')\n",
      "('Epoch:', '0031', 'cost=', '0.597505762')\n",
      "('Epoch:', '0036', 'cost=', '0.580306458')\n",
      "('Epoch:', '0041', 'cost=', '0.563490677')\n",
      "('Epoch:', '0046', 'cost=', '0.543954565')\n",
      "('Epoch:', '0051', 'cost=', '0.523847863')\n",
      "('Epoch:', '0056', 'cost=', '0.504199100')\n",
      "('Epoch:', '0061', 'cost=', '0.487608108')\n",
      "('Epoch:', '0066', 'cost=', '0.467590019')\n",
      "('Epoch:', '0071', 'cost=', '0.448969274')\n",
      "('Epoch:', '0076', 'cost=', '0.427723602')\n",
      "('Epoch:', '0081', 'cost=', '0.411105115')\n",
      "('Epoch:', '0086', 'cost=', '0.392379801')\n",
      "('Epoch:', '0091', 'cost=', '0.373563733')\n",
      "('Epoch:', '0096', 'cost=', '0.355503083')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '161.398468018')\n",
      "('Epoch:', '0006', 'cost=', '160.960906982')\n",
      "('Epoch:', '0011', 'cost=', '160.721923828')\n",
      "('Epoch:', '0016', 'cost=', '160.484893799')\n",
      "('Epoch:', '0021', 'cost=', '160.297119141')\n",
      "('Epoch:', '0026', 'cost=', '160.108734131')\n",
      "('Epoch:', '0031', 'cost=', '159.976989746')\n",
      "('Epoch:', '0036', 'cost=', '159.804428101')\n",
      "('Epoch:', '0041', 'cost=', '159.654144287')\n",
      "('Epoch:', '0046', 'cost=', '159.423171997')\n",
      "('Epoch:', '0051', 'cost=', '159.311538696')\n",
      "('Epoch:', '0056', 'cost=', '159.094589233')\n",
      "('Epoch:', '0061', 'cost=', '158.902114868')\n",
      "('Epoch:', '0066', 'cost=', '158.798278809')\n",
      "('Epoch:', '0071', 'cost=', '158.625442505')\n",
      "('Epoch:', '0076', 'cost=', '158.412979126')\n",
      "('Epoch:', '0081', 'cost=', '158.306686401')\n",
      "('Epoch:', '0086', 'cost=', '158.082778931')\n",
      "('Epoch:', '0091', 'cost=', '157.935653687')\n",
      "('Epoch:', '0096', 'cost=', '157.784851074')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '161.151062012')\n",
      "('Epoch:', '0006', 'cost=', '160.942535400')\n",
      "('Epoch:', '0011', 'cost=', '160.778991699')\n",
      "('Epoch:', '0016', 'cost=', '160.545578003')\n",
      "('Epoch:', '0021', 'cost=', '160.401657104')\n",
      "('Epoch:', '0026', 'cost=', '160.219863892')\n",
      "('Epoch:', '0031', 'cost=', '160.004379272')\n",
      "('Epoch:', '0036', 'cost=', '159.866607666')\n",
      "('Epoch:', '0041', 'cost=', '159.667617798')\n",
      "('Epoch:', '0046', 'cost=', '159.497741699')\n",
      "('Epoch:', '0051', 'cost=', '159.309768677')\n",
      "('Epoch:', '0056', 'cost=', '159.149963379')\n",
      "('Epoch:', '0061', 'cost=', '158.993713379')\n",
      "('Epoch:', '0066', 'cost=', '158.851699829')\n",
      "('Epoch:', '0071', 'cost=', '158.680740356')\n",
      "('Epoch:', '0076', 'cost=', '158.517395020')\n",
      "('Epoch:', '0081', 'cost=', '158.359725952')\n",
      "('Epoch:', '0086', 'cost=', '158.168533325')\n",
      "('Epoch:', '0091', 'cost=', '157.998184204')\n",
      "('Epoch:', '0096', 'cost=', '157.840515137')\n",
      "Optimization Finished!\n",
      "Accuracy at task 1: [0.985, 0.968]\n",
      "('Epoch:', '0001', 'cost=', '1.884539216')\n",
      "('Epoch:', '0006', 'cost=', '0.573649290')\n",
      "('Epoch:', '0011', 'cost=', '0.556841337')\n",
      "('Epoch:', '0016', 'cost=', '0.544758837')\n",
      "('Epoch:', '0021', 'cost=', '0.526697708')\n",
      "('Epoch:', '0026', 'cost=', '0.510440121')\n",
      "('Epoch:', '0031', 'cost=', '0.495839212')\n",
      "('Epoch:', '0036', 'cost=', '0.478731153')\n",
      "('Epoch:', '0041', 'cost=', '0.462552236')\n",
      "('Epoch:', '0046', 'cost=', '0.446602788')\n",
      "('Epoch:', '0051', 'cost=', '0.428635735')\n",
      "('Epoch:', '0056', 'cost=', '0.410530676')\n",
      "('Epoch:', '0061', 'cost=', '0.392047866')\n",
      "('Epoch:', '0066', 'cost=', '0.372969728')\n",
      "('Epoch:', '0071', 'cost=', '0.354356875')\n",
      "('Epoch:', '0076', 'cost=', '0.335938265')\n",
      "('Epoch:', '0081', 'cost=', '0.318578546')\n",
      "('Epoch:', '0086', 'cost=', '0.299864136')\n",
      "('Epoch:', '0091', 'cost=', '0.283532722')\n",
      "('Epoch:', '0096', 'cost=', '0.265470302')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '168.167922974')\n",
      "('Epoch:', '0006', 'cost=', '167.277206421')\n",
      "('Epoch:', '0011', 'cost=', '164.029434204')\n",
      "('Epoch:', '0016', 'cost=', '161.144241333')\n",
      "('Epoch:', '0021', 'cost=', '160.746826172')\n",
      "('Epoch:', '0026', 'cost=', '160.709747314')\n",
      "('Epoch:', '0031', 'cost=', '160.771545410')\n",
      "('Epoch:', '0036', 'cost=', '160.480712891')\n",
      "('Epoch:', '0041', 'cost=', '160.182861328')\n",
      "('Epoch:', '0046', 'cost=', '160.377212524')\n",
      "('Epoch:', '0051', 'cost=', '159.812484741')\n",
      "('Epoch:', '0056', 'cost=', '159.637786865')\n",
      "('Epoch:', '0061', 'cost=', '159.712280273')\n",
      "('Epoch:', '0066', 'cost=', '159.551010132')\n",
      "('Epoch:', '0071', 'cost=', '159.133728027')\n",
      "('Epoch:', '0076', 'cost=', '158.820556641')\n",
      "('Epoch:', '0081', 'cost=', '158.807388306')\n",
      "('Epoch:', '0086', 'cost=', '158.591537476')\n",
      "('Epoch:', '0091', 'cost=', '158.409851074')\n",
      "('Epoch:', '0096', 'cost=', '158.246734619')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '161.737503052')\n",
      "('Epoch:', '0006', 'cost=', '161.148773193')\n",
      "('Epoch:', '0011', 'cost=', '160.848114014')\n",
      "('Epoch:', '0016', 'cost=', '160.940704346')\n",
      "('Epoch:', '0021', 'cost=', '160.486160278')\n",
      "('Epoch:', '0026', 'cost=', '160.714538574')\n",
      "('Epoch:', '0031', 'cost=', '160.163253784')\n",
      "('Epoch:', '0036', 'cost=', '159.948028564')\n",
      "('Epoch:', '0041', 'cost=', '159.959182739')\n",
      "('Epoch:', '0046', 'cost=', '159.727142334')\n",
      "('Epoch:', '0051', 'cost=', '159.570373535')\n",
      "('Epoch:', '0056', 'cost=', '159.293258667')\n",
      "('Epoch:', '0061', 'cost=', '159.195220947')\n",
      "('Epoch:', '0066', 'cost=', '159.022384644')\n",
      "('Epoch:', '0071', 'cost=', '158.889984131')\n",
      "('Epoch:', '0076', 'cost=', '158.625762939')\n",
      "('Epoch:', '0081', 'cost=', '158.437454224')\n",
      "('Epoch:', '0086', 'cost=', '158.288894653')\n",
      "('Epoch:', '0091', 'cost=', '158.092971802')\n",
      "('Epoch:', '0096', 'cost=', '157.928955078')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '160.948348999')\n",
      "('Epoch:', '0006', 'cost=', '160.768951416')\n",
      "('Epoch:', '0011', 'cost=', '160.588729858')\n",
      "('Epoch:', '0016', 'cost=', '160.410400391')\n",
      "('Epoch:', '0021', 'cost=', '160.230789185')\n",
      "('Epoch:', '0026', 'cost=', '160.072662354')\n",
      "('Epoch:', '0031', 'cost=', '159.880737305')\n",
      "('Epoch:', '0036', 'cost=', '159.707901001')\n",
      "('Epoch:', '0041', 'cost=', '159.543334961')\n",
      "('Epoch:', '0046', 'cost=', '159.366058350')\n",
      "('Epoch:', '0051', 'cost=', '159.195831299')\n",
      "('Epoch:', '0056', 'cost=', '159.027053833')\n",
      "('Epoch:', '0061', 'cost=', '158.868682861')\n",
      "('Epoch:', '0066', 'cost=', '158.691497803')\n",
      "('Epoch:', '0071', 'cost=', '158.524871826')\n",
      "('Epoch:', '0076', 'cost=', '158.357620239')\n",
      "('Epoch:', '0081', 'cost=', '158.193481445')\n",
      "('Epoch:', '0086', 'cost=', '158.027481079')\n",
      "('Epoch:', '0091', 'cost=', '157.864913940')\n",
      "('Epoch:', '0096', 'cost=', '157.704269409')\n",
      "Optimization Finished!\n",
      "Accuracy at task 2: [0.976, 0.966, 0.9995]\n",
      "('Epoch:', '0001', 'cost=', '1.635972240')\n",
      "('Epoch:', '0006', 'cost=', '0.555447830')\n",
      "('Epoch:', '0011', 'cost=', '0.544552831')\n",
      "('Epoch:', '0016', 'cost=', '0.533529497')\n",
      "('Epoch:', '0021', 'cost=', '0.522873334')\n",
      "('Epoch:', '0026', 'cost=', '0.512742391')\n",
      "('Epoch:', '0031', 'cost=', '0.500399912')\n",
      "('Epoch:', '0036', 'cost=', '0.487210539')\n",
      "('Epoch:', '0041', 'cost=', '0.477128776')\n",
      "('Epoch:', '0046', 'cost=', '0.460109489')\n",
      "('Epoch:', '0051', 'cost=', '0.446370697')\n",
      "('Epoch:', '0056', 'cost=', '0.431737571')\n",
      "('Epoch:', '0061', 'cost=', '0.418145508')\n",
      "('Epoch:', '0066', 'cost=', '0.401808130')\n",
      "('Epoch:', '0071', 'cost=', '0.384894325')\n",
      "('Epoch:', '0076', 'cost=', '0.369097338')\n",
      "('Epoch:', '0081', 'cost=', '0.353145057')\n",
      "('Epoch:', '0086', 'cost=', '0.336413563')\n",
      "('Epoch:', '0091', 'cost=', '0.319881519')\n",
      "('Epoch:', '0096', 'cost=', '0.303449181')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '192.488143921')\n",
      "('Epoch:', '0006', 'cost=', '184.887771606')\n",
      "('Epoch:', '0011', 'cost=', '181.706695557')\n",
      "('Epoch:', '0016', 'cost=', '168.689407349')\n",
      "('Epoch:', '0021', 'cost=', '163.502349854')\n",
      "('Epoch:', '0026', 'cost=', '162.477539062')\n",
      "('Epoch:', '0031', 'cost=', '161.854293823')\n",
      "('Epoch:', '0036', 'cost=', '161.170822144')\n",
      "('Epoch:', '0041', 'cost=', '161.132812500')\n",
      "('Epoch:', '0046', 'cost=', '160.818527222')\n",
      "('Epoch:', '0051', 'cost=', '161.156509399')\n",
      "('Epoch:', '0056', 'cost=', '160.585861206')\n",
      "('Epoch:', '0061', 'cost=', '160.179306030')\n",
      "('Epoch:', '0066', 'cost=', '160.230911255')\n",
      "('Epoch:', '0071', 'cost=', '159.863647461')\n",
      "('Epoch:', '0076', 'cost=', '159.791580200')\n",
      "('Epoch:', '0081', 'cost=', '159.582489014')\n",
      "('Epoch:', '0086', 'cost=', '159.273651123')\n",
      "('Epoch:', '0091', 'cost=', '159.091918945')\n",
      "('Epoch:', '0096', 'cost=', '158.852127075')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '162.120742798')\n",
      "('Epoch:', '0006', 'cost=', '161.715164185')\n",
      "('Epoch:', '0011', 'cost=', '161.360595703')\n",
      "('Epoch:', '0016', 'cost=', '161.620223999')\n",
      "('Epoch:', '0021', 'cost=', '161.221893311')\n",
      "('Epoch:', '0026', 'cost=', '160.433731079')\n",
      "('Epoch:', '0031', 'cost=', '160.763504028')\n",
      "('Epoch:', '0036', 'cost=', '160.345153809')\n",
      "('Epoch:', '0041', 'cost=', '160.394302368')\n",
      "('Epoch:', '0046', 'cost=', '159.889694214')\n",
      "('Epoch:', '0051', 'cost=', '159.722946167')\n",
      "('Epoch:', '0056', 'cost=', '159.562576294')\n",
      "('Epoch:', '0061', 'cost=', '159.295959473')\n",
      "('Epoch:', '0066', 'cost=', '159.122604370')\n",
      "('Epoch:', '0071', 'cost=', '159.115982056')\n",
      "('Epoch:', '0076', 'cost=', '158.812301636')\n",
      "('Epoch:', '0081', 'cost=', '158.756729126')\n",
      "('Epoch:', '0086', 'cost=', '158.440338135')\n",
      "('Epoch:', '0091', 'cost=', '158.644271851')\n",
      "('Epoch:', '0096', 'cost=', '158.122146606')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '160.955551147')\n",
      "('Epoch:', '0006', 'cost=', '160.798400879')\n",
      "('Epoch:', '0011', 'cost=', '160.594543457')\n",
      "('Epoch:', '0016', 'cost=', '160.413711548')\n",
      "('Epoch:', '0021', 'cost=', '160.244430542')\n",
      "('Epoch:', '0026', 'cost=', '160.060043335')\n",
      "('Epoch:', '0031', 'cost=', '159.886352539')\n",
      "('Epoch:', '0036', 'cost=', '159.710723877')\n",
      "('Epoch:', '0041', 'cost=', '159.539886475')\n",
      "('Epoch:', '0046', 'cost=', '159.370498657')\n",
      "('Epoch:', '0051', 'cost=', '159.199645996')\n",
      "('Epoch:', '0056', 'cost=', '159.031295776')\n",
      "('Epoch:', '0061', 'cost=', '158.861053467')\n",
      "('Epoch:', '0066', 'cost=', '158.693115234')\n",
      "('Epoch:', '0071', 'cost=', '158.526275635')\n",
      "('Epoch:', '0076', 'cost=', '158.359115601')\n",
      "('Epoch:', '0081', 'cost=', '158.194717407')\n",
      "('Epoch:', '0086', 'cost=', '158.032714844')\n",
      "('Epoch:', '0091', 'cost=', '157.874511719')\n",
      "('Epoch:', '0096', 'cost=', '157.715042114')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '160.977218628')\n",
      "('Epoch:', '0006', 'cost=', '160.775390625')\n",
      "('Epoch:', '0011', 'cost=', '160.593139648')\n",
      "('Epoch:', '0016', 'cost=', '160.413162231')\n",
      "('Epoch:', '0021', 'cost=', '160.234649658')\n",
      "('Epoch:', '0026', 'cost=', '160.057830811')\n",
      "('Epoch:', '0031', 'cost=', '159.883117676')\n",
      "('Epoch:', '0036', 'cost=', '159.710037231')\n",
      "('Epoch:', '0041', 'cost=', '159.537734985')\n",
      "('Epoch:', '0046', 'cost=', '159.371292114')\n",
      "('Epoch:', '0051', 'cost=', '159.197143555')\n",
      "('Epoch:', '0056', 'cost=', '159.027374268')\n",
      "('Epoch:', '0061', 'cost=', '158.858993530')\n",
      "('Epoch:', '0066', 'cost=', '158.691085815')\n",
      "('Epoch:', '0071', 'cost=', '158.524047852')\n",
      "('Epoch:', '0076', 'cost=', '158.357727051')\n",
      "('Epoch:', '0081', 'cost=', '158.192550659')\n",
      "('Epoch:', '0086', 'cost=', '158.029800415')\n",
      "('Epoch:', '0091', 'cost=', '157.864379883')\n",
      "('Epoch:', '0096', 'cost=', '157.699325562')\n",
      "Optimization Finished!\n",
      "Accuracy at task 3: [0.9785, 0.963, 0.997, 1.0]\n",
      "('Epoch:', '0001', 'cost=', '2.859116387')\n",
      "('Epoch:', '0006', 'cost=', '0.597798488')\n",
      "('Epoch:', '0011', 'cost=', '0.573705498')\n",
      "('Epoch:', '0016', 'cost=', '0.560661954')\n",
      "('Epoch:', '0021', 'cost=', '0.550166311')\n",
      "('Epoch:', '0026', 'cost=', '0.537351154')\n",
      "('Epoch:', '0031', 'cost=', '0.526946502')\n",
      "('Epoch:', '0036', 'cost=', '0.513053179')\n",
      "('Epoch:', '0041', 'cost=', '0.500027571')\n",
      "('Epoch:', '0046', 'cost=', '0.484343063')\n",
      "('Epoch:', '0051', 'cost=', '0.474514548')\n",
      "('Epoch:', '0056', 'cost=', '0.454767500')\n",
      "('Epoch:', '0061', 'cost=', '0.439556146')\n",
      "('Epoch:', '0066', 'cost=', '0.421388226')\n",
      "('Epoch:', '0071', 'cost=', '0.407251293')\n",
      "('Epoch:', '0076', 'cost=', '0.390355138')\n",
      "('Epoch:', '0081', 'cost=', '0.374078077')\n",
      "('Epoch:', '0086', 'cost=', '0.357684707')\n",
      "('Epoch:', '0091', 'cost=', '0.339686089')\n",
      "('Epoch:', '0096', 'cost=', '0.326846717')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '194.804244995')\n",
      "('Epoch:', '0006', 'cost=', '185.365081787')\n",
      "('Epoch:', '0011', 'cost=', '180.707901001')\n",
      "('Epoch:', '0016', 'cost=', '172.330657959')\n",
      "('Epoch:', '0021', 'cost=', '166.779693604')\n",
      "('Epoch:', '0026', 'cost=', '164.237167358')\n",
      "('Epoch:', '0031', 'cost=', '162.232955933')\n",
      "('Epoch:', '0036', 'cost=', '161.609390259')\n",
      "('Epoch:', '0041', 'cost=', '161.523574829')\n",
      "('Epoch:', '0046', 'cost=', '162.141326904')\n",
      "('Epoch:', '0051', 'cost=', '160.996795654')\n",
      "('Epoch:', '0056', 'cost=', '160.981185913')\n",
      "('Epoch:', '0061', 'cost=', '160.557403564')\n",
      "('Epoch:', '0066', 'cost=', '160.399307251')\n",
      "('Epoch:', '0071', 'cost=', '160.071395874')\n",
      "('Epoch:', '0076', 'cost=', '160.352874756')\n",
      "('Epoch:', '0081', 'cost=', '159.728027344')\n",
      "('Epoch:', '0086', 'cost=', '159.599533081')\n",
      "('Epoch:', '0091', 'cost=', '159.586776733')\n",
      "('Epoch:', '0096', 'cost=', '159.400360107')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '161.906753540')\n",
      "('Epoch:', '0006', 'cost=', '161.126358032')\n",
      "('Epoch:', '0011', 'cost=', '161.238433838')\n",
      "('Epoch:', '0016', 'cost=', '160.987777710')\n",
      "('Epoch:', '0021', 'cost=', '160.842880249')\n",
      "('Epoch:', '0026', 'cost=', '160.960937500')\n",
      "('Epoch:', '0031', 'cost=', '160.461288452')\n",
      "('Epoch:', '0036', 'cost=', '160.508728027')\n",
      "('Epoch:', '0041', 'cost=', '160.033187866')\n",
      "('Epoch:', '0046', 'cost=', '159.871719360')\n",
      "('Epoch:', '0051', 'cost=', '160.745452881')\n",
      "('Epoch:', '0056', 'cost=', '159.564239502')\n",
      "('Epoch:', '0061', 'cost=', '159.390823364')\n",
      "('Epoch:', '0066', 'cost=', '159.279769897')\n",
      "('Epoch:', '0071', 'cost=', '159.049331665')\n",
      "('Epoch:', '0076', 'cost=', '159.102920532')\n",
      "('Epoch:', '0081', 'cost=', '158.599716187')\n",
      "('Epoch:', '0086', 'cost=', '158.403671265')\n",
      "('Epoch:', '0091', 'cost=', '158.257965088')\n",
      "('Epoch:', '0096', 'cost=', '158.508193970')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '161.209075928')\n",
      "('Epoch:', '0006', 'cost=', '160.887130737')\n",
      "('Epoch:', '0011', 'cost=', '160.687149048')\n",
      "('Epoch:', '0016', 'cost=', '160.494857788')\n",
      "('Epoch:', '0021', 'cost=', '160.316314697')\n",
      "('Epoch:', '0026', 'cost=', '160.166351318')\n",
      "('Epoch:', '0031', 'cost=', '159.996765137')\n",
      "('Epoch:', '0036', 'cost=', '159.814819336')\n",
      "('Epoch:', '0041', 'cost=', '159.605239868')\n",
      "('Epoch:', '0046', 'cost=', '159.431549072')\n",
      "('Epoch:', '0051', 'cost=', '159.252212524')\n",
      "('Epoch:', '0056', 'cost=', '159.107177734')\n",
      "('Epoch:', '0061', 'cost=', '159.017639160')\n",
      "('Epoch:', '0066', 'cost=', '158.752639771')\n",
      "('Epoch:', '0071', 'cost=', '158.595046997')\n",
      "('Epoch:', '0076', 'cost=', '158.432785034')\n",
      "('Epoch:', '0081', 'cost=', '158.246917725')\n",
      "('Epoch:', '0086', 'cost=', '158.078857422')\n",
      "('Epoch:', '0091', 'cost=', '157.945281982')\n",
      "('Epoch:', '0096', 'cost=', '157.770767212')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '161.882141113')\n",
      "('Epoch:', '0006', 'cost=', '160.853210449')\n",
      "('Epoch:', '0011', 'cost=', '160.842208862')\n",
      "('Epoch:', '0016', 'cost=', '160.650024414')\n",
      "('Epoch:', '0021', 'cost=', '160.355987549')\n",
      "('Epoch:', '0026', 'cost=', '160.164627075')\n",
      "('Epoch:', '0031', 'cost=', '160.010940552')\n",
      "('Epoch:', '0036', 'cost=', '159.803848267')\n",
      "('Epoch:', '0041', 'cost=', '159.603683472')\n",
      "('Epoch:', '0046', 'cost=', '159.470092773')\n",
      "('Epoch:', '0051', 'cost=', '159.277114868')\n",
      "('Epoch:', '0056', 'cost=', '159.087203979')\n",
      "('Epoch:', '0061', 'cost=', '158.945388794')\n",
      "('Epoch:', '0066', 'cost=', '158.777603149')\n",
      "('Epoch:', '0071', 'cost=', '158.637710571')\n",
      "('Epoch:', '0076', 'cost=', '158.418411255')\n",
      "('Epoch:', '0081', 'cost=', '158.251052856')\n",
      "('Epoch:', '0086', 'cost=', '158.127593994')\n",
      "('Epoch:', '0091', 'cost=', '157.920654297')\n",
      "('Epoch:', '0096', 'cost=', '157.777862549')\n",
      "Optimization Finished!\n",
      "('Epoch:', '0001', 'cost=', '160.949966431')\n",
      "('Epoch:', '0006', 'cost=', '160.772872925')\n",
      "('Epoch:', '0011', 'cost=', '160.590591431')\n",
      "('Epoch:', '0016', 'cost=', '160.414138794')\n",
      "('Epoch:', '0021', 'cost=', '160.232788086')\n",
      "('Epoch:', '0026', 'cost=', '160.055664062')\n",
      "('Epoch:', '0031', 'cost=', '159.880249023')\n",
      "('Epoch:', '0036', 'cost=', '159.709701538')\n",
      "('Epoch:', '0041', 'cost=', '159.536666870')\n",
      "('Epoch:', '0046', 'cost=', '159.363464355')\n",
      "('Epoch:', '0051', 'cost=', '159.193603516')\n",
      "('Epoch:', '0056', 'cost=', '159.024185181')\n",
      "('Epoch:', '0061', 'cost=', '158.859619141')\n",
      "('Epoch:', '0066', 'cost=', '158.688385010')\n",
      "('Epoch:', '0071', 'cost=', '158.521697998')\n",
      "('Epoch:', '0076', 'cost=', '158.356887817')\n",
      "('Epoch:', '0081', 'cost=', '158.191757202')\n",
      "('Epoch:', '0086', 'cost=', '158.030578613')\n",
      "('Epoch:', '0091', 'cost=', '157.861389160')\n",
      "('Epoch:', '0096', 'cost=', '157.696731567')\n",
      "Optimization Finished!\n",
      "Accuracy at task 4: [0.9755, 0.9665, 0.997, 1.0, 0.9975]\n",
      "[[0.9945    nan    nan    nan    nan]\n",
      " [0.985  0.968     nan    nan    nan]\n",
      " [0.976  0.966  0.9995    nan    nan]\n",
      " [0.9785 0.963  0.997  1.        nan]\n",
      " [0.9755 0.9665 0.997  1.     0.9975]]\n",
      "\n",
      "0.9873000000000001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "logdir = read_config_and_run(\"vcl_random_coreset.pkl\", task_sequence, runner=\"vcl\")\n",
    "exp = lutils.read_exp(logdir)\n",
    "show_final_average_accuracy(exp, runner=\"vcl\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fsvi",
   "language": "python",
   "name": "fsvi"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}