Approach1_BaselineModel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Baseline Model Implementation - CS598 Deep Learning\n",
    "\n",
    "## Drug Discovery: Variational Autoencoder Techniques for Molecule Generation\n",
    "\n",
    "## Approach 1: Baseline Model\n",
    "\n",
    "Team Members:\n",
    "\n",
    "- Andrew Jacobson jonaj2@illinois.edu\n",
    "- Dixon Liang dixonl2@illinois.edu\n",
    "- John Judge jmjudge2@illinois.edu\n",
    "- Megan Masanz mjneuman@illinois.edu\n",
    "\n",
    "Implementation Description:\n",
    "\n",
    "- Baseline Model\n",
    "- Character Based Chemical VAE\n",
    "- Aspuru-Guzik \n",
    "\n",
    "References include\n",
    "* [Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules](https://arxiv.org/abs/1610.02415)\n",
    "* https://github.com/aspuru-guzik-group/chemical_vae\n",
    "* https://github.com/deepchem/deepchem\n",
    "* https://github.com/molecularsets/moses (Leveraged in Notebook #3, but included here for completeness)\n",
    "* https://github.com/Azure/azureml-examples/blob/main/tutorials/an-introduction/2.pytorch-model.ipynb\n",
    "\n",
    "Requirements for notebook to run:\n",
    "* Run this notebook inside an AzureML workspace (or provide configuration)\n",
    "* No data is required as the training script will download the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Below is importing the requirements for this notebook to run inside an Azure ML environment for :\n",
    "- connecting to a workspace\n",
    "- creating remote computer for training "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml.core #adding core - this by default is in notebooks run on computer in Azure ML\n",
    "from azureml.core import Workspace #needed for connecting to workspace\n",
    "from azureml.core.compute import AmlCompute, ComputeTarget\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "from azureml.core.runconfig import RunConfiguration, DEFAULT_GPU_IMAGE \n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "from azureml.core import Environment\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "\n",
    "\n",
    "ws = Workspace.from_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating compute cluster.  The Azure ML SDK will look for compute with the provided name, if it doesn't exist, it will create it.  Standard code found in Azure ML notebook examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "found existing: mmdsvm04d\n"
     ]
    }
   ],
   "source": [
    "cluster_name = 'mmdsvm04d'\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws,  name=cluster_name )\n",
    "    print('found existing:', compute_target.name)\n",
    "    \n",
    "except ComputeTargetException:\n",
    "    print('creating new.')\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size='STANDARD_NC12',\n",
    "        min_nodes=0,\n",
    "        max_nodes=1)\n",
    "    \n",
    "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "    compute_target.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Below are the required packaged for running an Azure ML Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"databricks\": {\n",
       "        \"eggLibraries\": [],\n",
       "        \"jarLibraries\": [],\n",
       "        \"mavenLibraries\": [],\n",
       "        \"pypiLibraries\": [],\n",
       "        \"rcranLibraries\": []\n",
       "    },\n",
       "    \"docker\": {\n",
       "        \"arguments\": [],\n",
       "        \"baseDockerfile\": null,\n",
       "        \"baseImage\": \"mcr.microsoft.com/azureml/intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04:20210113.v1\",\n",
       "        \"baseImageRegistry\": {\n",
       "            \"address\": null,\n",
       "            \"password\": null,\n",
       "            \"registryIdentity\": null,\n",
       "            \"username\": null\n",
       "        },\n",
       "        \"enabled\": true,\n",
       "        \"platform\": {\n",
       "            \"architecture\": \"amd64\",\n",
       "            \"os\": \"Linux\"\n",
       "        },\n",
       "        \"sharedVolumes\": true,\n",
       "        \"shmSize\": null\n",
       "    },\n",
       "    \"environmentVariables\": {\n",
       "        \"EXAMPLE_ENV_VAR\": \"EXAMPLE_VALUE\"\n",
       "    },\n",
       "    \"inferencingStackVersion\": null,\n",
       "    \"name\": \"deepchem_backend2\",\n",
       "    \"python\": {\n",
       "        \"baseCondaEnvironment\": null,\n",
       "        \"condaDependencies\": {\n",
       "            \"channels\": [\n",
       "                \"anaconda\",\n",
       "                \"conda-forge\",\n",
       "                \"omnia\"\n",
       "            ],\n",
       "            \"dependencies\": [\n",
       "                \"python=3.7.10\",\n",
       "                {\n",
       "                    \"pip\": [\n",
       "                        \"azureml-defaults~=1.22.0\",\n",
       "                        \"azureml-sdk\",\n",
       "                        \"deepchem\",\n",
       "                        \"numpy==1.19.4\",\n",
       "                        \"IPython\"\n",
       "                    ]\n",
       "                },\n",
       "                \"tensorflow-gpu==2.4.1\",\n",
       "                \"rdkit\",\n",
       "                \"openmm\",\n",
       "                \"pdbfixer\"\n",
       "            ],\n",
       "            \"name\": \"azureml_5b7ddb3503df2a9fa94545e28acd9198\"\n",
       "        },\n",
       "        \"condaDependenciesFile\": null,\n",
       "        \"interpreterPath\": \"python\",\n",
       "        \"userManagedDependencies\": false\n",
       "    },\n",
       "    \"r\": null,\n",
       "    \"spark\": {\n",
       "        \"packages\": [],\n",
       "        \"precachePackages\": true,\n",
       "        \"repositories\": []\n",
       "    },\n",
       "    \"version\": \"5\"\n",
       "}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "myenv = Environment('deepchem_backend2')\n",
    "\n",
    "#already created\n",
    "conda_dep = CondaDependencies().create(python_version='3.7.10', conda_packages=['tensorflow-gpu==2.4.1', 'rdkit', 'openmm', 'pdbfixer'])\n",
    "conda_dep.add_channel(\"conda-forge\")\n",
    "conda_dep.add_channel(\"omnia\")\n",
    "conda_dep.add_pip_package(\"azureml-sdk\")\n",
    "conda_dep.add_pip_package(\"deepchem\")\n",
    "#1.19.4\n",
    "conda_dep.add_pip_package(\"numpy==1.19.4\")\n",
    "#IPython\n",
    "conda_dep.add_pip_package(\"IPython\")\n",
    "conda_dep.save(path=\"./train/condadep.yml\")\n",
    "myenv.python.conda_dependencies=conda_dep\n",
    "myenv.docker.enabled = True\n",
    "myenv.docker.base_image = DEFAULT_GPU_IMAGE\n",
    "myenv.register(workspace=ws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the current working directory and inside of it, create a subdirectory called \"train\" to hold the training script - contents of this will be passed to the remote machine for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/mnt/batch/tasks/shared/LS_root/mounts/clusters/mm-deepchem/code/Users/meganmasanz.work/vae_training\n"
     ]
    }
   ],
   "source": [
    "cwd = os.getcwd()\n",
    "current_dir = cwd\n",
    "print(cwd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/mnt/batch/tasks/shared/LS_root/mounts/clusters/mm-deepchem/code/Users/meganmasanz.work/vae_training/train\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "script_folder = os.path.join(os.getcwd(), \"train\")\n",
    "print(script_folder)\n",
    "os.makedirs(script_folder, exist_ok = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The code below is actually the training and the model evaluation script.  Modifiying the script below will specifically in the main section will change parameters in the training.\n",
    "\n",
    "### Parameters this script will allow changes to include:\n",
    "- epoch_count = 2\n",
    "- learning_rate = .0001\n",
    "- sampling_size = 100000 - during model evaluation how many molecules to attempt to generate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $script_folder/train.py\n",
    "\n",
    "\n",
    "\n",
    "import sys\n",
    "import os\n",
    "import requests\n",
    "import subprocess\n",
    "import shutil\n",
    "import IPython\n",
    "from logging import getLogger, StreamHandler, INFO\n",
    "from deepchem.models.optimizers import Adam, ExponentialDecay\n",
    "from deepchem.models.seqtoseq import AspuruGuzikAutoEncoder\n",
    "import rdkit\n",
    "import numpy as np\n",
    "import deepchem\n",
    "import rdkit\n",
    "import tensorflow as tf\n",
    "from azureml.core import Run\n",
    "from rdkit import Chem\n",
    "from rdkit.Chem import Draw\n",
    "from rdkit.Chem.Draw import IPythonConsole\n",
    "from rdkit.Chem import Descriptors\n",
    "from rdkit.Chem import AllChem\n",
    "from rdkit import DataStructs\n",
    "import random\n",
    "\n",
    "# DEFINE THE MODEL\n",
    "def get_model(train_smiles, tokens, max_length, learning_rate):\n",
    "    run = Run.get_context()\n",
    "    batch_size = 100\n",
    "    \n",
    "    #A learning rate that decreases exponentially with the number of training steps.\n",
    "    #(initial_rate: float, decay_rate: float, decay_steps: int, staircase: bool = True)\n",
    "    run.log('learningRate', learning_rate)\n",
    "    learning_rate = ExponentialDecay(learning_rate, 0.95, len(train_smiles)/batch_size)\n",
    "    \n",
    "    model = AspuruGuzikAutoEncoder(tokens, max_length, model_dir='vae', batch_size=batch_size, learning_rate=learning_rate)\n",
    "    return model\n",
    "\n",
    "# GENERATE MOLECULES AND TEST IF THEY ARE VALID\n",
    "def generate_molecules(model, zinc_data, n_molecules=5000):\n",
    "    run = Run.get_context()\n",
    "    run.log('n_moleculesd', n_molecules)\n",
    "    \n",
    "    predictions = model.predict_from_embeddings(np.random.normal(size=(n_molecules,196))) \n",
    "    valid = []\n",
    "\n",
    "    #using chem from rdkit to ensure generated molecules are valid\n",
    "    count = 0\n",
    "    for p in predictions:\n",
    "        count += 1\n",
    "        smiles = ''.join(p)\n",
    "        if count < 25:\n",
    "            print(smiles)\n",
    "        if rdkit.Chem.MolFromSmiles(smiles) is not None:\n",
    "            valid.append(smiles) \n",
    "\n",
    "    print(len(valid) / n_molecules)\n",
    "    \n",
    "    run.log('valid', (len(valid) / n_molecules))\n",
    "    i = 0\n",
    "    print('**************************')\n",
    "    for s in valid:\n",
    "        i = i + 1\n",
    "        print(s)\n",
    "        output_dir = './outputs/'\n",
    "        os.makedirs(output_dir, exist_ok=True)\n",
    "        \n",
    "        mol = rdkit.Chem.MolFromSmiles(s)\n",
    "        filename = os.path.join(output_dir, 'image' + str(i) + '_' + s + '.png')\n",
    "        print(filename)\n",
    "        rdkit.Chem.Draw.MolToFile(mol, filename)\n",
    "  \n",
    "    print('***************************')\n",
    "    \n",
    "    \n",
    "    print('looking for winners')\n",
    "    valid_in_zinc = []\n",
    "    for x in valid:\n",
    "        if x in zinc_data:\n",
    "            print('Found a winner')\n",
    "            valid_in_zinc.append(x)\n",
    "            \n",
    "    \n",
    "    run.log('valid_in_zinc', len(valid_in_zinc))\n",
    "    print(len(valid_in_zinc))\n",
    "    \n",
    "    if len(valid) > 0:\n",
    "        print(len(valid_in_zinc)/len(valid))\n",
    "        run.log('valid_in_zinc_out_of_valid', len(valid_in_zinc)/len(valid))\n",
    "    else:\n",
    "        run.log('valid_in_zinc_out_of_valid', 0)\n",
    "\n",
    "    \n",
    "    print(len(valid_in_zinc)/n_molecules)\n",
    "    run.log('valid_in_zinc_out_of_all_generated', len(valid_in_zinc)/n_molecules)\n",
    "    \n",
    "\n",
    "\n",
    "def get_mol(smiles):\n",
    "    mol = rdkit.Chem.MolFromSmiles(smiles)\n",
    "    if mol is None:\n",
    "        return None\n",
    "    return rdkit.Chem.Kekulize(mol)\n",
    "   \n",
    "\n",
    "\n",
    "def generate_sequences(epochs, train_smiles): \n",
    "    run = Run.get_context()\n",
    "    run.log('epochs', epochs)\n",
    "    for i in range(epochs):\n",
    "        print('epoch:', i+1)\n",
    "        for s in train_smiles: \n",
    "            yield (s, s)\n",
    "            \n",
    "\n",
    "#deepchem has its own fit model variation\n",
    "def train(model, train_smiles, epochs=1):\n",
    "    model.fit_sequences(generate_sequences(epochs, train_smiles))\n",
    "\n",
    "    \n",
    "def modeltrain(epoch_count, learning_rate, sampling_size): \n",
    "\n",
    "    os.makedirs('data', exist_ok = True)\n",
    "    run = Run.get_context()\n",
    "    tasks, datasets, _ = deepchem.molnet.load_zinc15(\n",
    "    featurizer='raw',\n",
    "    splitter=None,\n",
    "    transformers=[],\n",
    "    data_dir='data', \n",
    "    save_dir='data')\n",
    "    print(tasks)\n",
    "\n",
    "\n",
    "    data = datasets[0]\n",
    "    train_smiles = []\n",
    "    for X, _, _, _ in data.itersamples():\n",
    "        train_smiles.append(rdkit.Chem.MolToSmiles(X))\n",
    "    print(len(train_smiles))\n",
    "    run.log('datasetsize', len(train_smiles))\n",
    "    for smile in train_smiles[0:5]:\n",
    "        print(smile)\n",
    "\n",
    "    # DEFINE THE SMILES TOKENS AND MAX_LENGTHS\n",
    "    tokens = set()\n",
    "    for s in train_smiles:\n",
    "        tokens = tokens.union(set(s))\n",
    "    tokens = sorted(list(tokens))\n",
    "    max_length = max(len(s) for s in train_smiles)\n",
    "\n",
    "    try:\n",
    "        seed = 123\n",
    "        tf.random.set_seed(seed)\n",
    "        device_name = tf.test.gpu_device_name()\n",
    "        print('***************')\n",
    "        print(device_name)\n",
    "        print('***************')\n",
    "        run.log('device_name', device_name)\n",
    "\n",
    "        with tf.device(device_name):\n",
    "            model = get_model(train_smiles, tokens, max_length, learning_rate)\n",
    "            train(model, train_smiles, epoch_count)\n",
    "            generate_molecules(model, train_smiles, sampling_size)\n",
    "\n",
    "    except Exception as e: \n",
    "        print(e)\n",
    "\n",
    "\n",
    "\n",
    "def main():\n",
    "    seed = 123\n",
    "    random.seed(seed)\n",
    "    tf.random.set_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    #os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'\n",
    "    print('in main')\n",
    "    epoch_count = 2\n",
    "    learning_rate = .0001\n",
    "    sampling_size = 100000\n",
    "    modeltrain(epoch_count, learning_rate, sampling_size)\n",
    "    \n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Below will create an experiment with the name 'vae-no-teacher-forcing-run-410e' using the train.py file in the training folder leveraging the compute cluster created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Experiment, ScriptRunConfig\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "experiment = Experiment(workspace = ws, name = \"vae-no-teacher-forcing-run-410e\")\n",
    "script_config = ScriptRunConfig(source_directory = script_folder, script = 'train.py', environment=myenv, compute_target = cluster_name)\n",
    "\n",
    "experiment = Experiment(workspace=ws, name = \"vae-no-teacher-forcing-run-410e\" )\n",
    "run = experiment.submit(config= script_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "### The widiget below will show ouput while submitting the job for real-time review of results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a37786a45ba34fe79d9a38aa2a9609c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_UserRunWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', '…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/vae-no-teacher-forcing-run-410e/runs/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3?wsid=/subscriptions/aa0e72d2-d6df-43e6-bb7a-39157d733cce/resourcegroups/uiuc-cs598/workspaces/cs598-finalproject-aml-dev\", \"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"run_properties\": {\"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"created_utc\": \"2021-04-16T17:51:05.441778Z\", \"properties\": {\"_azureml.ComputeTargetType\": \"amlcompute\", \"ContentSnapshotId\": \"871affdb-59d3-4c09-ad50-2f4f908496db\", \"ProcessInfoFile\": \"azureml-logs/process_info.json\", \"ProcessStatusFile\": \"azureml-logs/process_status.json\"}, \"tags\": {\"_aml_system_ComputeTargetStatus\": \"{\\\"AllocationState\\\":\\\"steady\\\",\\\"PreparingNodeCount\\\":0,\\\"RunningNodeCount\\\":0,\\\"CurrentNodeCount\\\":0}\"}, \"script_name\": null, \"arguments\": null, \"end_time_utc\": \"2021-04-16T18:52:26.677546Z\", \"status\": \"Completed\", \"log_files\": {\"azureml-logs/55_azureml-execution-tvmps_2efa2d0f354641d799bf9e06b5c73092f48f79c82a9f4514b1f29e5abc3cf5e8_d.txt\": \"https://cs598finalproj1009401349.blob.core.windows.net/azureml/ExperimentRun/dcid.vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/azureml-logs/55_azureml-execution-tvmps_2efa2d0f354641d799bf9e06b5c73092f48f79c82a9f4514b1f29e5abc3cf5e8_d.txt?sv=2019-02-02&sr=b&sig=t2OCkogQ40B71%2Fog8DQU3XAgOBnCGdjQG3RYPY07Iuw%3D&st=2021-04-16T22%3A45%3A32Z&se=2021-04-17T06%3A55%3A32Z&sp=r\", \"azureml-logs/65_job_prep-tvmps_2efa2d0f354641d799bf9e06b5c73092f48f79c82a9f4514b1f29e5abc3cf5e8_d.txt\": \"https://cs598finalproj1009401349.blob.core.windows.net/azureml/ExperimentRun/dcid.vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/azureml-logs/65_job_prep-tvmps_2efa2d0f354641d799bf9e06b5c73092f48f79c82a9f4514b1f29e5abc3cf5e8_d.txt?sv=2019-02-02&sr=b&sig=YPM5%2FbpXX2A80N%2FXOeREKcB9xEVoseuTq5RWp0lw1VA%3D&st=2021-04-16T22%3A45%3A32Z&se=2021-04-17T06%3A55%3A32Z&sp=r\", \"azureml-logs/70_driver_log.txt\": \"https://cs598finalproj1009401349.blob.core.windows.net/azureml/ExperimentRun/dcid.vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/azureml-logs/70_driver_log.txt?sv=2019-02-02&sr=b&sig=h2i5MkquMqnSiQGlJpT2VOgaXtnJMKbBcrMjhox278Y%3D&st=2021-04-16T22%3A45%3A32Z&se=2021-04-17T06%3A55%3A32Z&sp=r\", \"azureml-logs/75_job_post-tvmps_2efa2d0f354641d799bf9e06b5c73092f48f79c82a9f4514b1f29e5abc3cf5e8_d.txt\": \"https://cs598finalproj1009401349.blob.core.windows.net/azureml/ExperimentRun/dcid.vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/azureml-logs/75_job_post-tvmps_2efa2d0f354641d799bf9e06b5c73092f48f79c82a9f4514b1f29e5abc3cf5e8_d.txt?sv=2019-02-02&sr=b&sig=gDu%2B6X%2BxT8ySurdbJyTJRp4SjTzEiwHIJFx5V46Rh%2FM%3D&st=2021-04-16T22%3A45%3A32Z&se=2021-04-17T06%3A55%3A32Z&sp=r\", \"azureml-logs/process_info.json\": \"https://cs598finalproj1009401349.blob.core.windows.net/azureml/ExperimentRun/dcid.vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/azureml-logs/process_info.json?sv=2019-02-02&sr=b&sig=ISuZb9C57S7FXrtwPufqAqVRTcc007dKJlT2dMKESw0%3D&st=2021-04-16T22%3A45%3A32Z&se=2021-04-17T06%3A55%3A32Z&sp=r\", \"azureml-logs/process_status.json\": \"https://cs598finalproj1009401349.blob.core.windows.net/azureml/ExperimentRun/dcid.vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/azureml-logs/process_status.json?sv=2019-02-02&sr=b&sig=MF7NTVIo1E5rrdcqdKHbf2m4nLAGCbBlTAArhUSRDGQ%3D&st=2021-04-16T22%3A45%3A32Z&se=2021-04-17T06%3A55%3A32Z&sp=r\", \"logs/azureml/122_azureml.log\": \"https://cs598finalproj1009401349.blob.core.windows.net/azureml/ExperimentRun/dcid.vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/logs/azureml/122_azureml.log?sv=2019-02-02&sr=b&sig=OTdAqUXOAe5QPbqNBVI039ufKXYJVx5x0kAoC6FGEZs%3D&st=2021-04-16T22%3A45%3A32Z&se=2021-04-17T06%3A55%3A32Z&sp=r\", \"logs/azureml/job_prep_azureml.log\": \"https://cs598finalproj1009401349.blob.core.windows.net/azureml/ExperimentRun/dcid.vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/logs/azureml/job_prep_azureml.log?sv=2019-02-02&sr=b&sig=n2pK%2BO7zej9jRn1IDfvZdMJAVpgU893uJYz1iuwA8g0%3D&st=2021-04-16T22%3A45%3A32Z&se=2021-04-17T06%3A55%3A32Z&sp=r\", \"logs/azureml/job_release_azureml.log\": \"https://cs598finalproj1009401349.blob.core.windows.net/azureml/ExperimentRun/dcid.vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/logs/azureml/job_release_azureml.log?sv=2019-02-02&sr=b&sig=RqNzL88iZiLTtvGTmmi0jGvZWFubUQgL%2FPhxp3Lfsm8%3D&st=2021-04-16T22%3A45%3A32Z&se=2021-04-17T06%3A55%3A32Z&sp=r\"}, \"log_groups\": [[\"azureml-logs/process_info.json\", \"azureml-logs/process_status.json\", \"logs/azureml/job_prep_azureml.log\", \"logs/azureml/job_release_azureml.log\"], [\"azureml-logs/55_azureml-execution-tvmps_2efa2d0f354641d799bf9e06b5c73092f48f79c82a9f4514b1f29e5abc3cf5e8_d.txt\"], [\"azureml-logs/65_job_prep-tvmps_2efa2d0f354641d799bf9e06b5c73092f48f79c82a9f4514b1f29e5abc3cf5e8_d.txt\"], [\"azureml-logs/70_driver_log.txt\"], [\"azureml-logs/75_job_post-tvmps_2efa2d0f354641d799bf9e06b5c73092f48f79c82a9f4514b1f29e5abc3cf5e8_d.txt\"], [\"logs/azureml/122_azureml.log\"]], \"run_duration\": \"1:01:21\", \"run_number\": \"2\", \"run_queued_details\": {\"status\": \"Completed\", \"details\": null}}, \"child_runs\": [], \"children_metrics\": {}, \"run_metrics\": [{\"name\": \"datasetsize\", \"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"categories\": [0], \"series\": [{\"data\": [250000]}]}, {\"name\": \"device_name\", \"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"categories\": [0], \"series\": [{\"data\": [\"/device:GPU:0\"]}]}, {\"name\": \"learningRate\", \"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"categories\": [0], \"series\": [{\"data\": [0.0001]}]}, {\"name\": \"epochs\", \"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"categories\": [0], \"series\": [{\"data\": [2]}]}, {\"name\": \"n_moleculesd\", \"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"categories\": [0], \"series\": [{\"data\": [100000]}]}, {\"name\": \"valid\", \"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"categories\": [0], \"series\": [{\"data\": [0.11597]}]}, {\"name\": \"valid_in_zinc\", \"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"categories\": [0], \"series\": [{\"data\": [0]}]}, {\"name\": \"valid_in_zinc_out_of_valid\", \"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"categories\": [0], \"series\": [{\"data\": [0.0]}]}, {\"name\": \"valid_in_zinc_out_of_all_generated\", \"run_id\": \"vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\", \"categories\": [0], \"series\": [{\"data\": [0.0]}]}], \"run_logs\": \"2021-04-16 17:56:59,230|azureml|DEBUG|Inputs:: kwargs: {'OutputCollection': True, 'EnableMLflowTracking': True, 'snapshotProject': True, 'only_in_process_features': True, 'skip_track_logs_dir': True}, track_folders: None, deny_list: None, directories_to_watch: []\\n2021-04-16 17:56:59,230|azureml.history._tracking.PythonWorkingDirectory|DEBUG|Execution target type: batchai\\n2021-04-16 17:56:59,236|azureml.history._tracking.PythonWorkingDirectory|DEBUG|Failed to import pyspark with error: No module named 'pyspark'\\n2021-04-16 17:56:59,236|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Pinning working directory for filesystems: ['pyfs']\\n2021-04-16 17:56:59,380|azureml.core|WARNING|Failure while loading azureml_run_type_providers. Failed to load entrypoint hyperdrive = azureml.train.hyperdrive:HyperDriveRun._from_run_dto with exception (azureml-core 1.22.0 (/azureml-envs/azureml_5b7ddb3503df2a9fa94545e28acd9198/lib/python3.7/site-packages), Requirement.parse('azureml-core~=1.26.0'), {'azureml-telemetry'}).\\n2021-04-16 17:56:59,384|azureml.core|WARNING|Failure while loading azureml_run_type_providers. Failed to load entrypoint automl = azureml.train.automl.run:AutoMLRun._from_run_dto with exception (azureml-core 1.22.0 (/azureml-envs/azureml_5b7ddb3503df2a9fa94545e28acd9198/lib/python3.7/site-packages), Requirement.parse('azureml-core~=1.26.0'), {'azureml-telemetry'}).\\n2021-04-16 17:56:59,385|azureml.core|WARNING|Failure while loading azureml_run_type_providers. Failed to load entrypoint azureml.PipelineRun = azureml.pipeline.core.run:PipelineRun._from_dto with exception (azureml-core 1.22.0 (/azureml-envs/azureml_5b7ddb3503df2a9fa94545e28acd9198/lib/python3.7/site-packages), Requirement.parse('azureml-core~=1.26.0')).\\n2021-04-16 17:56:59,385|azureml.core|WARNING|Failure while loading azureml_run_type_providers. Failed to load entrypoint azureml.ReusedStepRun = azureml.pipeline.core.run:StepRun._from_reused_dto with exception (azureml-core 1.22.0 (/azureml-envs/azureml_5b7ddb3503df2a9fa94545e28acd9198/lib/python3.7/site-packages), Requirement.parse('azureml-core~=1.26.0')).\\n2021-04-16 17:56:59,385|azureml.core|WARNING|Failure while loading azureml_run_type_providers. Failed to load entrypoint azureml.StepRun = azureml.pipeline.core.run:StepRun._from_dto with exception (azureml-core 1.22.0 (/azureml-envs/azureml_5b7ddb3503df2a9fa94545e28acd9198/lib/python3.7/site-packages), Requirement.parse('azureml-core~=1.26.0')).\\n2021-04-16 17:56:59,572|azureml.core.run|DEBUG|Adding new factory <function ScriptRun._from_run_dto at 0x7fa7648a5ef0> for run source azureml.scriptrun\\n2021-04-16 17:56:59,573|azureml.core.run|DEBUG|Identity in use is not set. Falling back to using AMLToken\\n2021-04-16 17:56:59,573|azureml.core.run|DEBUG|Using AMLToken auth for remote run\\n2021-04-16 17:56:59,575|azureml.core.authentication.TokenRefresherDaemon|DEBUG|Starting daemon and triggering first instance\\n2021-04-16 17:56:59,581|azureml._restclient.clientbase|INFO|Created a worker pool for first use\\n2021-04-16 17:56:59,581|azureml.core.authentication|DEBUG|Time to expire 1814045.418531 seconds\\n2021-04-16 17:56:59,581|azureml._restclient.service_context|DEBUG|Created a static thread pool for ServiceContext class\\n2021-04-16 17:56:59,581|azureml._restclient.clientbase|DEBUG|ClientBase: Calling get with url None\\n2021-04-16 17:56:59,610|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:56:59,610|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:56:59,611|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:56:59,611|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:56:59,611|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:56:59,611|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:56:59,611|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:56:59,640|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.RunClient.get_by_exp_id-async:False|DEBUG|[START]\\n2021-04-16 17:56:59,640|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.RunClient|DEBUG|ClientBase: Calling get_by_exp_id with url /history/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/experimentids/{experimentId}/runs/{runId}\\n2021-04-16 17:56:59,715|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.RunClient.get_by_exp_id-async:False|DEBUG|[STOP]\\n2021-04-16 17:56:59,715|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3|DEBUG|Constructing run from dto. type: azureml.scriptrun, source: None, props: {'_azureml.ComputeTargetType': 'amlcompute', 'ContentSnapshotId': '871affdb-59d3-4c09-ad50-2f4f908496db', 'ProcessInfoFile': 'azureml-logs/process_info.json', 'ProcessStatusFile': 'azureml-logs/process_status.json'}\\n2021-04-16 17:56:59,715|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunContextManager|DEBUG|Valid logs dir, setting up content loader\\n2021-04-16 17:56:59,716|azureml|WARNING|Could not import azureml.mlflow or azureml.contrib.mlflow mlflow APIs will not run against AzureML services.  Add azureml-mlflow as a conda dependency for the run if this behavior is desired\\n2021-04-16 17:56:59,716|azureml.WorkerPool|DEBUG|[START]\\n2021-04-16 17:56:59,716|azureml.SendRunKillSignal|DEBUG|[START]\\n2021-04-16 17:56:59,716|azureml.RunStatusContext|DEBUG|[START]\\n2021-04-16 17:56:59,716|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunContextManager.RunStatusContext|DEBUG|[START]\\n2021-04-16 17:56:59,716|azureml.MetricsClient|DEBUG|[START]\\n2021-04-16 17:56:59,716|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|[START]\\n2021-04-16 17:56:59,716|azureml.WorkingDirectoryCM|DEBUG|[START]\\n2021-04-16 17:56:59,716|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|[START]\\n2021-04-16 17:56:59,716|azureml.history._tracking.PythonWorkingDirectory|INFO|Current working dir: /mnt/batch/tasks/shared/LS_root/jobs/cs598-finalproject-aml-dev/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/mounts/workspaceblobstore/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\\n2021-04-16 17:56:59,716|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Calling pyfs\\n2021-04-16 17:56:59,716|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Storing working dir for pyfs as /mnt/batch/tasks/shared/LS_root/jobs/cs598-finalproject-aml-dev/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/mounts/workspaceblobstore/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\\n2021-04-16 17:57:06,375|azureml.core.run|DEBUG|Identity in use is not set. Falling back to using AMLToken\\n2021-04-16 17:57:06,376|azureml.core.run|DEBUG|Using AMLToken auth for remote run\\n2021-04-16 17:57:06,376|azureml._restclient.service_context|DEBUG|Access an existing static threadpool for ServiceContext class\\n2021-04-16 17:57:06,376|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:57:06,377|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:57:06,377|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:57:06,377|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:57:06,377|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:57:06,377|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:57:06,378|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 17:57:06,399|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.RunClient.get_by_exp_id-async:False|DEBUG|[START]\\n2021-04-16 17:57:06,399|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.RunClient|DEBUG|ClientBase: Calling get_by_exp_id with url /history/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/experimentids/{experimentId}/runs/{runId}\\n2021-04-16 17:57:06,453|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.RunClient.get_by_exp_id-async:False|DEBUG|[STOP]\\n2021-04-16 17:57:06,454|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3|DEBUG|Constructing run from dto. type: azureml.scriptrun, source: None, props: {'_azureml.ComputeTargetType': 'amlcompute', 'ContentSnapshotId': '871affdb-59d3-4c09-ad50-2f4f908496db', 'ProcessInfoFile': 'azureml-logs/process_info.json', 'ProcessStatusFile': 'azureml-logs/process_status.json'}\\n2021-04-16 17:57:06,454|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunContextManager|DEBUG|Valid logs dir, setting up content loader\\n2021-04-16 17:57:29,576|azureml.core.authentication|DEBUG|Time to expire 1814015.424002 seconds\\n2021-04-16 17:57:59,597|azureml.core.authentication|DEBUG|Time to expire 1813985.402851 seconds\\n2021-04-16 17:58:29,602|azureml.core.authentication|DEBUG|Time to expire 1813955.397374 seconds\\n2021-04-16 17:58:59,608|azureml.core.authentication|DEBUG|Time to expire 1813925.391474 seconds\\n2021-04-16 17:59:29,613|azureml.core.authentication|DEBUG|Time to expire 1813895.386243 seconds\\n2021-04-16 17:59:59,619|azureml.core.authentication|DEBUG|Time to expire 1813865.380792 seconds\\n2021-04-16 18:00:29,619|azureml.core.authentication|DEBUG|Time to expire 1813835.380303 seconds\\n2021-04-16 18:00:33,173|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2021-04-16 18:00:33,173|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.PostMetricsBatchV2Daemon|DEBUG|Starting daemon and triggering first instance\\n2021-04-16 18:00:33,174|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2021-04-16 18:00:35,046|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2021-04-16 18:00:35,047|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2021-04-16 18:00:35,047|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Batch size 1.\\n2021-04-16 18:00:35,053|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:00:35,053|azureml._restclient.service_context.WorkerPool|DEBUG|submitting future: _log_batch_v2\\n2021-04-16 18:00:35,068|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2021-04-16 18:00:35,068|azureml.core.run|DEBUG|Identity in use is not set. Falling back to using AMLToken\\n2021-04-16 18:00:35,068|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Metrics Client: _log_batch_v2 is calling post_run_metrics posting 1 values.\\n2021-04-16 18:00:35,068|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.0__log_batch_v2|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:00:35,068|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2021-04-16 18:00:35,069|azureml.core.run|DEBUG|Using AMLToken auth for remote run\\n2021-04-16 18:00:35,069|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[START]\\n2021-04-16 18:00:35,069|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Adding task 0__log_batch_v2 to queue of approximate size: 0\\n2021-04-16 18:00:35,069|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2021-04-16 18:00:35,069|azureml._restclient.service_context|DEBUG|Access an existing static threadpool for ServiceContext class\\n2021-04-16 18:00:35,069|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling _post_run_metrics_log_failed_validations with url None\\n2021-04-16 18:00:35,070|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2021-04-16 18:00:35,074|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2021-04-16 18:00:35,074|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2021-04-16 18:00:35,074|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:35,074|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2021-04-16 18:00:35,075|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:00:35,075|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2021-04-16 18:00:35,075|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2021-04-16 18:00:35,075|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:35,075|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:35,075|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:35,076|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:35,076|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:35,076|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:35,574|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[STOP]\\n2021-04-16 18:00:36,047|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2021-04-16 18:00:36,050|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2021-04-16 18:00:36,051|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Batch size 2.\\n2021-04-16 18:00:36,051|azureml._restclient.service_context.WorkerPool|DEBUG|submitting future: _log_batch_v2\\n2021-04-16 18:00:36,051|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:00:36,052|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Metrics Client: _log_batch_v2 is calling post_run_metrics posting 2 values.\\n2021-04-16 18:00:36,053|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.1__log_batch_v2|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:00:36,053|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2021-04-16 18:00:36,053|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[START]\\n2021-04-16 18:00:36,054|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Adding task 1__log_batch_v2 to queue of approximate size: 1\\n2021-04-16 18:00:36,054|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2021-04-16 18:00:36,054|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling _post_run_metrics_log_failed_validations with url None\\n2021-04-16 18:00:36,054|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2021-04-16 18:00:36,058|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2021-04-16 18:00:36,058|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2021-04-16 18:00:36,059|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2021-04-16 18:00:36,059|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2021-04-16 18:00:36,059|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:00:36,059|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2021-04-16 18:00:36,059|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2021-04-16 18:00:36,215|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[STOP]\\n2021-04-16 18:00:37,169|azureml.core.run|DEBUG|Identity in use is not set. Falling back to using AMLToken\\n2021-04-16 18:00:37,169|azureml.core.run|DEBUG|Using AMLToken auth for remote run\\n2021-04-16 18:00:37,170|azureml._restclient.service_context|DEBUG|Access an existing static threadpool for ServiceContext class\\n2021-04-16 18:00:37,170|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:37,170|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:37,171|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:37,171|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:37,171|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:37,171|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:37,171|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:00:38,048|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2021-04-16 18:00:38,048|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2021-04-16 18:00:38,049|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Batch size 1.\\n2021-04-16 18:00:38,049|azureml._restclient.service_context.WorkerPool|DEBUG|submitting future: _log_batch_v2\\n2021-04-16 18:00:38,049|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:00:38,050|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2021-04-16 18:00:38,050|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Metrics Client: _log_batch_v2 is calling post_run_metrics posting 1 values.\\n2021-04-16 18:00:38,050|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2021-04-16 18:00:38,050|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.2__log_batch_v2|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:00:38,051|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[START]\\n2021-04-16 18:00:38,051|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2021-04-16 18:00:38,051|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Adding task 2__log_batch_v2 to queue of approximate size: 2\\n2021-04-16 18:00:38,052|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling _post_run_metrics_log_failed_validations with url None\\n2021-04-16 18:00:38,052|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2021-04-16 18:00:38,055|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2021-04-16 18:00:38,058|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2021-04-16 18:00:38,058|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2021-04-16 18:00:38,059|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:00:38,059|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2021-04-16 18:00:38,059|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2021-04-16 18:00:38,290|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[STOP]\\n2021-04-16 18:00:59,620|azureml.core.authentication|DEBUG|Time to expire 1813805.379948 seconds\\n2021-04-16 18:01:29,620|azureml.core.authentication|DEBUG|Time to expire 1813775.379759 seconds\\n2021-04-16 18:01:59,620|azureml.core.authentication|DEBUG|Time to expire 1813745.37951 seconds\\n2021-04-16 18:02:29,620|azureml.core.authentication|DEBUG|Time to expire 1813715.37925 seconds\\n2021-04-16 18:02:59,621|azureml.core.authentication|DEBUG|Time to expire 1813685.378996 seconds\\n2021-04-16 18:03:29,621|azureml.core.authentication|DEBUG|Time to expire 1813655.378699 seconds\\n2021-04-16 18:03:59,621|azureml.core.authentication|DEBUG|Time to expire 1813625.378459 seconds\\n2021-04-16 18:04:29,621|azureml.core.authentication|DEBUG|Time to expire 1813595.378215 seconds\\n2021-04-16 18:04:59,622|azureml.core.authentication|DEBUG|Time to expire 1813565.377942 seconds\\n2021-04-16 18:05:29,622|azureml.core.authentication|DEBUG|Time to expire 1813535.3777 seconds\\n2021-04-16 18:05:59,622|azureml.core.authentication|DEBUG|Time to expire 1813505.377473 seconds\\n2021-04-16 18:06:29,622|azureml.core.authentication|DEBUG|Time to expire 1813475.377197 seconds\\n2021-04-16 18:06:59,623|azureml.core.authentication|DEBUG|Time to expire 1813445.376931 seconds\\n2021-04-16 18:07:29,623|azureml.core.authentication|DEBUG|Time to expire 1813415.376687 seconds\\n2021-04-16 18:07:59,623|azureml.core.authentication|DEBUG|Time to expire 1813385.376455 seconds\\n2021-04-16 18:08:29,623|azureml.core.authentication|DEBUG|Time to expire 1813355.376202 seconds\\n2021-04-16 18:08:59,624|azureml.core.authentication|DEBUG|Time to expire 1813325.375946 seconds\\n2021-04-16 18:09:29,624|azureml.core.authentication|DEBUG|Time to expire 1813295.375716 seconds\\n2021-04-16 18:09:59,624|azureml.core.authentication|DEBUG|Time to expire 1813265.375476 seconds\\n2021-04-16 18:10:29,624|azureml.core.authentication|DEBUG|Time to expire 1813235.375226 seconds\\n2021-04-16 18:10:59,625|azureml.core.authentication|DEBUG|Time to expire 1813205.37499 seconds\\n2021-04-16 18:11:29,625|azureml.core.authentication|DEBUG|Time to expire 1813175.374729 seconds\\n2021-04-16 18:11:59,625|azureml.core.authentication|DEBUG|Time to expire 1813145.374418 seconds\\n2021-04-16 18:12:29,625|azureml.core.authentication|DEBUG|Time to expire 1813115.374096 seconds\\n2021-04-16 18:12:59,626|azureml.core.authentication|DEBUG|Time to expire 1813085.373875 seconds\\n2021-04-16 18:13:29,626|azureml.core.authentication|DEBUG|Time to expire 1813055.3736 seconds\\n2021-04-16 18:13:59,626|azureml.core.authentication|DEBUG|Time to expire 1813025.373371 seconds\\n2021-04-16 18:14:29,626|azureml.core.authentication|DEBUG|Time to expire 1812995.373131 seconds\\n2021-04-16 18:14:59,627|azureml.core.authentication|DEBUG|Time to expire 1812965.372872 seconds\\n2021-04-16 18:15:29,627|azureml.core.authentication|DEBUG|Time to expire 1812935.372633 seconds\\n2021-04-16 18:15:59,627|azureml.core.authentication|DEBUG|Time to expire 1812905.372415 seconds\\n2021-04-16 18:16:29,627|azureml.core.authentication|DEBUG|Time to expire 1812875.372113 seconds\\n2021-04-16 18:16:59,628|azureml.core.authentication|DEBUG|Time to expire 1812845.371841 seconds\\n2021-04-16 18:17:29,628|azureml.core.authentication|DEBUG|Time to expire 1812815.371544 seconds\\n2021-04-16 18:17:59,628|azureml.core.authentication|DEBUG|Time to expire 1812785.371304 seconds\\n2021-04-16 18:18:29,629|azureml.core.authentication|DEBUG|Time to expire 1812755.370989 seconds\\n2021-04-16 18:18:59,629|azureml.core.authentication|DEBUG|Time to expire 1812725.370739 seconds\\n2021-04-16 18:19:29,629|azureml.core.authentication|DEBUG|Time to expire 1812695.370516 seconds\\n2021-04-16 18:19:59,629|azureml.core.authentication|DEBUG|Time to expire 1812665.370286 seconds\\n2021-04-16 18:20:29,630|azureml.core.authentication|DEBUG|Time to expire 1812635.370045 seconds\\n2021-04-16 18:20:59,630|azureml.core.authentication|DEBUG|Time to expire 1812605.36978 seconds\\n2021-04-16 18:21:29,630|azureml.core.authentication|DEBUG|Time to expire 1812575.369581 seconds\\n2021-04-16 18:21:40,262|azureml.core.run|DEBUG|Identity in use is not set. Falling back to using AMLToken\\n2021-04-16 18:21:40,262|azureml.core.run|DEBUG|Using AMLToken auth for remote run\\n2021-04-16 18:21:40,262|azureml._restclient.service_context|DEBUG|Access an existing static threadpool for ServiceContext class\\n2021-04-16 18:21:40,263|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:21:40,263|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:21:40,263|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:21:40,263|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:21:40,264|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:21:40,264|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:21:40,264|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.experiments.azureml.net.\\n2021-04-16 18:21:40,478|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2021-04-16 18:21:40,478|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2021-04-16 18:21:40,478|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Batch size 1.\\n2021-04-16 18:21:40,478|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:21:40,478|azureml._restclient.service_context.WorkerPool|DEBUG|submitting future: _log_batch_v2\\n2021-04-16 18:21:40,478|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2021-04-16 18:21:40,479|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Metrics Client: _log_batch_v2 is calling post_run_metrics posting 1 values.\\n2021-04-16 18:21:40,479|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.3__log_batch_v2|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:21:40,479|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2021-04-16 18:21:40,479|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[START]\\n2021-04-16 18:21:40,479|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Adding task 3__log_batch_v2 to queue of approximate size: 3\\n2021-04-16 18:21:40,480|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2021-04-16 18:21:40,480|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling _post_run_metrics_log_failed_validations with url None\\n2021-04-16 18:21:40,480|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2021-04-16 18:21:40,484|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2021-04-16 18:21:40,484|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2021-04-16 18:21:40,484|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2021-04-16 18:21:40,484|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:21:40,484|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2021-04-16 18:21:40,484|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2021-04-16 18:21:41,785|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[STOP]\\n2021-04-16 18:21:59,647|azureml.core.authentication|DEBUG|Time to expire 1812545.35264 seconds\\n2021-04-16 18:22:29,642|azureml.core.authentication|DEBUG|Time to expire 1812515.357311 seconds\\n2021-04-16 18:22:59,656|azureml.core.authentication|DEBUG|Time to expire 1812485.343314 seconds\\n2021-04-16 18:23:29,662|azureml.core.authentication|DEBUG|Time to expire 1812455.337928 seconds\\n2021-04-16 18:23:59,668|azureml.core.authentication|DEBUG|Time to expire 1812425.331675 seconds\\n2021-04-16 18:24:29,677|azureml.core.authentication|DEBUG|Time to expire 1812395.322963 seconds\\n2021-04-16 18:24:59,678|azureml.core.authentication|DEBUG|Time to expire 1812365.321737 seconds\\n2021-04-16 18:25:29,688|azureml.core.authentication|DEBUG|Time to expire 1812335.311166 seconds\\n2021-04-16 18:25:59,693|azureml.core.authentication|DEBUG|Time to expire 1812305.306581 seconds\\n2021-04-16 18:26:29,689|azureml.core.authentication|DEBUG|Time to expire 1812275.310523 seconds\\n2021-04-16 18:26:59,702|azureml.core.authentication|DEBUG|Time to expire 1812245.29769 seconds\\n2021-04-16 18:27:29,704|azureml.core.authentication|DEBUG|Time to expire 1812215.295686 seconds\\n2021-04-16 18:27:59,709|azureml.core.authentication|DEBUG|Time to expire 1812185.290293 seconds\\n2021-04-16 18:28:29,709|azureml.core.authentication|DEBUG|Time to expire 1812155.290122 seconds\\n2021-04-16 18:28:59,716|azureml.core.authentication|DEBUG|Time to expire 1812125.28336 seconds\\n2021-04-16 18:29:29,730|azureml.core.authentication|DEBUG|Time to expire 1812095.269602 seconds\\n2021-04-16 18:29:59,728|azureml.core.authentication|DEBUG|Time to expire 1812065.27182 seconds\\n2021-04-16 18:30:29,734|azureml.core.authentication|DEBUG|Time to expire 1812035.265374 seconds\\n2021-04-16 18:30:59,731|azureml.core.authentication|DEBUG|Time to expire 1812005.268706 seconds\\n2021-04-16 18:31:29,742|azureml.core.authentication|DEBUG|Time to expire 1811975.257362 seconds\\n2021-04-16 18:31:59,744|azureml.core.authentication|DEBUG|Time to expire 1811945.255441 seconds\\n2021-04-16 18:32:29,743|azureml.core.authentication|DEBUG|Time to expire 1811915.256724 seconds\\n2021-04-16 18:32:59,746|azureml.core.authentication|DEBUG|Time to expire 1811885.253661 seconds\\n2021-04-16 18:33:29,746|azureml.core.authentication|DEBUG|Time to expire 1811855.253759 seconds\\n2021-04-16 18:33:59,754|azureml.core.authentication|DEBUG|Time to expire 1811825.245868 seconds\\n2021-04-16 18:34:29,765|azureml.core.authentication|DEBUG|Time to expire 1811795.234737 seconds\\n2021-04-16 18:34:59,762|azureml.core.authentication|DEBUG|Time to expire 1811765.237174 seconds\\n2021-04-16 18:35:29,768|azureml.core.authentication|DEBUG|Time to expire 1811735.231913 seconds\\n2021-04-16 18:35:59,766|azureml.core.authentication|DEBUG|Time to expire 1811705.234003 seconds\\n2021-04-16 18:36:29,775|azureml.core.authentication|DEBUG|Time to expire 1811675.224055 seconds\\n2021-04-16 18:36:59,786|azureml.core.authentication|DEBUG|Time to expire 1811645.21342 seconds\\n2021-04-16 18:37:29,790|azureml.core.authentication|DEBUG|Time to expire 1811615.209888 seconds\\n2021-04-16 18:37:59,789|azureml.core.authentication|DEBUG|Time to expire 1811585.210728 seconds\\n2021-04-16 18:38:29,796|azureml.core.authentication|DEBUG|Time to expire 1811555.203522 seconds\\n2021-04-16 18:38:59,796|azureml.core.authentication|DEBUG|Time to expire 1811525.2038 seconds\\n2021-04-16 18:39:29,804|azureml.core.authentication|DEBUG|Time to expire 1811495.19513 seconds\\n2021-04-16 18:39:59,805|azureml.core.authentication|DEBUG|Time to expire 1811465.194153 seconds\\n2021-04-16 18:40:29,815|azureml.core.authentication|DEBUG|Time to expire 1811435.184694 seconds\\n2021-04-16 18:40:59,824|azureml.core.authentication|DEBUG|Time to expire 1811405.175569 seconds\\n2021-04-16 18:41:29,826|azureml.core.authentication|DEBUG|Time to expire 1811375.173403 seconds\\n2021-04-16 18:41:59,825|azureml.core.authentication|DEBUG|Time to expire 1811345.175023 seconds\\n2021-04-16 18:42:29,837|azureml.core.authentication|DEBUG|Time to expire 1811315.162736 seconds\\n2021-04-16 18:42:59,845|azureml.core.authentication|DEBUG|Time to expire 1811285.154116 seconds\\n2021-04-16 18:43:29,851|azureml.core.authentication|DEBUG|Time to expire 1811255.148528 seconds\\n2021-04-16 18:43:59,852|azureml.core.authentication|DEBUG|Time to expire 1811225.147367 seconds\\n2021-04-16 18:44:29,865|azureml.core.authentication|DEBUG|Time to expire 1811195.134992 seconds\\n2021-04-16 18:44:59,876|azureml.core.authentication|DEBUG|Time to expire 1811165.123344 seconds\\n2021-04-16 18:45:29,881|azureml.core.authentication|DEBUG|Time to expire 1811135.118441 seconds\\n2021-04-16 18:45:59,888|azureml.core.authentication|DEBUG|Time to expire 1811105.111395 seconds\\n2021-04-16 18:46:29,899|azureml.core.authentication|DEBUG|Time to expire 1811075.100239 seconds\\n2021-04-16 18:46:59,900|azureml.core.authentication|DEBUG|Time to expire 1811045.099675 seconds\\n2021-04-16 18:47:29,904|azureml.core.authentication|DEBUG|Time to expire 1811015.095609 seconds\\n2021-04-16 18:47:59,909|azureml.core.authentication|DEBUG|Time to expire 1810985.090687 seconds\\n2021-04-16 18:48:20,259|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2021-04-16 18:48:20,259|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2021-04-16 18:48:20,260|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Batch size 1.\\n2021-04-16 18:48:20,260|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:48:20,260|azureml._restclient.service_context.WorkerPool|DEBUG|submitting future: _log_batch_v2\\n2021-04-16 18:48:20,260|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2021-04-16 18:48:20,260|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Metrics Client: _log_batch_v2 is calling post_run_metrics posting 1 values.\\n2021-04-16 18:48:20,261|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.4__log_batch_v2|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:48:20,261|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2021-04-16 18:48:20,261|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[START]\\n2021-04-16 18:48:20,261|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Adding task 4__log_batch_v2 to queue of approximate size: 4\\n2021-04-16 18:48:20,262|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2021-04-16 18:48:20,262|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling _post_run_metrics_log_failed_validations with url None\\n2021-04-16 18:48:20,262|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2021-04-16 18:48:20,266|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2021-04-16 18:48:20,266|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2021-04-16 18:48:20,266|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2021-04-16 18:48:20,266|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:48:20,266|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2021-04-16 18:48:20,266|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2021-04-16 18:48:20,465|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[STOP]\\n2021-04-16 18:48:29,903|azureml.core.authentication|DEBUG|Time to expire 1810955.096222 seconds\\n2021-04-16 18:48:59,908|azureml.core.authentication|DEBUG|Time to expire 1810925.09132 seconds\\n2021-04-16 18:49:29,908|azureml.core.authentication|DEBUG|Time to expire 1810895.091076 seconds\\n2021-04-16 18:49:59,909|azureml.core.authentication|DEBUG|Time to expire 1810865.090858 seconds\\n2021-04-16 18:50:29,911|azureml.core.authentication|DEBUG|Time to expire 1810835.088116 seconds\\n2021-04-16 18:50:59,918|azureml.core.authentication|DEBUG|Time to expire 1810805.081753 seconds\\n2021-04-16 18:51:29,937|azureml.core.authentication|DEBUG|Time to expire 1810775.062953 seconds\\n2021-04-16 18:51:53,467|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2021-04-16 18:51:53,468|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2021-04-16 18:51:53,474|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Batch size 3.\\n2021-04-16 18:51:53,474|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:51:53,474|azureml._restclient.service_context.WorkerPool|DEBUG|submitting future: _log_batch_v2\\n2021-04-16 18:51:53,475|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2021-04-16 18:51:53,475|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Metrics Client: _log_batch_v2 is calling post_run_metrics posting 3 values.\\n2021-04-16 18:51:53,475|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2021-04-16 18:51:53,475|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.5__log_batch_v2|DEBUG|Using basic handler - no exception handling\\n2021-04-16 18:51:53,475|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[START]\\n2021-04-16 18:51:53,475|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2021-04-16 18:51:53,476|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Adding task 5__log_batch_v2 to queue of approximate size: 5\\n2021-04-16 18:51:53,476|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling _post_run_metrics_log_failed_validations with url None\\n2021-04-16 18:51:53,476|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2021-04-16 18:51:53,480|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2021-04-16 18:51:53,480|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2021-04-16 18:51:53,480|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2021-04-16 18:51:53,480|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:51:53,480|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2021-04-16 18:51:53,481|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2021-04-16 18:51:53,519|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Calling pyfs\\n2021-04-16 18:51:53,519|azureml.history._tracking.PythonWorkingDirectory|INFO|Current working dir: /mnt/batch/tasks/shared/LS_root/jobs/cs598-finalproject-aml-dev/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/mounts/workspaceblobstore/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\\n2021-04-16 18:51:53,519|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Reverting working dir from /mnt/batch/tasks/shared/LS_root/jobs/cs598-finalproject-aml-dev/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/mounts/workspaceblobstore/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3 to /mnt/batch/tasks/shared/LS_root/jobs/cs598-finalproject-aml-dev/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/mounts/workspaceblobstore/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\\n2021-04-16 18:51:53,519|azureml.history._tracking.PythonWorkingDirectory|INFO|Working dir is already updated /mnt/batch/tasks/shared/LS_root/jobs/cs598-finalproject-aml-dev/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3/mounts/workspaceblobstore/azureml/vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3\\n2021-04-16 18:51:53,519|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|[STOP]\\n2021-04-16 18:51:53,520|azureml.WorkingDirectoryCM|DEBUG|[STOP]\\n2021-04-16 18:51:53,520|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,520|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2021-04-16 18:51:53,520|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.PostMetricsBatchDaemon|DEBUG|Starting daemon and triggering first instance\\n2021-04-16 18:51:53,520|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2021-04-16 18:51:53,520|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,520|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 120 is different from task queue timeout 120, using flush timeout\\n2021-04-16 18:51:53,520|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 120 seconds on tasks: [].\\n2021-04-16 18:51:53,520|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2021-04-16 18:51:53,520|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,521|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2021-04-16 18:51:53,521|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.PostMetricsBatchV2Daemon|DEBUG|Starting daemon and triggering first instance\\n2021-04-16 18:51:53,521|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2021-04-16 18:51:53,521|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,521|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|flush timeout 120 is different from task queue timeout 120, using flush timeout\\n2021-04-16 18:51:53,521|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|Waiting 120 seconds on tasks: [].\\n2021-04-16 18:51:53,521|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|\\n2021-04-16 18:51:53,521|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,521|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,522|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[START]\\n2021-04-16 18:51:53,522|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling wait_on_ingest with url /history/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/experiments/{experimentName}/runs/{runId}/metricsingest/wait\\n2021-04-16 18:51:53,578|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[STOP]\\n2021-04-16 18:51:53,578|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,578|azureml.MetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,578|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300 is different from task queue timeout 120, using flush timeout\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300 seconds on tasks: [].\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300 is different from task queue timeout 120, using flush timeout\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|Waiting 300 seconds on tasks: [].\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,579|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,580|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[START]\\n2021-04-16 18:51:53,580|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling wait_on_ingest with url /history/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/experiments/{experimentName}/runs/{runId}/metricsingest/wait\\n2021-04-16 18:51:53,644|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[STOP]\\n2021-04-16 18:51:53,645|azureml.RunStatusContext|DEBUG|[STOP]\\n2021-04-16 18:51:53,645|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,645|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,645|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 900.0 is different from task queue timeout 120, using flush timeout\\n2021-04-16 18:51:53,645|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 900.0 seconds on tasks: [].\\n2021-04-16 18:51:53,645|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2021-04-16 18:51:53,645|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,645|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,646|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|flush timeout 900.0 is different from task queue timeout 120, using flush timeout\\n2021-04-16 18:51:53,646|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|Waiting 900.0 seconds on tasks: [].\\n2021-04-16 18:51:53,646|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|\\n2021-04-16 18:51:53,646|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,646|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,646|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[START]\\n2021-04-16 18:51:53,646|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling wait_on_ingest with url /history/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/experiments/{experimentName}/runs/{runId}/metricsingest/wait\\n2021-04-16 18:51:53,721|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[STOP]\\n2021-04-16 18:51:53,722|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,722|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2021-04-16 18:51:53,722|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.PostMetricsBatchDaemon|DEBUG|Starting daemon and triggering first instance\\n2021-04-16 18:51:53,722|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2021-04-16 18:51:53,722|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,722|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 900.0 is different from task queue timeout 120, using flush timeout\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 900.0 seconds on tasks: [].\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|flush timeout 900.0 is different from task queue timeout 120, using flush timeout\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|Waiting 900.0 seconds on tasks: [AsyncTask(0__log_batch_v2), AsyncTask(1__log_batch_v2), AsyncTask(2__log_batch_v2), AsyncTask(3__log_batch_v2), AsyncTask(4__log_batch_v2), AsyncTask(5__log_batch_v2)].\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.0__log_batch_v2.WaitingTask|DEBUG|[START]\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.0__log_batch_v2.WaitingTask|DEBUG|Awaiter is PostMetricsBatchV2\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.0__log_batch_v2.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.1__log_batch_v2.WaitingTask|DEBUG|[START]\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.1__log_batch_v2.WaitingTask|DEBUG|Awaiter is PostMetricsBatchV2\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.1__log_batch_v2.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:51:53,723|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.2__log_batch_v2.WaitingTask|DEBUG|[START]\\n2021-04-16 18:51:53,724|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.2__log_batch_v2.WaitingTask|DEBUG|Awaiter is PostMetricsBatchV2\\n2021-04-16 18:51:53,724|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.2__log_batch_v2.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:51:53,724|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.3__log_batch_v2.WaitingTask|DEBUG|[START]\\n2021-04-16 18:51:53,724|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.3__log_batch_v2.WaitingTask|DEBUG|Awaiter is PostMetricsBatchV2\\n2021-04-16 18:51:53,724|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.3__log_batch_v2.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:51:53,724|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.4__log_batch_v2.WaitingTask|DEBUG|[START]\\n2021-04-16 18:51:53,724|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.4__log_batch_v2.WaitingTask|DEBUG|Awaiter is PostMetricsBatchV2\\n2021-04-16 18:51:53,724|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.4__log_batch_v2.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:51:53,797|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient._post_run_metrics_log_failed_validations-async:False|DEBUG|[STOP]\\n2021-04-16 18:51:53,975|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.5__log_batch_v2.WaitingTask|DEBUG|[START]\\n2021-04-16 18:51:53,975|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.5__log_batch_v2.WaitingTask|DEBUG|Awaiter is PostMetricsBatchV2\\n2021-04-16 18:51:53,975|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.5__log_batch_v2.WaitingTask|DEBUG|[STOP]\\n2021-04-16 18:51:53,975|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|Waiting on task: 5__log_batch_v2.\\n1 tasks left. Current duration of flush 0.001222372055053711 seconds.\\n\\n2021-04-16 18:51:53,975|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,975|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2021-04-16 18:51:53,975|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[START]\\n2021-04-16 18:51:53,975|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling wait_on_ingest with url /history/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/experiments/{experimentName}/runs/{runId}/metricsingest/wait\\n2021-04-16 18:51:54,035|azureml._SubmittedRun#vae-no-teacher-forcing-run-410e_1618595463_40f4ddc3.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[STOP]\\n2021-04-16 18:51:59,041|azureml._restclient.clientbase|DEBUG|ClientBase: Calling update_status with url None\\n2021-04-16 18:51:59,094|azureml.SendRunKillSignal|DEBUG|[STOP]\\n2021-04-16 18:51:59,095|azureml.HistoryTrackingWorkerPool.WorkerPoolShutdown|DEBUG|[START]\\n2021-04-16 18:51:59,095|azureml.HistoryTrackingWorkerPool.WorkerPoolShutdown|DEBUG|[STOP]\\n2021-04-16 18:51:59,095|azureml.WorkerPool|DEBUG|[STOP]\\n\\nRun is completed.\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.22.0\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "RunDetails(run).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method Run.wait_for_completion of Run(Experiment: vae-no-teacher-forcing-run-410e,\n",
       "Id: vae-no-teacher-forcing-run-410e_1618575134_dcf82369,\n",
       "Type: azureml.scriptrun,\n",
       "Status: Preparing)>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run.wait_for_completion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python3-azureml"
  },
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}