notebooks/04_deploy_logic_apps.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy Logic Apps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To operationalize our batch scoring workflow, we need a way to trigger our pipeline. Since we're applying style transfer to video data, lets trigger the pipeline everytime a new video is uploaded and detected. To do this, we'll need a mechanism that can detect the appearance of new video data. \n",
    "\n",
    "Logic Apps can solve this problem for us. In this notebook, we'll deploy a pre-built logic app that will look for new videos that appear in a specified storage location. When a new video is detected, the logic app will send an http request to the published pipeline (which we deployed in the previous notebook). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the packages we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Workspace, Run, Experiment\n",
    "from azureml.pipeline.core import PublishedPipeline\n",
    "from azureml.core.datastore import Datastore\n",
    "from dotenv import set_key, get_key, find_dotenv, load_dotenv\n",
    "from azureml.core.authentication import AzureCliAuthentication\n",
    "from pathlib import Path\n",
    "import re\n",
    "import json\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env_path = find_dotenv(raise_error_if_not_found=True)\n",
    "load_dotenv(env_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load our workspace from the config file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "stripout"
    ]
   },
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "print('Workspace name: ' + ws.name, \n",
    "      'Azure region: ' + ws.location, \n",
    "      'Subscription id: ' + ws.subscription_id, \n",
    "      'Resource group: ' + ws.resource_group, sep = '\\n')\n",
    "\n",
    "# Also create a Project and attach to Workspace\n",
    "project_folder = \"scripts\"\n",
    "run_history_name = project_folder\n",
    "\n",
    "if not os.path.isdir(project_folder):\n",
    "    os.mkdir(project_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get authentication information about our published pipeline so that we can use it during the deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "published_pipeline = PublishedPipeline.get(ws, id=get_key(env_path, \"AML_PUBLISHED_PIPELINE_ID\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "cli_auth = AzureCliAuthentication()\n",
    "aad_token = cli_auth.get_authentication_header()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy Logic App"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Logic Apps](https://happypathspublic.blob.core.windows.net/assets/batch_scoring_for_dl/azure_logic_app.jpg)\n",
    "\n",
    "The *logic* behind the Logic App deployment is shown above:\n",
    "1. When a blob is added, begin the workflow.\n",
    "2. Check the blob name. \n",
    "    - if the blob name ends with `.mp4`:\n",
    "        - make a request to the AKS endpoint\n",
    "    - otherwise:\n",
    "        - terminate in cancellation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the deployment for the Azure blob storage connector.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K{- Finished ..\n",
      "  \"id\": \"/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/jiataamltest02/providers/Microsoft.Resources/deployments/blob_connector\",\n",
      "  \"location\": null,\n",
      "  \"name\": \"blob_connector\",\n",
      "  \"properties\": {\n",
      "    \"correlationId\": \"1a6e909e-adb2-4390-9498-e2a37d6274ac\",\n",
      "    \"debugSetting\": null,\n",
      "    \"dependencies\": [],\n",
      "    \"duration\": \"PT4.7165887S\",\n",
      "    \"mode\": \"Incremental\",\n",
      "    \"onErrorDeployment\": null,\n",
      "    \"outputResources\": [\n",
      "      {\n",
      "        \"id\": \"/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/jiataamltest02/providers/Microsoft.Web/connections/azureblob\",\n",
      "        \"resourceGroup\": \"jiataamltest02\"\n",
      "      }\n",
      "    ],\n",
      "    \"outputs\": null,\n",
      "    \"parameters\": {\n",
      "      \"location\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"eastus\"\n",
      "      },\n",
      "      \"storage_account_key\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"uOpI4HeCmvKRzHSbo8qQcQe9z/LvxnIdA+f8oh5R3LlqTW6g3EhOWsn7BKgXq3otHWgdHtrf52QrBrVv+es62A==\"\n",
      "      },\n",
      "      \"storage_account_name\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"jiataamltest02sa\"\n",
      "      },\n",
      "      \"subscription_id\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"edf507a2-6235-46c5-b560-fd463ba2e771\"\n",
      "      }\n",
      "    },\n",
      "    \"parametersLink\": null,\n",
      "    \"providers\": [\n",
      "      {\n",
      "        \"id\": null,\n",
      "        \"namespace\": \"Microsoft.Web\",\n",
      "        \"registrationState\": null,\n",
      "        \"resourceTypes\": [\n",
      "          {\n",
      "            \"aliases\": null,\n",
      "            \"apiVersions\": null,\n",
      "            \"locations\": [\n",
      "              \"eastus\"\n",
      "            ],\n",
      "            \"properties\": null,\n",
      "            \"resourceType\": \"connections\"\n",
      "          }\n",
      "        ]\n",
      "      }\n",
      "    ],\n",
      "    \"provisioningState\": \"Succeeded\",\n",
      "    \"template\": null,\n",
      "    \"templateHash\": \"4552859792189313276\",\n",
      "    \"templateLink\": null,\n",
      "    \"timestamp\": \"2019-01-30T07:57:26.515212+00:00\"\n",
      "  },\n",
      "  \"resourceGroup\": \"jiataamltest02\",\n",
      "  \"type\": null\n",
      "}\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!az group deployment create \\\n",
    "    --name blob_connector \\\n",
    "    --resource-group {ws.resource_group} \\\n",
    "    --template-file template.blob_connector.json \\\n",
    "    --parameters \\\n",
    "        location={get_key(env_path, \"REGION\")} \\\n",
    "        subscription_id={get_key(env_path, \"SUBSCRIPTION_ID\")} \\\n",
    "        storage_account_name={get_key(env_path, \"STORAGE_ACCOUNT_NAME\")} \\\n",
    "        storage_account_key={get_key(env_path, \"STORAGE_ACCOUNT_KEY\")}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the deployment for the Logic App."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K{- Finished ..\n",
      "  \"id\": \"/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/jiataamltest02/providers/Microsoft.Resources/deployments/logic_app\",\n",
      "  \"location\": null,\n",
      "  \"name\": \"logic_app\",\n",
      "  \"properties\": {\n",
      "    \"correlationId\": \"ab0c3b91-1235-4afe-a68f-0fa70f8f0a31\",\n",
      "    \"debugSetting\": null,\n",
      "    \"dependencies\": [],\n",
      "    \"duration\": \"PT6.961205S\",\n",
      "    \"mode\": \"Incremental\",\n",
      "    \"onErrorDeployment\": null,\n",
      "    \"outputResources\": [\n",
      "      {\n",
      "        \"id\": \"/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/jiataamltest02/providers/Microsoft.Logic/workflows/logic_app\",\n",
      "        \"resourceGroup\": \"jiataamltest02\"\n",
      "      }\n",
      "    ],\n",
      "    \"outputs\": null,\n",
      "    \"parameters\": {\n",
      "      \"aad_token\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im5iQ3dXMTF3M1hrQi14VWFYd0tSU0xqTUhHUSIsImtpZCI6Im5iQ3dXMTF3M1hrQi14VWFYd0tSU0xqTUhHUSJ9.eyJhdWQiOiJodHRwczovL21hbmFnZW1lbnQuY29yZS53aW5kb3dzLm5ldC8iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC83MmY5ODhiZi04NmYxLTQxYWYtOTFhYi0yZDdjZDAxMWRiNDcvIiwiaWF0IjoxNTQ4ODMyMzUyLCJuYmYiOjE1NDg4MzIzNTIsImV4cCI6MTU0ODgzNjI1MiwiX2NsYWltX25hbWVzIjp7Imdyb3VwcyI6InNyYzEifSwiX2NsYWltX3NvdXJjZXMiOnsic3JjMSI6eyJlbmRwb2ludCI6Imh0dHBzOi8vZ3JhcGgud2luZG93cy5uZXQvNzJmOTg4YmYtODZmMS00MWFmLTkxYWItMmQ3Y2QwMTFkYjQ3L3VzZXJzLzIzYzljMWEyLWU3MjctNDE4OC1hYTI1LWFmMTA4MmM1ODI0NC9nZXRNZW1iZXJPYmplY3RzIn19LCJhY3IiOiIxIiwiYWlvIjoiQVZRQXEvOEtBQUFBcFdDY3lIb1h5M1J4S3lnbG5YWEpuOWN3Tm5PbnJ3dkhGaXdCdWw3UVNKUGYxVkh0TmxRdGd5cUtuN1FqYTVqTHhVSXV5T1NaSVg5V3ZlWHIxcTVuRS9ONTR3TlNmZGlrTDVXd0E3Um1MQkE9IiwiYW1yIjpbIndpYSIsIm1mYSJdLCJhcHBpZCI6IjA0YjA3Nzk1LThkZGItNDYxYS1iYmVlLTAyZjllMWJmN2I0NiIsImFwcGlkYWNyIjoiMCIsImZhbWlseV9uYW1lIjoiVGFuIiwiZ2l2ZW5fbmFtZSI6IkpTIiwiaXBhZGRyIjoiNDAuMTIxLjE5LjE4OCIsIm5hbWUiOiJKUyBUYW4iLCJvaWQiOiIyM2M5YzFhMi1lNzI3LTQxODgtYWEyNS1hZjEwODJjNTgyNDQiLCJvbnByZW1fc2lkIjoiUy0xLTUtMjEtMjEyNzUyMTE4NC0xNjA0MDEyOTIwLTE4ODc5Mjc1MjctMTgxOTA5NDIiLCJwdWlkIjoiMTAwMzAwMDA5M0Q1QUE2RiIsInNjcCI6InVzZXJfaW1wZXJzb25hdGlvbiIsInN1YiI6IktlRWRuOGJSZjNGYzcwX3lUbUxqRHBocDg4eVpRemFCWlJHRDFNWEFBTDgiLCJ0aWQiOiI3MmY5ODhiZi04NmYxLTQxYWYtOTFhYi0yZDdjZDAxMWRiNDciLCJ1bmlxdWVfbmFtZSI6ImppYXRhQG1pY3Jvc29mdC5jb20iLCJ1cG4iOiJqaWF0YUBtaWNyb3NvZnQuY29tIiwidXRpIjoiMlIyRnA4Z2R1ME9ERUhsNUY0ZEJBQSIsInZlciI6IjEuMCJ9.lCMuYCOiG8kpOJTBkbARhlKTGSMLZ4dfgSzgpZBrLU6mC5OhevF8Rk3r2wGADf-hTSQoxCuViCUBdch3fbs6THIqW9LGSVaWHoy3NekSbGbpJMBSqm_X_hBNj_wefz2tGUvUXv0ZB0n7nerio4uGh-yZwd7S1tfCf2oifrjusdp2MtYwTWH_Hz0_wrR8b_olaCQ953pP4W9Hp8MaCRevmAng_NPKcEruQXV8R2RCSu12eF3JLAmxvh0x6Fv6L1wO0oZy1FrgvJZdO9TxhImiTVqcsMlW9auDWOkdBwvcSNKWxWj9dw1LraTx_UwbT4hvntA-_B6qiRw6ieRe9_xG1A\"\n",
      "      },\n",
      "      \"datastore_name\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"datastore\"\n",
      "      },\n",
      "      \"experiment_name\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"logic_app_experiment\"\n",
      "      },\n",
      "      \"location\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"eastus\"\n",
      "      },\n",
      "      \"name\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"logic_app\"\n",
      "      },\n",
      "      \"resource_group\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"jiataamltest02\"\n",
      "      },\n",
      "      \"storage_container_name\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"aml\"\n",
      "      },\n",
      "      \"subscription_id\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"edf507a2-6235-46c5-b560-fd463ba2e771\"\n",
      "      },\n",
      "      \"url_endpoint\": {\n",
      "        \"type\": \"String\",\n",
      "        \"value\": \"https://eastus.aether.ms/api/v1.0/subscriptions/edf507a2-6235-46c5-b560-fd463ba2e771/resourceGroups/jiataamltest02/providers/Microsoft.MachineLearningServices/workspaces/jiataamltest02/PipelineRuns/PipelineSubmit/fe4d7b79-1835-4725-add4-7d6fd8ae6d47\"\n",
      "      }\n",
      "    },\n",
      "    \"parametersLink\": null,\n",
      "    \"providers\": [\n",
      "      {\n",
      "        \"id\": null,\n",
      "        \"namespace\": \"Microsoft.Logic\",\n",
      "        \"registrationState\": null,\n",
      "        \"resourceTypes\": [\n",
      "          {\n",
      "            \"aliases\": null,\n",
      "            \"apiVersions\": null,\n",
      "            \"locations\": [\n",
      "              \"eastus\"\n",
      "            ],\n",
      "            \"properties\": null,\n",
      "            \"resourceType\": \"workflows\"\n",
      "          }\n",
      "        ]\n",
      "      }\n",
      "    ],\n",
      "    \"provisioningState\": \"Succeeded\",\n",
      "    \"template\": null,\n",
      "    \"templateHash\": \"12356772322557656100\",\n",
      "    \"templateLink\": null,\n",
      "    \"timestamp\": \"2019-01-30T07:58:56.794250+00:00\"\n",
      "  },\n",
      "  \"resourceGroup\": \"jiataamltest02\",\n",
      "  \"type\": null\n",
      "}\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!az group deployment create \\\n",
    "    --name logic_app \\\n",
    "    --resource-group {ws.resource_group} \\\n",
    "    --template-file template.logic_app.json \\\n",
    "    --parameters \\\n",
    "        name=\"logic_app\" \\\n",
    "        location={get_key(env_path, \"REGION\")} \\\n",
    "        resource_group={ws.resource_group} \\\n",
    "        subscription_id={get_key(env_path, \"SUBSCRIPTION_ID\")} \\\n",
    "        storage_container_name={get_key(env_path, \"STORAGE_CONTAINER_NAME\")} \\\n",
    "        url_endpoint={published_pipeline.endpoint} \\\n",
    "        aad_token='{aad_token[\"Authorization\"]}' \\\n",
    "        datastore_name={get_key(env_path, \"AML_DATASTORE_NAME\")} \\\n",
    "        experiment_name=\"logic_app_experiment\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trigger logic app by adding a new video to the Azure blob container"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before testing the logic app by added a new video the blob container, check in the portal or cli that the logic app deployment has completed and that it looks correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp orangutan.mp4 trigger_test_orangutan.mp4\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "$AZUREML_DATAREFERENCE_datastore"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Upload new trigger file video\n",
    "ws.get_default_datastore().upload_files(\n",
    "    [\"./trigger_test_orangutan.mp4\"],\n",
    "    overwrite=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The appearance of the new `trigger_test_orangutan.mp4` video will trigger the Logic App flow. Inspect your logic app in the portal to see the progress."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are now ready to move on to the [next notebook](05_clean_up.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:batchscoringdl_aml]",
   "language": "python",
   "name": "conda-env-batchscoringdl_aml-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}