notebooks/official/pipelines/challenger_vs_blessed_deployment_method.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Challenger vs Blessed methodology for model deployment into production\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/challenger_vs_blessed_deployment_method.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fpipelines%2Fchallenger_vs_blessed_deployment_method.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/pipelines/challenger_vs_blessed_deployment_method.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/challenger_vs_blessed_deployment_method.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial shows how to use Vertex AI Pipelines for deploying the next version of a model into production using the challenger vs blessed method.\n",
        "\n",
        "Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) and [Model evaluation in Vertex AI](https://cloud.google.com/vertex-ai/docs/evaluation/introduction).\n",
        "\n",
        "\n",
        "**_NOTE_**: This notebook uses KFP 1.x and GCPC 1.x. It's recommended to use 2.x versions.\n",
        "\n",
        "**_NOTE_**: This notebook has been tested in the following environment:\n",
        "\n",
        "* Python version = 3.9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to construct a Vertex AI pipeline, which trains a new challenger version of a model, evaluates the model and compares the evaluation to the existing blessed model in production. Then, it determines whether the challenger model becomes the blessed model for replacement in production.\n",
        "\n",
        "This tutorial uses the following Vertex AI services and resources:\n",
        "\n",
        "- Vertex AI Pipeline\n",
        "- Vertex AI Model Evaluation\n",
        "- Vertex AI Model Registry\n",
        "- Vertex AI Endpoints\n",
        "\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Import a pretrained (blessed) model to the Vertex AI Model Registry.\n",
        "- Import synthetic model evaluation metrics to the corresponding (blessed) model.\n",
        "- Create a Vertex AI endpoint resource\n",
        "- Deploy the blessed model to the endpoint resource.\n",
        "- Create a Vertex AI Pipeline that runs the following steps:\n",
        "    - Get the blessed model.\n",
        "    - Import another instance (challenger) of the pretrained model.\n",
        "    - Register the pretrained (challenger) model as a new version of the existing blessed model.\n",
        "    - Create a synthetic model evaluation.\n",
        "    - Import the synthetic model evaluation metrics to the corresponding challenger model.\n",
        "    - Compare the evaluations and set the blessed or challenger model as the default.\n",
        "    - Deploy the new blessed model.\n",
        "\n",
        "Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08d289fa873f"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is the [CIFAR10 dataset](https://www.tensorflow.org/datasets/catalog/cifar10) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you use is built into TensorFlow. The trained model predicts which type of class an image is from ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, or truck."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), \n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f0316df526f8"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "outputs": [],
      "source": [
        "# Install the packages\n",
        "! pip3 install --upgrade google-cloud-aiplatform \\\n",
        "                                'google-cloud-pipeline-components<2'\n",
        "! pip3 install tensorflow==2.5 \\\n",
        "                tensorflow_hub\n",
        "  \n",
        "! pip3 install --upgrade 'kfp<2'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ee775571c2b5"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92e68cfc3a90"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "46604f70e831"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "### Service Account\n",
        "\n",
        "You use a service account to create Vertex AI Pipeline jobs.\n",
        "\n",
        "If you don't want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "IS_COLAB = \"google.colab\" in sys.modules\n",
        "\n",
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your service account from gcloud\n",
        "    if not IS_COLAB:\n",
        "        shell_output = !gcloud auth list 2>/dev/null\n",
        "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
        "\n",
        "    else:  # IS_COLAB:\n",
        "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
        "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
        "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
        "\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Pipelines\n",
        "\n",
        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run these once per service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "960505627ddf"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import kfp\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from google.cloud import aiplatform\n",
        "from google.cloud.aiplatform import gapic\n",
        "from kfp.v2 import compiler\n",
        "from kfp.v2.dsl import component"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com) in your Google Cloud project.\n",
        "\n",
        "Then, initialize the Vertex AI SDK for Python for your project."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "source": [
        "#### Set hardware accelerators\n",
        "\n",
        "You can set hardware accelerators for training and prediction.\n",
        "\n",
        "Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa T4 GPUs allocated to each VM, you would specify:\n",
        "\n",
        "    (aip.gapic.AcceleratorType.NVIDIA_TESLA_T4, 4)\n",
        "\n",
        "\n",
        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
        "\n",
        "Learn more about [hardware accelerator support for your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "outputs": [],
      "source": [
        "DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "container:training,prediction"
      },
      "source": [
        "#### Set prebuilt containers\n",
        "\n",
        "Set the prebuilt Docker container image for prediction.\n",
        "\n",
        "For the latest list, see [Prebuilt containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "container:training,prediction"
      },
      "outputs": [],
      "source": [
        "TF = \"2.5\".replace(\".\", \"-\")\n",
        "\n",
        "if DEPLOY_GPU:\n",
        "    DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n",
        "else:\n",
        "    DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n",
        "\n",
        "DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
        "    LOCATION.split(\"-\")[0], DEPLOY_VERSION\n",
        ")\n",
        "\n",
        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "machine:training"
      },
      "source": [
        "#### Set machine type\n",
        "\n",
        "Next, set the machine type to use for deployment.\n",
        "\n",
        "- Set the variable `DEPLOY_COMPUTE` to configure the compute resources for the VMs you use for deployment.\n",
        " - `machine_type`\n",
        "     - `n1-standard`: 3.75GB of memory per vCPU\n",
        "     - `n1-highmem`: 6.5GB of memory per vCPU\n",
        "     - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
        " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
        "\n",
        "**Note**: The following aren't supported.\n",
        "\n",
        " - `standard`: 2 vCPUs\n",
        " - `highcpu`: 2, 4 and 8 vCPUs\n",
        "\n",
        "**Note**: You may also use n2 and e2 machine types for training and deployment, but they don't support GPUs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "machine:training"
      },
      "outputs": [],
      "source": [
        "machine_type = \"n1-standard\"\n",
        "vCPUs = \"4\"\n",
        "DEPLOY_COMPUTE = f\"{machine_type}-{vCPUs}\"\n",
        "print(\"Deploy machine type\", DEPLOY_COMPUTE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d8128b8ff025"
      },
      "source": [
        "## Get pretrained model from TensorFlow Hub\n",
        "\n",
        "For demonstration purposes, this tutorial uses a pretrained model from TensorFlow Hub (TFHub), which is then uploaded to a Vertex AI model resource. Once you have a Vertex AI model resource, the model can be deployed to a Vertex AI endpoint resource.\n",
        "\n",
        "### Download the pretrained model\n",
        "\n",
        "First, you download the pretrained model from TensorFlow Hub. The model gets downloaded as a TF.Keras layer. To finalize the model, in this example, you create a `Sequential()` model with the downloaded TFHub model as a layer, and specify the input shape to the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c55fa4c826f7"
      },
      "outputs": [],
      "source": [
        "tfhub_model = tf.keras.Sequential(\n",
        "    [hub.KerasLayer(\"https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5\")]\n",
        ")\n",
        "\n",
        "tfhub_model.build([None, 32, 32, 3])\n",
        "\n",
        "tfhub_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "63de49055083"
      },
      "source": [
        "### Save the model artifacts\n",
        "\n",
        "At this point, the model is in memory. Next, you save the model artifacts to a Cloud Storage location."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "64618c713db9"
      },
      "outputs": [],
      "source": [
        "MODEL_DIR = BUCKET_URI + \"/model\"\n",
        "tfhub_model.save(MODEL_DIR)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e8ce91147c93"
      },
      "source": [
        "## Upload the TensorFlow Hub model to Vertex AI Model Registry\n",
        "\n",
        "Finally, you upload the model artifacts from the TFHub model into Vertex AI Model Registry and get a model resource object using the `upload()` method, with the following parameters:\n",
        "\n",
        "- `display_name`: A human readable name for the model resource.\n",
        "- `artifact_uri`: The Cloud Storage location of the model package.\n",
        "- `serving_container_image_uri`: The serving container image.\n",
        "\n",
        "Uploading a model into Vertex AI Model Registry returns a long running operation, since it may take a few moments. \n",
        "\n",
        "**Note**: When you upload the model artifacts to Vertex AI Model Registry, you specify the corresponding deployment container image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ad61e1429512"
      },
      "outputs": [],
      "source": [
        "blessed_model = aiplatform.Model.upload(\n",
        "    display_name=\"resnet\",\n",
        "    artifact_uri=MODEL_DIR,\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        "    is_default_version=True,\n",
        "    version_aliases=[\"v1\"],\n",
        ")\n",
        "\n",
        "print(blessed_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c11e98ef5391"
      },
      "source": [
        "## Create a model evaluation\n",
        "\n",
        "First, you create a model evaluation in a format that corresponds to one of the predefined schemas for model evaluations. In this example, you use the schema for a classification metric, and specify the following subset of evaluation metrics as a dictionary:\n",
        "\n",
        "- `logLoss`: The log loss.\n",
        "- `auPrc`: The accuracy.\n",
        "\n",
        "You then construct the `ModelEvaluation` object with the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the evaluation metric.\n",
        "- `metrics_schema_uri`: The schema for the specific type of evaluation metrics.\n",
        "- `metrics`: The dictionary with the evaluation metrics.\n",
        "\n",
        "Learn more about [Schemas for evaluation metrics](https://cloud.google.com/vertex-ai/docs/evaluation/introduction#features)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e9af222db292"
      },
      "outputs": [],
      "source": [
        "metrics = {\"logLoss\": 1.4, \"auPrc\": 0.85}\n",
        "print(metrics)\n",
        "\n",
        "blessed_eval = gapic.ModelEvaluation(\n",
        "    display_name=\"eval\",\n",
        "    metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
        "    metrics=metrics,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68870bd8194d"
      },
      "source": [
        "### Upload the evaluation metrics to the Model Registry\n",
        "\n",
        "Next, upload the model's evaluation from the custom training job to the corresponding entry in the Vertex AI Model Registry.\n",
        "\n",
        "Currently, the Vertex AI Python SDK has no support for this method. Instead, you use the lower level GAPIC API interface."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3044788848af"
      },
      "outputs": [],
      "source": [
        "API_ENDPOINT = f\"{LOCATION}-aiplatform.googleapis.com\"\n",
        "client = gapic.ModelServiceClient(client_options={\"api_endpoint\": API_ENDPOINT})\n",
        "\n",
        "client.import_model_evaluation(\n",
        "    parent=blessed_model.resource_name, model_evaluation=blessed_eval\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "628de0914ba1"
      },
      "source": [
        "## Creating an endpoint resource\n",
        "\n",
        "You create an endpoint resource using the `Endpoint.create()` method. At a minimum, you specify the display name for the endpoint. Optionally, you can specify the project and location (region); otherwise the settings are inherited by the values you set when you initialized the Vertex AI Python SDK with the `init()` method.\n",
        "\n",
        "In this example, the following parameters are specified:\n",
        "\n",
        "- `display_name`: A human readable name for the endpoint resource.\n",
        "- `project`: Your project ID.\n",
        "- `location`: Your region.\n",
        "\n",
        "This method returns an `Endpoint` object.\n",
        "\n",
        "Learn more about [Vertex AI Endpoints](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0ea443f9593b"
      },
      "outputs": [],
      "source": [
        "endpoint = aiplatform.Endpoint.create(\n",
        "    display_name=\"resnet\", project=PROJECT_ID, location=LOCATION\n",
        ")\n",
        "\n",
        "print(endpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ca3fa3f6a894"
      },
      "source": [
        "## Deploy the model resource to the endpoint resource\n",
        "\n",
        "Next, you deploy the blessed Vertex AI model resource to a Vertex AI endpoint resource. The container image defined for the Vertex AI model resource is used for deployment. To deploy, you specify the following additional configuration settings:\n",
        "\n",
        "- The machine type.\n",
        "- The (if any) type and number of GPUs.\n",
        "- Static, manual or auto-scaling of VM instances.\n",
        "\n",
        "In this example, you deploy the model with the minimal amount of specified parameters, as follows:\n",
        "\n",
        "- `model`: The model resource.\n",
        "- `deployed_model_displayed_name`: The human readable name for the deployed model instance.\n",
        "- `machine_type`: The machine type for each VM instance.\n",
        "\n",
        "This may take upto a few minutes due to provisioning of the configured resources."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4e93b034a72f"
      },
      "outputs": [],
      "source": [
        "response = endpoint.deploy(\n",
        "    model=blessed_model,\n",
        "    deployed_model_display_name=\"resnet\",\n",
        "    machine_type=DEPLOY_COMPUTE,\n",
        ")\n",
        "\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1899f5ca18a9"
      },
      "source": [
        "## Create custom components for pipeline\n",
        "\n",
        "Next, you create several custom components that you use in your pipeline.\n",
        "\n",
        "### Create component to upload the next version of the model\n",
        "\n",
        "First, you define a component to upload the trained challenger model as a version to the blessed model in the Vertex AI Model Registry. The component takes the following arguments:\n",
        "\n",
        "- `parent_model`: The full resource name of the blessed model.\n",
        "- `artifact_uri`: The Cloud Storage location of the model artifacts for the challenger model.\n",
        "- `serving_container`: The serving container for the challenger model.\n",
        "- `project`: Your Project ID.\n",
        "- `region`: The region where you want to create or use the resources."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ee3cd58c1671"
      },
      "outputs": [],
      "source": [
        "@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
        "def create_next_model_version(\n",
        "    parent_model: str,\n",
        "    artifact_uri: str,\n",
        "    serving_container: str,\n",
        "    project: str,\n",
        "    region: str,\n",
        ") -> str:\n",
        "    from google.cloud import aiplatform\n",
        "\n",
        "    aiplatform.init(project=project, location=region)\n",
        "\n",
        "    model = aiplatform.Model.upload(\n",
        "        display_name=\"resnet\",\n",
        "        artifact_uri=artifact_uri,\n",
        "        serving_container_image_uri=serving_container,\n",
        "        parent_model=parent_model,\n",
        "        is_default_version=True,\n",
        "        version_aliases=[\"v2\"],\n",
        "        version_description=\"This is the second version of the model\",\n",
        "    )\n",
        "\n",
        "    return model.resource_name"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7e4fb366d34a"
      },
      "source": [
        "### Create component to import classification metrics\n",
        "\n",
        "Next, you define a component to import the evaluation metrics for the challenger model to the Model Registry. The component takes the following arguments:\n",
        "\n",
        "- `display_name`: Human readable name for the evaluation metrics.\n",
        "- `metrics`: The evaluation metrics formatted for classification.\n",
        "- `parent_model_resource`: The full resource name for the challenger model version.\n",
        "- `region`: The region where you want to create or use the resources."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1af51a0b41ce"
      },
      "outputs": [],
      "source": [
        "@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
        "def import_classification_metrics(\n",
        "    display_name: str, metrics: dict, parent_model_resource: str, region: str\n",
        "):\n",
        "    from google.cloud.aiplatform import gapic\n",
        "\n",
        "    evaluation = gapic.ModelEvaluation(\n",
        "        display_name=display_name,\n",
        "        metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
        "        metrics=metrics,\n",
        "    )\n",
        "\n",
        "    API_ENDPOINT = f\"{region}-aiplatform.googleapis.com\"\n",
        "    client = gapic.ModelServiceClient(client_options={\"api_endpoint\": API_ENDPOINT})\n",
        "    client.import_model_evaluation(\n",
        "        parent=parent_model_resource, model_evaluation=evaluation\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5a49c33c9e12"
      },
      "source": [
        "###  Create component to compare metrics\n",
        "\n",
        "Next, you define a component to compare the `auPrc` metric between two versions of a model. In this case, you compare between the current blessed model and the new challenger model. Whichever of the two has the best `auPrc` value is set as the default model. When you subsequently deploy, the default model is deployed. The component takes the following arguments:\n",
        "\n",
        "- `blessed_model_resource_name`: The full resource name of the blessed model.\n",
        "- `challenger_model_resource_name`: The full resource name of the challenger model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e70c466ae39a"
      },
      "outputs": [],
      "source": [
        "@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
        "def compare_metrics(\n",
        "    blessed_model_resource_name: str, challenger_model_resource_name: str\n",
        "):\n",
        "    from google.cloud import aiplatform\n",
        "\n",
        "    # Get the metrics for the blessed model\n",
        "    blessed_model = aiplatform.Model(blessed_model_resource_name)\n",
        "    blessed_eval = blessed_model.list_model_evaluations()[0]\n",
        "    blessed_auPrc = blessed_eval.metrics[\"auPrc\"]\n",
        "\n",
        "    # Get the metrics for the challenger model\n",
        "    challenger_model = aiplatform.Model(challenger_model_resource_name)\n",
        "    challenger_eval = challenger_model.list_model_evaluations()[0]\n",
        "    challenger_auPrc = challenger_eval.metrics[\"auPrc\"]\n",
        "\n",
        "    # Which model has the best accuracy becomes the default model\n",
        "    if challenger_auPrc > blessed_auPrc:\n",
        "        challenger_model.versioning_registry.add_version_aliases(\n",
        "            new_aliases=[\"default\"], version=challenger_model.version_id\n",
        "        )\n",
        "    else:\n",
        "        blessed_model.versioning_registry.add_version_aliases(\n",
        "            new_aliases=[\"default\"], version=blessed_model.version_id\n",
        "        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8a0cd316dc5e"
      },
      "source": [
        "## Blessed vs challenger pipeline\n",
        "\n",
        "In this section, you construct a pipeline to fetch the blessed, and challenger models and compare them to deploy the best one to the production endpoint.\n",
        "\n",
        "\n",
        "### Define the pipeline\n",
        "\n",
        "Your pipeline runs the following tasks:\n",
        "\n",
        "- Get the blessed version of a model.\n",
        "- Get the endpoint for the deployed blessed model.\n",
        "- Train (faked) the challenger model.\n",
        "- Upload the challenger model as the next version of the model.\n",
        "- Import the challenger model's evaluation metrics.\n",
        "- Compare the blessed and challenger evaluation metrics, and set the default accordingly.\n",
        "- Import the existing production endpoint.\n",
        "- Deploy the new default model to the production endpoint."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4b52fc0e5a8c"
      },
      "outputs": [],
      "source": [
        "@kfp.dsl.pipeline(name=\"blessed-vs-challenger\")\n",
        "def pipeline(\n",
        "    blessed_model_resource: str,\n",
        "    serving_container: str,\n",
        "    machine_type: str,\n",
        "    endpoint_resource_name: str,\n",
        "    endpoint_resource_uri: str,\n",
        "    project: str = PROJECT_ID,\n",
        "    region: str = LOCATION,\n",
        "):\n",
        "    from google_cloud_pipeline_components.experimental.evaluation import \\\n",
        "        GetVertexModelOp\n",
        "    from google_cloud_pipeline_components.types import artifact_types\n",
        "    from google_cloud_pipeline_components.v1.endpoint import ModelDeployOp\n",
        "    from kfp.v2.components import importer_node\n",
        "\n",
        "    # Get the Vertex AI model resource of the blessed model\n",
        "    model = GetVertexModelOp(model_resource_name=blessed_model_resource)\n",
        "\n",
        "    # pretend that you trained a new version of the model (artifacts at MODEL_DIR)\n",
        "\n",
        "    # create the next version in the Model Registry as the challenger model\n",
        "    next_version = create_next_model_version(\n",
        "        parent_model=blessed_model_resource,\n",
        "        artifact_uri=MODEL_DIR,\n",
        "        serving_container=serving_container,\n",
        "        project=project,\n",
        "        region=region,\n",
        "    ).after(model)\n",
        "\n",
        "    # pretend to evaluate the challenger\n",
        "    challenger_metrics = {\"logLoss\": 1.3, \"auPrc\": 0.88}\n",
        "\n",
        "    # upload the metrics for the challenger version\n",
        "    import_metrics = import_classification_metrics(\n",
        "        display_name=\"challenger\",\n",
        "        metrics=challenger_metrics,\n",
        "        parent_model_resource=next_version.output,\n",
        "        region=region,\n",
        "    ).after(next_version)\n",
        "\n",
        "    # test metrics\n",
        "    compare = compare_metrics(blessed_model_resource, next_version.output).after(\n",
        "        import_metrics\n",
        "    )\n",
        "\n",
        "    # import the production Endpoint\n",
        "    endpoint = importer_node.importer(\n",
        "        artifact_uri=endpoint_resource_uri,\n",
        "        artifact_class=artifact_types.VertexEndpoint,\n",
        "        metadata={\"resourceName\": endpoint_resource_name},\n",
        "    )\n",
        "\n",
        "    # deploy model to endpoint\n",
        "    _ = ModelDeployOp(\n",
        "        model=model.outputs[\"model\"],\n",
        "        endpoint=endpoint.output,  # .outputs[\"endpoint\"],\n",
        "        dedicated_resources_min_replica_count=1,\n",
        "        dedicated_resources_max_replica_count=1,\n",
        "        dedicated_resources_machine_type=machine_type,\n",
        "        traffic_split={\"0\": 100},\n",
        "    ).after(compare)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_custom_train_pipeline:icn"
      },
      "source": [
        "### Compile the pipeline\n",
        "\n",
        "Next, you compile the pipeline to a JSON file. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2521d9cef73d"
      },
      "outputs": [],
      "source": [
        "# Compile the pipeline to a json file\n",
        "compiler.Compiler().compile(\n",
        "    pipeline_func=pipeline, package_path=\"challenger_vs_blessed.json\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ceed9f75a9d"
      },
      "source": [
        "### Execute the pipeline\n",
        "\n",
        "Finally, you execute your pipeline, passing the following pipeline parameter values:\n",
        "\n",
        "- `blessed_model_resource`: The full resource name of the current blessed version of the model.\n",
        "- `serving_container`: The serving container for deploying the model.\n",
        "- `machine_type`: The machine type for deploying the model.\n",
        "- `endpoint_resource_name`: The full resource name of the production endpoint.\n",
        "- `endpoint_resource_uri`: The full URI for the production endpoint.\n",
        "- `project`:The project ID.\n",
        "- `region`: The region where you want to create or use the resources."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_pipeline:control"
      },
      "outputs": [],
      "source": [
        "# Define the root folder for your pipeline artifacts\n",
        "PIPELINE_ROOT = \"{}/pipeline_root/control\".format(BUCKET_URI)\n",
        "\n",
        "# Define the pipeline job\n",
        "job = aiplatform.PipelineJob(\n",
        "    display_name=\"challenger_vs_blessed\",\n",
        "    template_path=\"challenger_vs_blessed.json\",\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    parameter_values={\n",
        "        \"blessed_model_resource\": blessed_model.resource_name,\n",
        "        \"serving_container\": DEPLOY_IMAGE,\n",
        "        \"machine_type\": DEPLOY_COMPUTE,\n",
        "        \"endpoint_resource_name\": endpoint.resource_name,\n",
        "        \"endpoint_resource_uri\": \"https://us-central1-aiplatform.googleapis.com/v1/\"\n",
        "        + endpoint.resource_name,\n",
        "        \"project\": PROJECT_ID,\n",
        "        \"region\": LOCATION,\n",
        "    },\n",
        "    enable_caching=False,\n",
        ")\n",
        "\n",
        "# Run the pipeline job\n",
        "job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e1d089d1d0d9"
      },
      "source": [
        "## Get the latest state of the production endpoint\n",
        "\n",
        "Once your pipeline execution is finished, notice that the challenger (version 2) model has replaced the previous blessed model on the production endpoint.\n",
        "\n",
        "Now, display the latest details of the deployed models from the production endpoint, and then display the traffic split. Notice the resource ID for the 100% entry is the resource ID for the challenger (v2) model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9dbf991841d9"
      },
      "outputs": [],
      "source": [
        "gca_resource = endpoint.list(filter=\"display_name=resnet\")[0].gca_resource\n",
        "print(\"Deployed Models\", gca_resource.deployed_models)\n",
        "print(\"\\n\")\n",
        "print(\"Traffic Split\", gca_resource.traffic_split)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "# Undeploy the model from endpoint\n",
        "endpoint.undeploy_all()\n",
        "\n",
        "# Delete the endpoint resource\n",
        "endpoint.delete()\n",
        "\n",
        "# Delete the model resource(s)\n",
        "blessed_model.delete()\n",
        "\n",
        "# Delete the pipeline job\n",
        "job.delete()\n",
        "\n",
        "# Delete Cloud Storage objects that were created\n",
        "delete_bucket = True\n",
        "if delete_bucket:\n",
        "    ! gsutil -m rm -r $BUCKET_URI\n",
        "\n",
        "# Delete the locally saved pipeline package file\n",
        "! rm challenger_vs_blessed.json"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "challenger_vs_blessed_deployment_method.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}