notebooks/official/explainable_ai/sdk_automl_tabular_classification_online_explain.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title"
      },
      "source": [
        "# Vertex SDK: AutoML training tabular classification model for online explanation\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/explainable_ai/sdk_automl_tabular_classification_online_explain.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fexplainable_ai%2Fsdk_automl_tabular_classification_online_explain.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/explainable_ai/sdk_automl_tabular_classification_online_explain.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/explainable_ai/sdk_automl_tabular_classification_online_explain.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:automl,xai"
      },
      "source": [
        "## Overview\n",
        "\n",
        "\n",
        "This tutorial demonstrates how to use the Vertex AI SDK to create tabular classification models and get online predictions with explanation using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users) model.\n",
        "\n",
        "Learn more about [Classification for tabular data](https://cloud.google.com/vertex-ai/docs/tabular-data/classification-regression/overview) and [Vertex Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "objective:automl,training,online_prediction,xai"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use AutoML to create a tabular binary classification model from a Python script. Then, you learn to use the Vertex AI prediction service to make a online prediction request with explanations. You can alternatively create and deploy models using the `gcloud` command-line tool or online using the Cloud Console.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services:\n",
        "\n",
        "- Vertex AI AutoML\n",
        "- Vertex AI Prediction\n",
        "- Vertex Explainable AI\n",
        "- Vertex AI model resource\n",
        "- Vertex AI endpoint resource\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Create a Vertex AI dataset resource.\n",
        "- Train an AutoML tabular binary classification model.\n",
        "- View the model evaluation metrics for the trained model.\n",
        "- Create a serving endpoint resource.\n",
        "- Deploy the Model resource to a serving endpoint resource.\n",
        "- Make an online prediction request with explainability.\n",
        "- Undeploy the Model resource."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:iris,lcn"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is the [Iris dataset](https://www.tensorflow.org/datasets/catalog/iris) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The dataset does'nt require any feature engineering. The version of the dataset you use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of Iris flower species from a class of three species: setosa, virginica, or versicolor."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform \\\n",
        "                                 google-cloud-storage \n",
        "! pip3 install --upgrade --quiet tensorflow  \n",
        "! pip3 install -U --quiet tabulate  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project. Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nqwi-5ufWp_B"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "53c6c71ac857"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "673ef17c3cff"
      },
      "outputs": [],
      "source": [
        "from google.cloud import aiplatform\n",
        "\n",
        "aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tutorial_start:automl"
      },
      "source": [
        "# Tutorial\n",
        "\n",
        "Now you're ready to create your own AutoML tabular classification model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "import_file:u_dataset,csv"
      },
      "source": [
        "#### Location of Cloud Storage training data\n",
        "\n",
        "Set `IMPORT_FILE` variable to the location of the CSV index file in Cloud Storage."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_file:iris,csv,lcn"
      },
      "outputs": [],
      "source": [
        "IMPORT_FILE = \"gs://cloud-samples-data/tables/iris_1000.csv\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "quick_peek:tabular"
      },
      "source": [
        "#### Quick peek at your data\n",
        "\n",
        "You can count the number of examples by counting the number of rows in the CSV index file  (`wc -l`) and then take a peek at the first few rows.\n",
        "\n",
        "The heading name of the label column, is `label_column`, which is the last column in the CSV file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "quick_peek:tabular"
      },
      "outputs": [],
      "source": [
        "count = ! gsutil cat $IMPORT_FILE | wc -l\n",
        "print(\"Number of Examples\", int(count[0]))\n",
        "\n",
        "print(\"First 10 rows\")\n",
        "! gsutil cat $IMPORT_FILE | head\n",
        "\n",
        "heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
        "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
        "print(\"Label Column Name\", label_column)\n",
        "if label_column is None:\n",
        "    raise Exception(\"label column missing\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_dataset:tabular,lcn"
      },
      "source": [
        "### Create the Dataset\n",
        "\n",
        "Next, create the dataset using the `create` method for the `TabularDataset` class, which takes the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the dataset.\n",
        "- `gcs_source`: A list of one or more dataset index files to import the data items into the dataset.\n",
        "- `bq_source`: Alternatively, you can import data items from a BigQuery table into the dataset.\n",
        "\n",
        "This operation may take several minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_dataset:tabular,lcn"
      },
      "outputs": [],
      "source": [
        "dataset = aiplatform.TabularDataset.create(\n",
        "    display_name=\"Iris\", gcs_source=[IMPORT_FILE]\n",
        ")\n",
        "\n",
        "print(dataset.resource_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_automl_pipeline:tabular,lcn"
      },
      "source": [
        "### Create and run training pipeline\n",
        "\n",
        "To train an AutoML model, you need perform two steps:\n",
        "1) Create a training pipeline.\n",
        "2) Specify your training parameters and run the job.\n",
        "\n",
        "#### Create training pipeline\n",
        "\n",
        "Create an AutoML training pipeline using the `AutoMLTabularTrainingJob` class, with the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the training job.\n",
        "- `optimization_prediction_type`: The type of prediction the model is to produce.\n",
        "  - `classification`: A tabuar classification model.\n",
        "  - `regression`: A tabular regression model.\n",
        "- `column_transformations`: (Optional): Transformations to apply to the input columns\n",
        "- `optimization_objective`: The optimization objective to minimize or maximize.\n",
        "  - binary classification:\n",
        "    - `minimize-log-loss`\n",
        "    - `maximize-au-roc`\n",
        "    - `maximize-au-prc`\n",
        "    - `maximize-precision-at-recall`\n",
        "    - `maximize-recall-at-precision`\n",
        "  - multi-class classification:\n",
        "    - `minimize-log-loss`\n",
        "  - regression:\n",
        "    - `minimize-rmse`\n",
        "    - `minimize-mae`\n",
        "    - `minimize-rmsle`\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_automl_pipeline:tabular,lcn"
      },
      "outputs": [],
      "source": [
        "job = aiplatform.AutoMLTabularTrainingJob(\n",
        "    display_name=\"iris\",\n",
        "    optimization_prediction_type=\"classification\",\n",
        "    optimization_objective=\"minimize-log-loss\",\n",
        ")\n",
        "\n",
        "print(job)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_automl_pipeline:tabular"
      },
      "source": [
        "#### Run the training pipeline\n",
        "\n",
        "Start the training job by invoking the `run` method, with the following parameters:\n",
        "\n",
        "- `dataset`: The dataset to train the model.\n",
        "- `model_display_name`: The human readable name for the trained model.\n",
        "- `training_fraction_split`: The percentage of the dataset to use for training.\n",
        "- `test_fraction_split`: The percentage of the dataset to use for test (holdout data).\n",
        "- `validation_fraction_split`: The percentage of the dataset to use for validation.\n",
        "- `target_column`: The name of the column to train as the label.\n",
        "- `budget_milli_node_hours`: (optional) Maximum training time specified in unit of millihours (1000 = hour).\n",
        "- `disable_early_stopping`: If `True`, training maybe completed before using the entire budget if the service believes it cannot further improve on the model objective measurements.\n",
        "\n",
        "The `run` method when completed returns the model resource.\n",
        "\n",
        "The execution of the training pipeline may take upto 20 minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_automl_pipeline:tabular"
      },
      "outputs": [],
      "source": [
        "model = job.run(\n",
        "    dataset=dataset,\n",
        "    model_display_name=\"iris\",\n",
        "    training_fraction_split=0.6,\n",
        "    validation_fraction_split=0.2,\n",
        "    test_fraction_split=0.2,\n",
        "    budget_milli_node_hours=8000,\n",
        "    disable_early_stopping=False,\n",
        "    target_column=label_column,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "evaluate_the_model:mbsdk"
      },
      "source": [
        "## Review model evaluation scores\n",
        "\n",
        "After your model training is complete, you can review the evaluation scores using the `list_model_evaluations()` method. This method returns an iterator for each evaluation slice."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "evaluate_the_model:mbsdk"
      },
      "outputs": [],
      "source": [
        "model_evaluations = model.list_model_evaluations()\n",
        "\n",
        "for model_evaluation in model_evaluations:\n",
        "    print(model_evaluation.to_dict())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "deploy_model:mbsdk,dedicated"
      },
      "source": [
        "## Deploy the model\n",
        "\n",
        "Next, deploy your model for online prediction. To deploy the model, invoke the `deploy` method, with the following parameters:\n",
        "\n",
        "- `machine_type`: The type of compute machine."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "deploy_model:mbsdk,dedicated"
      },
      "outputs": [],
      "source": [
        "endpoint = model.deploy(machine_type=\"n1-standard-4\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "make_prediction:xai"
      },
      "source": [
        "## Send an online prediction request\n",
        "\n",
        "Send a online prediction with explainability to your deployed model. In this method, the predicted response includes an explanation on how the features contributed to the explanation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "make_test_item:automl,online_prediction"
      },
      "source": [
        "### Create test item\n",
        "\n",
        "For demostration purpose, use synthetic data to generate predictions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "make_test_item:automl,tabular,iris"
      },
      "outputs": [],
      "source": [
        "INSTANCE = {\n",
        "    \"petal_length\": \"1.4\",\n",
        "    \"petal_width\": \"1.3\",\n",
        "    \"sepal_length\": \"5.1\",\n",
        "    \"sepal_width\": \"2.8\",\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "explain_request:mbsdk,lcn"
      },
      "source": [
        "### Generate predictions with explanations\n",
        "\n",
        "Now that your model is deployed to an endpoint, you can get online explanations by sending prediction requests to your endpoint.\n",
        "\n",
        "#### Request\n",
        "\n",
        "The format of each instance is:\n",
        "\n",
        "    [feature_list]\n",
        "\n",
        "Since the `explain()` method can take multiple items (instances), send your single test item as a list of one test item.\n",
        "\n",
        "#### Response\n",
        "\n",
        "The response from the `explain()` call is a Python dictionary with the following entries:\n",
        "\n",
        "- `ids`: The internal assigned unique identifiers for each prediction request.\n",
        "- `displayNames`: The class names for each class label.\n",
        "- `confidences`: For classification, the predicted confidence, between 0 and 1, per class label.\n",
        "- `values`: For regression, the predicted value.\n",
        "- `deployed_model_id`: The Vertex AI identifier for the deployed model resource which generated the predictions.\n",
        "- `explanations`: The feature attributions returned by Explainable AI."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "explain_request:mbsdk,lcn"
      },
      "outputs": [],
      "source": [
        "instances_list = [INSTANCE]\n",
        "\n",
        "prediction = endpoint.explain(instances_list)\n",
        "print(prediction)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "understand_explanations"
      },
      "source": [
        "### Understanding the explanations response\n",
        "\n",
        "Take a look at model predictions and compare it to the actual value."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "understand_explanations:mbsdk,lcn"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "try:\n",
        "    label = np.argmax(prediction[0][0][\"scores\"])\n",
        "    cls = prediction[0][0][\"classes\"][label]\n",
        "    print(\"Predicted Value:\", cls, prediction[0][0][\"scores\"][label])\n",
        "except:\n",
        "    pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "examine_feature_attributions"
      },
      "source": [
        "### Examine feature attributions\n",
        "\n",
        "Next look at the feature attributions for this particular example. Positive attribution values mean a particular feature pushed your model prediction up by that amount, and vice versa for negative attribution values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "examine_feature_attributions:mbsdk,iris"
      },
      "outputs": [],
      "source": [
        "from tabulate import tabulate\n",
        "\n",
        "feature_names = [\"sepal_length\", \"sepal_width\", \"petal_length\", \"petal_width\"]\n",
        "attributions = prediction.explanations[0].attributions[0].feature_attributions\n",
        "\n",
        "rows = []\n",
        "for i, val in enumerate(feature_names):\n",
        "    rows.append([val, INSTANCE[val], attributions[val]])\n",
        "print(tabulate(rows, headers=[\"Feature name\", \"Feature value\", \"Attribution value\"]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "check_explanations_baselines"
      },
      "source": [
        "### Check your explanations and baselines\n",
        "\n",
        "To make better sense of the feature attributions you're getting, you should compare them with your model's baseline. In most cases, the sum of your attribution values + the baseline should be very close to your model's predicted value for each input. Also note that for regression models, the `baseline_score` returned from explanations remain the same for each example sent to your model. For classification models, each class has its own baseline.\n",
        "\n",
        "In this section, send 10 test examples to your model for prediction in order to compare the feature attributions with the baseline. Then, run each test example's attributions through a sanity check in the `sanity_check_explanations` method.\n",
        "\n",
        "#### Get explanations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "check_explanations_baselines:mbsdk,iris"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "\n",
        "# Prepare 10 test examples to your model for prediction using a random distribution to generate\n",
        "# test instances\n",
        "instances = []\n",
        "for i in range(10):\n",
        "    pl = str(random.uniform(1.0, 2.0))\n",
        "    pw = str(random.uniform(1.0, 2.0))\n",
        "    sl = str(random.uniform(4.0, 6.0))\n",
        "    sw = str(random.uniform(2.0, 4.0))\n",
        "    instances.append(\n",
        "        {\"petal_length\": pl, \"petal_width\": pw, \"sepal_length\": sl, \"sepal_width\": sw}\n",
        "    )\n",
        "\n",
        "response = endpoint.explain(instances)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sanity_check_explanations"
      },
      "source": [
        "#### Sanity check\n",
        "\n",
        "In the function below you perform a sanity check on the explanations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sanity_check_explanations"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "\n",
        "def sanity_check_explanations(\n",
        "    explanation, prediction, mean_tgt_value=None, variance_tgt_value=None\n",
        "):\n",
        "    passed_test = 0\n",
        "    total_test = 1\n",
        "    # `attributions` is a dict where keys are the feature names\n",
        "    # and values are the feature attributions for each feature\n",
        "    baseline_score = explanation.attributions[0].baseline_output_value\n",
        "    print(\"baseline:\", baseline_score)\n",
        "\n",
        "    # Sanity check 1\n",
        "    # The prediction at the input is equal to that at the baseline.\n",
        "    #  Please use a different baseline. Some suggestions are: random input, training\n",
        "    #  set mean.\n",
        "    if abs(prediction - baseline_score) <= 0.05:\n",
        "        print(\"Warning: example score and baseline score are too close.\")\n",
        "        print(\"You might not get attributions.\")\n",
        "    else:\n",
        "        passed_test += 1\n",
        "        print(\"Sanity Check 1: Passed\")\n",
        "\n",
        "    print(passed_test, \" out of \", total_test, \" sanity checks passed.\")\n",
        "\n",
        "\n",
        "i = 0\n",
        "for explanation in response.explanations:\n",
        "    try:\n",
        "        prediction = np.max(response.predictions[i][\"scores\"])\n",
        "    except TypeError:\n",
        "        prediction = np.max(response.predictions[i])\n",
        "    sanity_check_explanations(explanation, prediction)\n",
        "    i += 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "outputs": [],
      "source": [
        "# Undeploy the model from the endpoint\n",
        "endpoint.undeploy_all()\n",
        "\n",
        "# Delete the model\n",
        "model.delete()\n",
        "\n",
        "# Delete the endpoint\n",
        "endpoint.delete()\n",
        "\n",
        "# Delete the dataset\n",
        "dataset.delete()\n",
        "\n",
        "# Delete the automl training job\n",
        "job.delete()\n",
        "\n",
        "# Delete the Cloud Storage bucket\n",
        "delete_bucket = False  # set True for deletion\n",
        "if delete_bucket:\n",
        "    ! gsutil rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "sdk_automl_tabular_classification_online_explain.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}