vision/getting-started/visual_question_answering.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uxCkB_DXTHzf"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hny4I-ODTIS6"
      },
      "source": [
        "# Visual Question Answering (VQA) with Imagen on Vertex AI\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/visual_question_answering.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/visual_question_answering.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/visual_question_answering.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/visual_question_answering.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/visual_question_answering.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/visual_question_answering.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/53/X_logo_2023_original.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/visual_question_answering.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/visual_question_answering.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "048a84091064"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Thu Ya Kyaw](https://github.com/iamthuya) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-nLS57E2TO5y"
      },
      "source": [
        "## Overview\n",
        "\n",
        "[Imagen on Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/image/overview) (image Generative AI) offers a variety of features:\n",
        "- Image generation\n",
        "- Image editing\n",
        "- Visual captioning\n",
        "- Visual question answering\n",
        "\n",
        "This notebook focuses on **visual question answering** only.\n",
        "\n",
        "[Visual question answering (VQA) with Imagen](https://cloud.google.com/vertex-ai/docs/generative-ai/image/visual-question-answering) can understand the content of an image and answer questions about it. The model takes in an image and a question as input, and then using the image as context to produce one or more answers to the question.\n",
        "\n",
        "The visual question answering (VQA) can be used for a variety of use cases, including:\n",
        "- assisting the visually impaired to gain more information about the images\n",
        "- answering customer questions about products or services in the image\n",
        "- creating interactive learning environment and providing interactive learning experiences"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iXsvgIuwTPZw"
      },
      "source": [
        "### Objectives\n",
        "\n",
        "In this notebook, you will learn how to use the Vertex AI Python SDK to:\n",
        "\n",
        "- Answering questions about images using the Imagen's visual question answering features\n",
        "\n",
        "- Experiment with different parameters, such as:\n",
        "    - number of answers to be provided by the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "skXAu__iqks_"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "- Vertex AI (Imagen)\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mvKl-BtQTRiQ"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PwFMpIMrTV_4"
      },
      "source": [
        "### Install Vertex AI SDK, other packages and their dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WYUu8VMdJs3V"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --user google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart current runtime\n",
        "\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which will restart the current kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "# Restart kernel after installs so that your environment can access the new packages\n",
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "opUxT_k5TdgP"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, you will need to authenticate your environment. To do this, run the new cell below. This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vbNgv4q1T2Mi"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    # Authenticate user to Google Cloud\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ybBXSukZkgjg"
      },
      "source": [
        "### Define Google Cloud project information and initialize Vertex AI\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MkVX07nOC90T"
      },
      "outputs": [],
      "source": [
        "# Define project information\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
        "\n",
        "# Initialize Vertex AI\n",
        "import vertexai\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lhfneknwEDHT"
      },
      "source": [
        "### Load the image question answering model\n",
        "\n",
        "The model names from Vertex AI Imagen have two components: model name and version number. The naming convention follows this format: `<model-name>@<version-number>`.\n",
        "\n",
        "For example, `imagetext@001` represents the version **001** of the **imagetext** model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VuEbYyfM4RR7"
      },
      "outputs": [],
      "source": [
        "from vertexai.preview.vision_models import ImageQnAModel\n",
        "\n",
        "image_qna_model = ImageQnAModel.from_pretrained(\"imagetext@001\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1zoPQnQoED10"
      },
      "source": [
        "### Load the image file\n",
        "\n",
        "To use the image question answering model, you first need to create an `Image` class using the image file. The model only accepts `Image` class objects, so this is a necessary step before you can ask questions.\n",
        "\n",
        "Additionally, [Visual Question Answering with Imagen](https://cloud.google.com/vertex-ai/docs/generative-ai/image/visual-question-answering) only accepts specific image file formats (e.g. PNG, JPEG), and may have file size limitations (e.g. 10 MB). You can find out specific details from the [official documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/image/visual-question-answering#img-vqa-rest)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xnH-MARxgGrX"
      },
      "outputs": [],
      "source": [
        "from vertexai.preview.vision_models import Image\n",
        "\n",
        "# Use either a Google Cloud Storage URI or a local file path.\n",
        "file_path = (\n",
        "    \"gs://github-repo/img/vision/google-cloud-next.jpeg\"  # @param {type:\"string\"}\n",
        ")\n",
        "\n",
        "# Load the image file as Image object\n",
        "cloud_next_image = Image.load_from_file(file_path)\n",
        "cloud_next_image.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eYjS3nL5LTbY"
      },
      "source": [
        "### Ask questions about the image\n",
        "\n",
        "Now ask questions about the image using the model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i3FliDlsm6ah"
      },
      "outputs": [],
      "source": [
        "# Ask a question about the image\n",
        "image_qna_model.ask_question(\n",
        "    image=cloud_next_image, question=\"What is happening in this image?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mVFKttepEPB9"
      },
      "outputs": [],
      "source": [
        "# Ask a follow up question about the image\n",
        "image_qna_model.ask_question(\n",
        "    image=cloud_next_image, question=\"What are the people in the image doing?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PwBBeYNORmTf"
      },
      "source": [
        "You can get up to three answers from a single image by changing the `number_of_results` parameter from 1 to 3."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X1GJwlZUt4-E"
      },
      "outputs": [],
      "source": [
        "# Get 3 answers from the image\n",
        "image_qna_model.ask_question(\n",
        "    image=cloud_next_image,\n",
        "    question=\"What are the people in the image doing?\",\n",
        "    number_of_results=3,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r0FVAjg-okiW"
      },
      "source": [
        "## Try it yourself\n",
        "\n",
        "You can also try using the visual question answering model with images of your choice. If you need to download an image file, you can use the provided auxiliary function `download_image`.\n",
        "\n",
        "Feel free to experiment with different images and model parameters to see how the results change."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xAytTFqoINyl"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "import requests\n",
        "\n",
        "\n",
        "def download_image(url: str) -> str:\n",
        "    \"\"\"Downloads an image from the specified URL.\"\"\"\n",
        "\n",
        "    # Send a get request to the url\n",
        "    response = requests.get(url)\n",
        "\n",
        "    if response.status_code != 200:\n",
        "        raise Exception(f\"Failed to download image from {url}\")\n",
        "\n",
        "    # Define image related variables\n",
        "    image_path = os.path.basename(url)\n",
        "    image_bytes = response.content\n",
        "    image_type = response.headers[\"Content-Type\"]\n",
        "\n",
        "    # Check for image type, currently only PNG or JPEG format are supported\n",
        "    if image_type not in {\"image/png\", \"image/jpeg\"}:\n",
        "        raise ValueError(\"Image can only be in PNG or JPEG format\")\n",
        "\n",
        "    # Write image data to a file\n",
        "    with open(image_path, \"wb\") as f:\n",
        "        f.write(image_bytes)\n",
        "    return image_path"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ll8KFbCujQ9y"
      },
      "outputs": [],
      "source": [
        "# Download an image\n",
        "url = \"https://storage.googleapis.com/gweb-cloudblog-publish/images/GettyImages-871168786.max-2600x2600.jpg\"\n",
        "image_path = download_image(url)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d61b54eb8670"
      },
      "outputs": [],
      "source": [
        "# Load the newly downloaded image\n",
        "user_image = Image.load_from_file(image_path)\n",
        "user_image.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9g6DsfKWo5S-"
      },
      "outputs": [],
      "source": [
        "# Ask a question about the image\n",
        "image_qna_model.ask_question(\n",
        "    image=user_image,\n",
        "    question=\"What is happening in this photo?\",\n",
        "    number_of_results=3,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YC90LG50Fcru"
      },
      "outputs": [],
      "source": [
        "# Ask a question about the image\n",
        "image_qna_model.ask_question(\n",
        "    image=user_image,\n",
        "    question=\"What advertising channels would this image be suitable for?\",\n",
        "    number_of_results=3,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ydXxZJvLKvtI"
      },
      "outputs": [],
      "source": [
        "# Ask a question about the image\n",
        "image_qna_model.ask_question(\n",
        "    image=user_image,\n",
        "    question=\"What type of insects could live in this area?\",\n",
        "    number_of_results=3,\n",
        ")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "visual_question_answering.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}