diff --git a/notebook/gen_ai_jss.ipynb b/notebook/gen_ai_jss.ipynb new file mode 100644 index 0000000..9a3c360 --- /dev/null +++ b/notebook/gen_ai_jss.ipynb @@ -0,0 +1,670 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ur8xi4C7S06n" + }, + "outputs": [], + "source": [ + "# Copyright 2023 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JAPoU8Sm5E6e", + "tags": [] + }, + "source": [ + "# Generative AI Document Summarization\n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Colab Run in Colab\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + " \n", + " \n", + " \"Vertex\n", + " Open in Vertex AI Workbench\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "24743cf4a1e1" + }, + "source": [ + "**_NOTE_**: This notebook has been tested in the following environment:\n", + "\n", + "* Python version = 3.9" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tvgnzT1CKxrO" + }, + "source": [ + "## Overview\n", + "\n", + "This notebook is a companion to the [Generative AI Document Summarization Jump Start Solution](https://cloud.google.com/blog/products/application-modernization/introducing-google-cloud-jump-start-solutions) **TODO: better link target**. With this notebook, you can use the summarization solution to create summaries of academic PDF files. In the notebook, you will programmatically upload a PDF file to a Cloud Storage bucket and then view the summary of that PDF in a BigQuery table. \n", + "\n", + "+ Learn more about [using text chat LLM with Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview).\n", + "+ Learn more about [querying tables in Cloud BigQuery](https://cloud.google.com/bigquery/docs/tables).\n", + "+ Learn more about [creating EventArc triggers for Cloud Functions](https://cloud.google.com/functions/docs/calling/eventarc).\n", + "+ Learn more about [storing data in Cloud Storage](https://cloud.google.com/storage/docs/uploading-objects).\n", + "+ Learn more about [transcribing PDFs with Cloud Vision OCR](https://cloud.google.com/vision/docs/pdf)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "d975e698c9a4" + }, + "source": [ + "### Objective\n", + "\n", + "In this tutorial, you learn how to create a Cloud Function process that transcribes characters from a PDF, stores the complete PDF text in a Storage bucket, summarizes the PDF, and then upserts the document data (summary, complete text, URI) into a BigQuery table.\n", + "\n", + "This tutorial uses the following Google Cloud services and resources:\n", + "\n", + "- Vertex AI Generative AI\n", + "- CLoud BigQuery\n", + "- Cloud Vision OCR\n", + "- Cloud EventArc triggers\n", + "- Cloud Functions\n", + "- Cloud Storage\n", + "\n", + "The steps performed include:\n", + "\n", + "- Trigger an EventArc event by uploading a PDF to a Cloud Storage bucket\n", + "- Query the BigQuery table to see the results of the summarization process" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "08d289fa873f" + }, + "source": [ + "### Dataset\n", + "\n", + "This notebook uses a [Kaggle dataset](https://www.kaggle.com/datasets/Cornell-University/arxiv) that contains a large collection of academic summaries from [arXiv.org](https://arxiv.org/). This dataset is made publicly available through a Cloud Storage bucket." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aed92deeb4a0" + }, + "source": [ + "### Costs \n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "* Vertex AI\n", + "* BigQuery\n", + "* Vision\n", + "* Cloud Functions\n", + "* EventArc\n", + "* Cloud Storage\n", + "\n", + "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n", + "and [BigQuery pricing](https://cloud.google.com/bigquery/pricing),\n", + "and [Cloud Vision pricing](https://cloud.google.com/vision/pricing),\n", + "and [Cloud Functions pricing](https://cloud.google.com/functions/pricing),\n", + "and [Cloud EventArc pricing](https://cloud.google.com/eventarc/pricing),\n", + "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), \n", + "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n", + "to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "i7EUnXsZhAGF" + }, + "source": [ + "## Installation\n", + "\n", + "Install the following packages required to execute this notebook. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "%%writefile requirements.txt\n", + "\n", + "google-cloud-aiplatform\n", + "google-cloud-bigquery\n", + "google-cloud-logging\n", + "google-cloud-storage\n", + "polling2\n", + "tqdm" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "2b4ef9b72d43" + }, + "outputs": [], + "source": [ + "# Install the packages\n", + "import os\n", + "\n", + "if not os.getenv(\"IS_TESTING\"):\n", + " USER = \"--user\"\n", + "else:\n", + " USER = \"\"\n", + "! pip3 install {USER} --upgrade -r requirements.txt" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "58707a750154" + }, + "source": [ + "### Colab only: Uncomment the following cell to restart the kernel." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "f200f10a1da3" + }, + "outputs": [], + "source": [ + "# Automatically restart kernel after installs so that your environment can access the new packages\n", + "# import IPython\n", + "\n", + "# app = IPython.Application.instance()\n", + "# app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BF1j6f9HApxa" + }, + "source": [ + "## Before you begin\n", + "\n", + "### Set up your Google Cloud project\n", + "\n", + "This notebook assumes that you have already deployed this solution using either the [Terraform script]() **TODO: fix target for link** or using the [Solutions console](https://console.cloud.google.com/products/solutions/catalog). During this deployment, several actions required to run this solution were performed on your behalf:\n", + "\n", + "1. The [Cloud Function](https://console.cloud.google.com/functions/list) was deployed.\n", + "\n", + "2. The [EventArc trigger](https://console.cloud.google.com/eventarc/triggers) was applied to the input Cloud Storage bucket.\n", + "\n", + "3. The following APIs were enabled for you: \n", + "\n", + " - [Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)\n", + " - [BigQuery API](https://console.cloud.google.com/flows/enableapi?apiid=bigquery.googleapis.com)\n", + " - [Cloud Vision API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)\n", + "\n", + "\n", + "
Note: It is recommended to run this notebook from Vertex AI Workbench. If you are running this notebook locally instead, you need to install the Cloud SDK.
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WReHDGG5g0XY" + }, + "source": [ + "#### Set your project ID\n", + "\n", + "**If you don't know your project ID**, try the following:\n", + "* Run `gcloud config list`.\n", + "* Run `gcloud projects list`.\n", + "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "oM1iC_MfAts1" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + "\n", + "# Set the project id\n", + "! gcloud config set project {PROJECT_ID}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "region" + }, + "source": [ + "#### Region\n", + "\n", + "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "region" + }, + "outputs": [], + "source": [ + "REGION = \"us-central1\" # @param {type: \"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sBCra4QMA2wR" + }, + "source": [ + "### Authenticate your Google Cloud account\n", + "\n", + "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "74ccc9e52986" + }, + "source": [ + "**1. Vertex AI Workbench**\n", + "* Do nothing as you are already authenticated." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "de775a3773ba" + }, + "source": [ + "**2. Local JupyterLab instance, uncomment and run:**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "254614fa0c46" + }, + "outputs": [], + "source": [ + "# ! gcloud auth login" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ef21552ccea8" + }, + "source": [ + "**3. Colab, uncomment and run:**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "603adbbf0532" + }, + "outputs": [], + "source": [ + "# from google.colab import auth\n", + "# auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f6b2ccc891ed" + }, + "source": [ + "**4. Service account or other**\n", + "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "960505627ddf" + }, + "source": [ + "### Import libraries" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "PyQmSRbKA8r-" + }, + "outputs": [], + "source": [ + "import os\n", + "import polling2\n", + "import re\n", + "import time\n", + "\n", + "from tqdm.notebook import tqdm\n", + "from google.cloud import aiplatform\n", + "from google.cloud import bigquery\n", + "from google.cloud import logging\n", + "from google.cloud import storage" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Download test data\n", + "\n", + "This Jump Start Solution uses data from [arXiv.org](https://arxiv.org/) to demonstrate the summarization capabilities of Vertex AI. arXiv, through [Kaggle.com](https://www.kaggle.com/datasets/Cornell-University/arxiv) has made many scholarly papers available, free of charge, from a Google Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# List all the comparative linguistics papers from Cloud Storage\n", + "! gsutil ls gs://arxiv-dataset/arxiv/cmp-lg/pdf/9404" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "filename = '9404002v1'\n", + "file_uri = f'gs://arxiv-dataset/arxiv/cmp-lg/pdf/9404/{filename}.pdf'\n", + "\n", + "# Create a local folder and download some test PDFs\n", + "if not os.path.exists('pdfs'):\n", + " os.mkdir('pdfs')\n", + "\n", + "! gsutil cp -r $file_uri pdfs/" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Upload test data to Storage bucket\n", + "\n", + "The Terraform scripts for this JSS applies an EventArc trigger to a Cloud Storage bucket. When a PDF is uploaded to the storage bucket, the EventArc trigger fires, starting the summarization process." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "INPUT_BUCKET = f'{PROJECT_ID}_uploads'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Running the next cell uploads a local PDF file (downloaded in the previous section) to the target Cloud Storage bucket. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "file_complete_text = f'{filename}_summary.txt'\n", + "pdf = f'pdfs/{filename}.pdf'\n", + "logger_name = 'summarization-by-llm'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "storage_client = storage.Client()\n", + "bucket = storage_client.bucket(INPUT_BUCKET)\n", + "blob = bucket.blob(pdf)\n", + "blob.upload_from_filename(pdf)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This upload process kicks off the summarization process. You can view the progress of the summarization process in the [Cloud Console](https://console.cloud.google.com/functions/details/us-central1/jss16-1).\n", + "\n", + "**TODO: Ensure that Cloud Console links go to correct console locations.**" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Optional: View summarization process in Cloud Logging\n", + "\n", + "You can view the results of the summarization Cloud Function as it writes updates to Cloud Logging. Each run of the summarization pipeline is associated with a `cloud_event_id`. By filtering for this ID, you can track the summarization process." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "@polling2.poll_decorator(check_success=lambda x: x != '', step=0.5, timeout=90)\n", + "def get_cloud_event_id(pdf_filename, bar):\n", + " logging_client = logging.Client()\n", + " logger = logging_client.logger(logger_name)\n", + " \n", + " pattern = 'cloud_event_id\\((.*)\\):'\n", + " cloud_id = ''\n", + " for entry in logger.list_entries(filter_=pdf_filename, max_results=100):\n", + " entry_text = entry.payload\n", + " res = re.search(pattern, entry_text)\n", + " if res != None:\n", + " cloud_id = res.group(1)\n", + " print(cloud_id)\n", + " bar.update(100)\n", + " \n", + " if cloud_id is not '':\n", + " return cloud_id\n", + " return cloud_id" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with tqdm(total=100) as bar:\n", + " cloud_event_id = get_cloud_event_id(filename, bar)\n", + " bar.close()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that we have the `cloud_event_id`, we can filter on just this cloud event and get updates for just this event." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "cloud_event_id" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "@polling2.poll_decorator(step=10, timeout=70)\n", + "def get_cloud_event_logs(cloud_event_id):\n", + " print(\"polling\")\n", + " logging_client = logging.Client()\n", + " logger = logging_client.logger(logger_name)\n", + " \n", + " entries = []\n", + " for entry in logger.list_entries(filter_=cloud_event_id, max_results=100):\n", + " entry_text = entry.payload\n", + " entries.append(entry_text)\n", + " return entries" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "entries = []\n", + "bar = tqdm(total=6)\n", + "\n", + "for _ in range(6):\n", + " tmp_entries = get_cloud_event_logs(cloud_event_id)\n", + " for e in tmp_entries:\n", + " if e not in entries:\n", + " bar.update(1)\n", + " entries.append(e)\n", + " print(e)\n", + " \n", + " " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Query the BigQuery table to see the summary\n", + "\n", + "Once the summarization flow has completed, the summary of the PDF document should be available for you to read. To get the summary of the PDF document, you can query the BigQuery table that contains the summary.\n", + "\n", + "If you do not get a result the first time you run the query, then the summarization pipeline might still be running. You might need to wait a minute to allow the pipeline to finish and to try the query again." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "bigquery_client = bigquery.Client()\n", + "\n", + "table_name = f\"{PROJECT_ID}.summary_dataset.summary_table\"\n", + "\n", + "# Compose the SQL query to select the summary for the PDF document\n", + "sql_query = f\"SELECT summary FROM `{table_name}` WHERE filename LIKE '%{file_complete_text}%'\"\n", + "\n", + "job = bigquery_client.query(sql_query)\n", + "rows = job.result()\n", + "row_list = list(rows)\n", + "\n", + "if len(row_list) is not 0:\n", + " summary = row_list[0]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "summary['summary']" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "notebook_template.ipynb", + "toc_visible": true + }, + "environment": { + "kernel": "python3", + "name": "common-cu110.m103", + "type": "gcloud", + "uri": "gcr.io/deeplearning-platform-release/base-cu110:m103" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.12" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}