diff --git a/gemini/sample-apps/conversational-genai-app-template/app/chain.py b/gemini/sample-apps/conversational-genai-app-template/app/chain.py index 81afed1acaf..586fb11a340 100644 --- a/gemini/sample-apps/conversational-genai-app-template/app/chain.py +++ b/gemini/sample-apps/conversational-genai-app-template/app/chain.py @@ -26,9 +26,9 @@ [ ( "system", - "You are a knowledgeable culinary assistant specializing in providing" \ - "detailed cooking recipes. Your responses should be informative, engaging, " \ - "and tailored to the user's specific requests. Include ingredients, " \ + "You are a knowledgeable culinary assistant specializing in providing" + "detailed cooking recipes. Your responses should be informative, engaging, " + "and tailored to the user's specific requests. Include ingredients, " "step-by-step instructions, cooking times, and any helpful tips or " "variations. If asked about dietary restrictions or substitutions, offer " "appropriate alternatives.", diff --git a/gemini/sample-apps/conversational-genai-app-template/app/server.py b/gemini/sample-apps/conversational-genai-app-template/app/server.py index f1f59fd6c08..61821624566 100644 --- a/gemini/sample-apps/conversational-genai-app-template/app/server.py +++ b/gemini/sample-apps/conversational-genai-app-template/app/server.py @@ -29,6 +29,7 @@ ## Import the chain to be used from app.chain import chain + # Or choose one of the following pattern chains to test by uncommenting it: # Custom RAG QA diff --git a/gemini/sample-apps/conversational-genai-app-template/notebooks/getting_started.ipynb b/gemini/sample-apps/conversational-genai-app-template/notebooks/getting_started.ipynb index 2c04acce766..475022d7978 100644 --- a/gemini/sample-apps/conversational-genai-app-template/notebooks/getting_started.ipynb +++ b/gemini/sample-apps/conversational-genai-app-template/notebooks/getting_started.ipynb @@ -1,1105 +1,1105 @@ { - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "OsXAs2gcIpbC" - }, - "outputs": [], - "source": [ - "# Copyright 2024 Google LLC\n", - "#\n", - "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", - "# you may not use this file except in compliance with the License.\n", - "# You may obtain a copy of the License at\n", - "#\n", - "# https://www.apache.org/licenses/LICENSE-2.0\n", - "#\n", - "# Unless required by applicable law or agreed to in writing, software\n", - "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", - "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", - "# See the License for the specific language governing permissions and\n", - "# limitations under the License." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7ZX50cNFOFBt" - }, - "source": [ - "# Getting Started - Template" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "84eed97da4c4" - }, - "source": [ - "\n", - " \n", - " \n", - " \n", - " \n", - "
\n", - " \n", - " \"Google
Open in Colab\n", - " \n", - " 		\n", - " \n", - " \"Google
Open in Colab Enterprise\n", - " \n", - " 		\n", - " \n", - " \"Vertex
Open in Vertex AI Workbench\n", - " \n", - " 		\n", - " \n", - " \"GitHub
View on GitHub\n", - " \n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "usd0d_LiOFBt" - }, - "source": [ - "| | |\n", - "|-|-|\n", - "|Author(s) | [Elia Secchi](https://github.com/eliasecchig) |" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "MjDmmmDaOFBt" - }, - "source": [ - "## Overview\n", - "\n", - "This tutorial walks you through the process of developing and assessing a chain - a sequence of steps that power an AI application. \n", - "These operations may include interactions with language models, utilization of tools, or data preprocessing steps, aiming to solve a given use case e.g a chatbot that provides grounded information.\n", - "\n", - "You'll learn how to:\n", - "\n", - "1. Build chains using three different approaches:\n", - " - [LangChain Expression Language (LCEL)](https://python.langchain.com/docs/expression_language/)\n", - " - [LangGraph](https://python.langchain.com/docs/langgraph/)\n", - " - A custom Python implementation. This is to enable implementation with other SDKs ( e.g [Vertex AI SDK](https://cloud.google.com/vertex-ai/docs/python-sdk/use-vertex-ai-python-sdk ), [LlamaIndex](https://www.llamaindex.ai/)) and to allow granular control on the sequence of steps in the chain\n", - " \n", - "2. Evaluate the performance of your chains using [Vertex AI Evaluation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview)\n", - "\n", - "Finally, the tutorial discusses next steps for deploying your chain in a production application\n", - "\n", - "By the end of this tutorial, you'll have a solid foundation for developing and refining your own Generative AI chains." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "w-OcPSC8_FUX" - }, - "source": [ - "## Get Started" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "c0a13ca7427f" - }, - "source": [ - "### Install required packages using Poetry (Recommended)\n", - "\n", - "This template uses [Poetry](https://python-poetry.org/) as tool to manage project dependencies. \n", - "Poetry makes it easy to install and keep track of the packages your project needs.\n", - "\n", - "To run this notebook with Poetry, follow these steps:\n", - "1. Make sure Poetry is installed. See the [relative guide for installation](https://python-poetry.org/docs/#installation).\n", - "\n", - "2. Make sure that dependencies are installed. From your command line:\n", - " ```bash\n", - " poetry install --with streamlit,jupyter\n", - " ```\n", - "\n", - "3. Run Jupyter:\n", - " ```bash\n", - " poetry run jupyter\n", - " ```\n", - " \n", - "4. Open this notebook in the Jupyter interface." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "-7Jso8-FO4N8" - }, - "source": [ - "### (Alternative) Install Vertex AI SDK and other required packages " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "tUat7NRq5JDC" - }, - "outputs": [], - "source": [ - "%pip install --quiet --upgrade nest_asyncio\n", - "%pip install --upgrade --user --quiet langchain-core langchain-google-vertexai langchain-google-community langchain langgraph\n", - "%pip install --upgrade --user --quiet \"google-cloud-aiplatform[rapid_evaluation]\"" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "R5Xep4W9lq-Z" - }, - "source": [ - "### Restart runtime\n", - "\n", - "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n", - "\n", - "The restart might take a minute or longer. After it's restarted, continue to the next step." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "XRvKdaPDTznN" - }, - "outputs": [], - "source": [ - "import IPython\n", - "\n", - "app = IPython.Application.instance()\n", - "app.kernel.do_shutdown(True)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "SbmM4z7FOBpM" - }, - "source": [ - "
\n", - "⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️\n", - "
\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "dmWOrTJ3gx13" - }, - "source": [ - "### Authenticate your notebook environment (Colab only)\n", - "\n", - "If you're running this notebook on Google Colab, run the cell below to authenticate your environment." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "NyKGtVQjgx13" - }, - "outputs": [], - "source": [ - "# import sys\n", - "\n", - "# if \"google.colab\" in sys.modules:\n", - "# from google.colab import auth\n", - "\n", - "# auth.authenticate_user()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "DF4l8DTdWgPY" - }, - "source": [ - "### Set Google Cloud project information and initialize Vertex AI SDK\n", - "\n", - "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n", - "\n", - "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Nqwi-5ufWp_B" - }, - "outputs": [], - "source": [ - "# Use the environment variable if the user doesn't provide Project ID.\n", - "import os\n", - "\n", - "import vertexai\n", - "\n", - "PROJECT_ID = \"production-ai-template\" # @param {type:\"string\", isTemplate: true}\n", - "if PROJECT_ID == \"[your-project-id]\":\n", - " PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n", - "\n", - "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n", - "\n", - "vertexai.init(project=PROJECT_ID, location=LOCATION)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "dvhI92xhQTzk" - }, - "source": [ - "### Import libraries" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "869d543465ac" - }, - "outputs": [], - "source": [ - "# Add the parent directory to the Python path. This allows importing modules from the parent directory\n", - "import sys\n", - "\n", - "sys.path.append(\"../\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "146b41115577" - }, - "outputs": [], - "source": [ - "import json\n", - "import pandas as pd\n", - "import yaml\n", - "from json import JSONDecodeError\n", - "from typing import Any, Dict, Iterator, Literal\n", - "\n", - "from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n", - "from langchain_core.runnables import RunnableConfig\n", - "from langchain_core.tools import tool\n", - "from langchain_google_community.vertex_rank import VertexAIRank\n", - "from langchain_google_vertexai import ChatVertexAI, VertexAI, VertexAIEmbeddings\n", - "from langgraph.graph import END, MessagesState, StateGraph\n", - "from langgraph.prebuilt import ToolNode\n", - "from google.cloud import aiplatform\n", - "from vertexai.evaluation import CustomMetric, EvalTask\n", - "\n", - "from app.eval.utils import batch_generate_messages, generate_multiturn_history\n", - "from app.patterns.custom_rag_qa.templates import query_rewrite_template, rag_template\n", - "from app.patterns.custom_rag_qa.vector_store import get_vector_store\n", - "from app.utils.output_types import OnChatModelStreamEvent, OnToolEndEvent, custom_chain" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "675dff1826d4" - }, - "source": [ - "## Chain Interface\n", - "\n", - "This section outlines a possible interface for the chain, which, if implemented, ensures compatibility with the FastAPI server application included in the template. However, it's important to note that you have the flexibility to explore and implement alternative interfaces that suit their specific needs and requirements.\n", - "\n", - "\n", - "### Input Interface\n", - "\n", - "The chain must provide an `astream_events` method that accepts a dictionary with a \"messages\" key.\n", - "The \"messages\" value should be a list of alternating LangChain [HumanMessage](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.human.HumanMessage.html) and [AIMessage](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessage.html) objects.\n", - "\n", - "For example a possible input might be:\n", - "\n", - "```python\n", - "{\n", - " \"messages\": [\n", - " HumanMessage(\"first\"),\n", - " AIMessage(\"a response\"),\n", - " HumanMessage(\"a follow up\")\n", - " ]\n", - "}\n", - "```\n", - "\n", - "Alternatively you can use the shortened form:\n", - "\n", - "```python\n", - "{\n", - " \"messages\": [\n", - " (\"user\", \"first\"),\n", - " (\"ai\", \"a response\"),\n", - " (\"user\", \"a follow up\")\n", - " ]\n", - "}\n", - "```\n", - "\n", - "### Output Interface\n", - "\n", - "All chains use the [LangChain Stream Events (v2) API](https://python.langchain.com/docs/how_to/streaming/#using-stream-events). This API supports various use cases (simple chains, RAG, Agents). This API emits asynchronous events that can be used to stream the chain's output.\n", - "\n", - "LangChain chains (LCEL, LangGraph) automatically implement the `astream_events` API. \n", - "\n", - "We provide examples of emitting `astream_events`-compatible events with custom Python code, allowing implementation with other SDKs (e.g., Vertex AI, LLamaIndex).\n", - "\n", - "### Customizing I/O Interfaces\n", - "\n", - "To modify the Input/Output interface, update `app/server.py` and related unit and integration tests.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "865ba0268d3b" - }, - "source": [ - "## Events supported\n", - "\n", - "The following list defines the events that are captured and supported by the Streamlit frontend." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "4ece481555a2" - }, - "outputs": [], - "source": [ - "SUPPORTED_EVENTS = [\n", - " \"on_tool_start\",\n", - " \"on_tool_end\",\n", - " \"on_retriever_start\",\n", - " \"on_retriever_end\",\n", - " \"on_chat_model_stream\",\n", - "]" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "874120e6a4d2" - }, - "source": [ - "### Define the LLM\n", - "We set up the Large Language Model (LLM) for our conversational bot.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "0d5c52272898" - }, - "outputs": [], - "source": [ - "llm = ChatVertexAI(model_name=\"gemini-1.5-flash-002\", temperature=0)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "abc296e9da88" - }, - "source": [ - "### Leveraging LangChain LCEL for Efficient Chain Composition\n", - "\n", - "LangChain Expression Language (LCEL) provides a declarative approach to composing chains seamlessly. Key benefits include:\n", - "\n", - "1. Rapid prototyping to production deployment without code changes\n", - "2. Scalability from simple \"prompt + LLM\" chains to complex, multi-step workflows\n", - "3. Enhanced readability and maintainability of chain logic\n", - "\n", - "For comprehensive guidance on LCEL implementation, refer to the [official documentation](https://python.langchain.com/docs/expression_language/get_started).\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "25d185f29f42" - }, - "outputs": [], - "source": [ - "template = ChatPromptTemplate.from_messages(\n", - " [\n", - " (\n", - " \"system\",\n", - " \"You are a knowledgeable culinary assistant specializing in providing\" \\\n", - " \"detailed cooking recipes. Your responses should be informative, engaging, \" \\\n", - " \"and tailored to the user's specific requests. Include ingredients, \" \\\n", - " \"step-by-step instructions, cooking times, and any helpful tips or \"\n", - " \"variations. If asked about dietary restrictions or substitutions, offer \"\n", - " \"appropriate alternatives.\",\n", - " ),\n", - " MessagesPlaceholder(variable_name=\"messages\"),\n", - " ]\n", - ")\n", - "\n", - "chain = template | llm" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "216f187e66d7" - }, - "source": [ - "Let's test the chain with a dummy question:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "267a75f53b45" - }, - "outputs": [], - "source": [ - "input_message = {\"messages\": [(\"user\", \"Can you provide me a Lasagne recipe?\")]}\n", - "\n", - "async for event in chain.astream_events(input=input_message, version=\"v2\"):\n", - " if event[\"event\"] in SUPPORTED_EVENTS:\n", - " print(event[\"data\"])" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "b262558c2375" - }, - "source": [ - "This methodology is used for the chain defined in the [`app/chain.py`](../app/chain.py) file.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1fa3c684b527" - }, - "source": [ - "We can also leverage the `invoke` method for synchronous invocation." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "5fc2bfec1d5b" - }, - "outputs": [], - "source": [ - "response = chain.invoke(input=input_message)\n", - "print(response.content)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "16a9fd5fdf2f" - }, - "source": [ - "### Use LangGraph\n", - "\n", - "LangGraph is a framework for building stateful, multi-actor applications with Large Language Models (LLMs). \n", - "It extends the LangChain library, allowing you to coordinate multiple chains (or actors) across multiple steps of computation in a cyclic manner." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "28b4ad81f8b7" - }, - "outputs": [], - "source": [ - "# 1. Define tools\n", - "@tool\n", - "def search(query: str):\n", - " \"\"\"Simulates a web search. Use it get information on weather. E.g what is the weather like in a region\"\"\"\n", - " if \"sf\" in query.lower() or \"san francisco\" in query.lower():\n", - " return \"It's 60 degrees and foggy.\"\n", - " return \"It's 90 degrees and sunny.\"\n", - "\n", - "\n", - "tools = [search]\n", - "\n", - "# 2. Set up the language model\n", - "llm = llm.bind_tools(tools)\n", - "\n", - "\n", - "# 3. Define workflow components\n", - "def should_continue(state: MessagesState) -> Literal[\"tools\", END]:\n", - " \"\"\"Determines whether to use tools or end the conversation.\"\"\"\n", - " last_message = state[\"messages\"][-1]\n", - " return \"tools\" if last_message.tool_calls else END\n", - "\n", - "\n", - "async def call_model(state: MessagesState, config: RunnableConfig):\n", - " \"\"\"Calls the language model and returns the response.\"\"\"\n", - " response = llm.invoke(state[\"messages\"], config)\n", - " return {\"messages\": response}\n", - "\n", - "\n", - "# 4. Create the workflow graph\n", - "workflow = StateGraph(MessagesState)\n", - "workflow.add_node(\"agent\", call_model)\n", - "workflow.add_node(\"tools\", ToolNode(tools))\n", - "workflow.set_entry_point(\"agent\")\n", - "\n", - "# 5. Define graph edges\n", - "workflow.add_conditional_edges(\"agent\", should_continue)\n", - "workflow.add_edge(\"tools\", \"agent\")\n", - "\n", - "# 6. Compile the workflow\n", - "chain = workflow.compile()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "bd27c49717e8" - }, - "source": [ - "Let's test the new chain with a dummy question:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "a33402d68f7b" - }, - "outputs": [], - "source": [ - "input_message = {\"messages\": [(\"user\", \"What is the weather like in NY?\")]}\n", - "\n", - "async for event in chain.astream_events(input=input_message, version=\"v2\"):\n", - " if event[\"event\"] in SUPPORTED_EVENTS:\n", - " print(event[\"data\"])" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "2cf0ad148bf8" - }, - "source": [ - "This methodology is used for the chain defined in the [`app/patterns/langgraph_dummy_agent/chain.py`](../app/patterns/langgraph_dummy_agent/chain.py) file.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0ae7dad0966b" - }, - "source": [ - "### Use custom python code\n", - "\n", - "You can also use pure python code to orchestrate the different steps of your chain and emit `astream_events` [API compatible events](https://python.langchain.com/docs/how_to/streaming/#using-stream-events). \n", - "\n", - "This offers full flexibility in how the different steps of a chain are orchestrated and allows you to include other SDK frameworks such as [Vertex AI SDK](https://cloud.google.com/vertex-ai/docs/python-sdk/use-vertex-ai-python-sdk ), [LlamaIndex](https://www.llamaindex.ai/).\n", - "\n", - "We demonstrate this third methodology by implementing a RAG chain. The function `get_vector_store` provides a brute force Vector store (scikit-learn) initialized with data obtained from the [practictioners guide for MLOps](https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf).\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "ea13644948d2" - }, - "outputs": [], - "source": [ - "llm = ChatVertexAI(model_name=\"gemini-1.5-flash-002\", temperature=0)\n", - "embedding = VertexAIEmbeddings(model_name=\"text-embedding-004\")\n", - "\n", - "\n", - "vector_store = get_vector_store(embedding=embedding)\n", - "retriever = vector_store.as_retriever(search_kwargs={\"k\": 20})\n", - "compressor = VertexAIRank(\n", - " project_id=PROJECT_ID,\n", - " location_id=\"global\",\n", - " ranking_config=\"default_ranking_config\",\n", - " title_field=\"id\",\n", - " top_n=5,\n", - ")\n", - "\n", - "query_gen = query_rewrite_template | llm\n", - "response_chain = rag_template | llm\n", - "\n", - "\n", - "@custom_chain\n", - "def chain(\n", - " input: Dict[str, Any], **kwargs\n", - ") -> Iterator[OnToolEndEvent | OnChatModelStreamEvent]:\n", - " \"\"\"\n", - " Implements a RAG QA chain. Decorated with `custom_chain` to offer LangChain compatible astream_events\n", - " and invoke interface and OpenTelemetry tracing.\n", - " \"\"\"\n", - " # Generate optimized query\n", - " query = query_gen.invoke(input=input).content\n", - "\n", - " # Retrieve and rank documents\n", - " retrieved_docs = retriever.get_relevant_documents(query)\n", - " ranked_docs = compressor.compress_documents(documents=retrieved_docs, query=query)\n", - "\n", - " # Yield tool results metadata\n", - " yield OnToolEndEvent(data={\"input\": {\"query\": query}, \"output\": ranked_docs})\n", - " # Stream LLM response\n", - " for chunk in response_chain.stream(\n", - " input={\"messages\": input[\"messages\"], \"relevant_documents\": ranked_docs}\n", - " ):\n", - " yield OnChatModelStreamEvent(data={\"chunk\": chunk})" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "2d59f2641aaf" - }, - "source": [ - "The `@custom_chain` decorator defined in `app/utils/output_types.py`:\n", - "- Enables compatibility with the `astream_events` Langchain API interface by offering a `chain.astream_events` method.\n", - "- Provides an `invoke` method for synchronous invocation. This method can be utilized for evaluation purposes.\n", - "- Adds OpenTelemetry tracing functionality.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1b0d16359361" - }, - "source": [ - "This methodology is used for the chain defined in `app/patterns/custom_rag_qa/chain.py` file.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "82f22f13fde6" - }, - "source": [ - "Let's test the custom chain we just created. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "7a1bcac73ff5" - }, - "outputs": [], - "source": [ - "input_message = {\"messages\": [(\"user\", \"What is MLOps?\")]}\n", - "\n", - "async for event in chain.astream_events(input=input_message, version=\"v2\"):\n", - " if event[\"event\"] in SUPPORTED_EVENTS:\n", - " print(event[\"data\"])" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8730b4b6bbff" - }, - "source": [ - "## Evaluation\n", - "\n", - "Evaluation is the activity of assessing the quality of the model's outputs, to gauge its understanding and success in fulfilling the prompt's instructions.\n", - "\n", - "In the context of Generative AI, evaluation extends beyond the evaluation of the model's outputs to include the evaluation of the chain's outputs and in some cases the evaluation of the intermediate steps (for example, the evaluation of the retriever's outputs).\n", - "\n", - "### Vertex AI Evaluation\n", - "To evaluate the chain's outputs, we'll utilize [Vertex AI Evaluation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview) to assess our AI application's performance. \n", - "Vertex AI Evaluation streamlines the evaluation process for generative AI by offering three key features:\n", - "\n", - "- [Pre-built Metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval): It provides a library of ready-to-use metrics for common evaluation tasks, saving you time and effort in defining your own. These metrics cover a range of areas, simplifying the assessment of different aspects of your model's performance.\n", - " \n", - "- [Custom Metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval): Beyond pre-built options, Vertex AI Evaluation allows you to define and implement custom metrics tailored to your specific needs and application requirements. \n", - " \n", - "- Strong Integration with [Vertex AI Experiments](https://cloud.google.com/vertex-ai/docs/experiments/intro-vertex-ai-experiments): Vertex AI Evaluation seamlessly integrates with Vertex AI Experiments, creating a unified workflow for tracking experiments and managing evaluation results.\n", - "\n", - "For a comprehensive list of samples on Vertex AI Evaluation, visit the [official documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-examples)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "e214b9a02547" - }, - "source": [ - "Let's start by defining again a simple chain:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "086298b785fb" - }, - "outputs": [], - "source": [ - "template = ChatPromptTemplate.from_messages(\n", - " [\n", - " (\n", - " \"system\",\n", - " \"You are a knowledgeable culinary assistant specializing in providing\" \\\n", - " \"detailed cooking recipes. Your responses should be informative, engaging, \" \\\n", - " \"and tailored to the user's specific requests. Include ingredients, \" \\\n", - " \"step-by-step instructions, cooking times, and any helpful tips or \"\n", - " \"variations. If asked about dietary restrictions or substitutions, offer \"\n", - " \"appropriate alternatives.\",\n", - " ),\n", - " MessagesPlaceholder(variable_name=\"messages\"),\n", - " ]\n", - ")\n", - "\n", - "chain = template | llm" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "284ec00b23c7" - }, - "source": [ - "We then import the ground truth data we will use for evaluation. Data is stored in [`app/eval/data/chats.yaml`](../app/eval/data/chats.yaml)\n", - "Note: You might need to adjust the path depending on where your Jupyter kernel was initialized.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "f7c05ab676d8" - }, - "outputs": [], - "source": [ - "y = yaml.safe_load(open(\"../app/eval/data/chats.yaml\"))\n", - "df = pd.DataFrame(y)\n", - "df" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "432ce6674c19" - }, - "source": [ - "We leverage the helper functions [`generate_multiturn_history`](../app/eval/utils.py) and [`batch_generate_messages`](../app/eval/utils.py) to prepare the data for evaluation and to generate the responses from the chain.\n", - "\n", - "You can see below the documentation for the two functions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "3a6899faf8b3" - }, - "outputs": [], - "source": [ - "help(generate_multiturn_history)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "6afe1b67475f" - }, - "outputs": [], - "source": [ - "help(batch_generate_messages)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "b5e1411cf0f4" - }, - "outputs": [], - "source": [ - "df = generate_multiturn_history(df)\n", - "df" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "9ff1aaed7eb0" - }, - "outputs": [], - "source": [ - "scored_data = batch_generate_messages(df, chain)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "933c797dc79e" - }, - "source": [ - "We extract the user message and the reference (ground truth) message from dataframe so that we can use them for evaluation." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "5e5ba1acd013" - }, - "outputs": [], - "source": [ - "scored_data[\"user\"] = scored_data[\"human_message\"].apply(lambda x: x[\"content\"])\n", - "scored_data[\"reference\"] = scored_data[\"ai_message\"].apply(lambda x: x[\"content\"])\n", - "scored_data" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "dc41cc55703e" - }, - "source": [ - "#### Define a CustomMetric using Gemini model\n", - "\n", - "Define a customized Gemini model-based metric function, with explanations for the score. The registered custom metrics are computed on the client side, without using online evaluation service APIs." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "51c0ffb825e1" - }, - "outputs": [], - "source": [ - "evaluator_llm = ChatVertexAI(\n", - " model_name=\"gemini-1.5-flash-001\",\n", - " temperature=0,\n", - " response_mime_type=\"application/json\",\n", - ")\n", - "\n", - "\n", - "def custom_faithfulness(instance):\n", - " prompt = f\"\"\"You are examining written text content. Here is the text:\n", - "************\n", - "Written content: {instance[\"response\"]}\n", - "************\n", - "Original source data: {instance[\"reference\"]}\n", - "\n", - "Examine the text and determine whether the text is faithful or not.\n", - "Faithfulness refers to how accurately a generated summary reflects the essential information and key concepts present in the original source document.\n", - "A faithful summary stays true to the facts and meaning of the source text, without introducing distortions, hallucinations, or information that wasn't originally there.\n", - "\n", - "Your response must be an explanation of your thinking along with single integer number on a scale of 0-5, 0\n", - "the least faithful and 5 being the most faithful.\n", - "\n", - "Produce results in JSON\n", - "\n", - "Expected format:\n", - "\n", - "```json\n", - "{{\n", - " \"explanation\": \"< your explanation>\",\n", - " \"custom_faithfulness\": \n", - "}}\n", - "```\n", - "\"\"\"\n", - "\n", - " result = evaluator_llm.invoke([(\"human\", prompt)])\n", - " result = json.loads(result.content)\n", - " return result\n", - "\n", - "\n", - "# Register Custom Metric\n", - "custom_faithfulness_metric = CustomMetric(\n", - " name=\"custom_faithfulness\",\n", - " metric_function=custom_faithfulness,\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "234f1b868cb9" - }, - "outputs": [], - "source": [ - "experiment_name = \"template-langchain-eval\" # @param {type:\"string\"}" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "f5b15279f9d2" - }, - "source": [ - "We are now ready to run the evaluation. We will use different metrics, combining the custom metric we defined above with some pre-built metrics.\n", - "\n", - "Results of the evaluation will be automatically tagged into the experiment_name we define.\n", - "\n", - "You can click `View Experiment`, to see the experiment in Google Cloud Console." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "4d3faf6014f5" - }, - "outputs": [], - "source": [ - "metrics = [\"fluency\", \"safety\", custom_faithfulness_metric]\n", - "\n", - "eval_task = EvalTask(\n", - " dataset=scored_data,\n", - " metrics=metrics,\n", - " experiment=experiment_name,\n", - " metric_column_mapping={\"prompt\": \"user\"},\n", - ")\n", - "eval_result = eval_task.evaluate()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "c51deffd590c" - }, - "source": [ - "Once an eval result is produced, we are able to display summary metrics:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "KheOvIvtiRlz" - }, - "outputs": [], - "source": [ - "eval_result.summary_metrics" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "JcALGGlwu0p_" - }, - "source": [ - "We are also able to display a pandas dataframe containing a detailed summary of how our eval dataset performed and relative granular metrics." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "9zJ686YYiWJC" - }, - "outputs": [], - "source": [ - "eval_result.metrics_table" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "378025ea68d0" - }, - "source": [ - "## Next Steps\n", - "\n", - "Congratulations on completing the getting started tutorial! You've learned different methodologies to build a chain and how to evaluate it. \n", - "Let's explore the next steps in your journey:\n", - "\n", - "### 1. Prepare for Production\n", - "\n", - "Once you're satisfied with your chain's evaluation results:\n", - "\n", - "1. Write your chain into the [`app/chain.py` file](../app/chain.py).\n", - "2. Remove the `patterns` folder and its associated tests (these are for demonstration only).\n", - "\n", - "### 2. Local Testing\n", - "\n", - "Test your chain using the playground:\n", - "\n", - "```bash\n", - "make playground\n", - "```\n", - "\n", - "This launches af feature-rich playground, including chat curation, user feedback collection, multimodal input, and more!\n", - "\n", - "\n", - "### 3. Production Deployment\n", - "\n", - "Once you are satisfied with the results, you can setup your CI/CD pipelines to deploy your chain to production.\n", - "\n", - "Please refer to the [deployment guide](../deployment/README.md) for more information on how to do that." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7bf6a5b0def0" - }, - "source": [ - "## Cleaning up\n", - "\n", - "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n", - "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", - "\n", - "Otherwise, you can delete the individual resources you created in this tutorial." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "0236a4c1471d" - }, - "outputs": [], - "source": [ - "import os\n", - "\n", - "# Delete Experiments\n", - "delete_experiments = True\n", - "if delete_experiments or os.getenv(\"IS_TESTING\"):\n", - " experiments_list = aiplatform.Experiment.list()\n", - " for experiment in experiments_list:\n", - " experiment.delete()" - ] - } - ], - "metadata": { - "colab": { - "name": "getting_started.ipynb", - "toc_visible": true - }, - "kernelspec": { - "display_name": "Python 3", - "name": "python3" - } - }, - "nbformat": 4, - "nbformat_minor": 0 + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "OsXAs2gcIpbC" + }, + "outputs": [], + "source": [ + "# Copyright 2024 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7ZX50cNFOFBt" + }, + "source": [ + "# Getting Started - Template" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "84eed97da4c4" + }, + "source": [ + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Google
Open in Colab\n", + " \n", + " 		\n", + " \n", + " \"Google
Open in Colab Enterprise\n", + " \n", + " 		\n", + " \n", + " \"Vertex
Open in Vertex AI Workbench\n", + " \n", + " 		\n", + " \n", + " \"GitHub
View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "usd0d_LiOFBt" + }, + "source": [ + "| | |\n", + "|-|-|\n", + "|Author(s) | [Elia Secchi](https://github.com/eliasecchig) |" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MjDmmmDaOFBt" + }, + "source": [ + "## Overview\n", + "\n", + "This tutorial walks you through the process of developing and assessing a chain - a sequence of steps that power an AI application. \n", + "These operations may include interactions with language models, utilization of tools, or data preprocessing steps, aiming to solve a given use case e.g a chatbot that provides grounded information.\n", + "\n", + "You'll learn how to:\n", + "\n", + "1. Build chains using three different approaches:\n", + " - [LangChain Expression Language (LCEL)](https://python.langchain.com/docs/expression_language/)\n", + " - [LangGraph](https://python.langchain.com/docs/langgraph/)\n", + " - A custom Python implementation. This is to enable implementation with other SDKs ( e.g [Vertex AI SDK](https://cloud.google.com/vertex-ai/docs/python-sdk/use-vertex-ai-python-sdk ), [LlamaIndex](https://www.llamaindex.ai/)) and to allow granular control on the sequence of steps in the chain\n", + " \n", + "2. Evaluate the performance of your chains using [Vertex AI Evaluation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview)\n", + "\n", + "Finally, the tutorial discusses next steps for deploying your chain in a production application\n", + "\n", + "By the end of this tutorial, you'll have a solid foundation for developing and refining your own Generative AI chains." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "w-OcPSC8_FUX" + }, + "source": [ + "## Get Started" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c0a13ca7427f" + }, + "source": [ + "### Install required packages using Poetry (Recommended)\n", + "\n", + "This template uses [Poetry](https://python-poetry.org/) as tool to manage project dependencies. \n", + "Poetry makes it easy to install and keep track of the packages your project needs.\n", + "\n", + "To run this notebook with Poetry, follow these steps:\n", + "1. Make sure Poetry is installed. See the [relative guide for installation](https://python-poetry.org/docs/#installation).\n", + "\n", + "2. Make sure that dependencies are installed. From your command line:\n", + " ```bash\n", + " poetry install --with streamlit,jupyter\n", + " ```\n", + "\n", + "3. Run Jupyter:\n", + " ```bash\n", + " poetry run jupyter\n", + " ```\n", + " \n", + "4. Open this notebook in the Jupyter interface." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-7Jso8-FO4N8" + }, + "source": [ + "### (Alternative) Install Vertex AI SDK and other required packages " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tUat7NRq5JDC" + }, + "outputs": [], + "source": [ + "%pip install --quiet --upgrade nest_asyncio\n", + "%pip install --upgrade --user --quiet langchain-core langchain-google-vertexai langchain-google-community langchain langgraph\n", + "%pip install --upgrade --user --quiet \"google-cloud-aiplatform[rapid_evaluation]\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "R5Xep4W9lq-Z" + }, + "source": [ + "### Restart runtime\n", + "\n", + "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n", + "\n", + "The restart might take a minute or longer. After it's restarted, continue to the next step." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "XRvKdaPDTznN" + }, + "outputs": [], + "source": [ + "import IPython\n", + "\n", + "app = IPython.Application.instance()\n", + "app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SbmM4z7FOBpM" + }, + "source": [ + "
\n", + "⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️\n", + "
\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dmWOrTJ3gx13" + }, + "source": [ + "### Authenticate your notebook environment (Colab only)\n", + "\n", + "If you're running this notebook on Google Colab, run the cell below to authenticate your environment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "NyKGtVQjgx13" + }, + "outputs": [], + "source": [ + "# import sys\n", + "\n", + "# if \"google.colab\" in sys.modules:\n", + "# from google.colab import auth\n", + "\n", + "# auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DF4l8DTdWgPY" + }, + "source": [ + "### Set Google Cloud project information and initialize Vertex AI SDK\n", + "\n", + "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n", + "\n", + "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Nqwi-5ufWp_B" + }, + "outputs": [], + "source": [ + "# Use the environment variable if the user doesn't provide Project ID.\n", + "import os\n", + "\n", + "import vertexai\n", + "\n", + "PROJECT_ID = \"production-ai-template\" # @param {type:\"string\", isTemplate: true}\n", + "if PROJECT_ID == \"[your-project-id]\":\n", + " PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n", + "\n", + "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n", + "\n", + "vertexai.init(project=PROJECT_ID, location=LOCATION)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dvhI92xhQTzk" + }, + "source": [ + "### Import libraries" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "869d543465ac" + }, + "outputs": [], + "source": [ + "# Add the parent directory to the Python path. This allows importing modules from the parent directory\n", + "import sys\n", + "\n", + "sys.path.append(\"../\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "146b41115577" + }, + "outputs": [], + "source": [ + "import json\n", + "import pandas as pd\n", + "import yaml\n", + "from json import JSONDecodeError\n", + "from typing import Any, Dict, Iterator, Literal\n", + "\n", + "from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n", + "from langchain_core.runnables import RunnableConfig\n", + "from langchain_core.tools import tool\n", + "from langchain_google_community.vertex_rank import VertexAIRank\n", + "from langchain_google_vertexai import ChatVertexAI, VertexAI, VertexAIEmbeddings\n", + "from langgraph.graph import END, MessagesState, StateGraph\n", + "from langgraph.prebuilt import ToolNode\n", + "from google.cloud import aiplatform\n", + "from vertexai.evaluation import CustomMetric, EvalTask\n", + "\n", + "from app.eval.utils import batch_generate_messages, generate_multiturn_history\n", + "from app.patterns.custom_rag_qa.templates import query_rewrite_template, rag_template\n", + "from app.patterns.custom_rag_qa.vector_store import get_vector_store\n", + "from app.utils.output_types import OnChatModelStreamEvent, OnToolEndEvent, custom_chain" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "675dff1826d4" + }, + "source": [ + "## Chain Interface\n", + "\n", + "This section outlines a possible interface for the chain, which, if implemented, ensures compatibility with the FastAPI server application included in the template. However, it's important to note that you have the flexibility to explore and implement alternative interfaces that suit their specific needs and requirements.\n", + "\n", + "\n", + "### Input Interface\n", + "\n", + "The chain must provide an `astream_events` method that accepts a dictionary with a \"messages\" key.\n", + "The \"messages\" value should be a list of alternating LangChain [HumanMessage](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.human.HumanMessage.html) and [AIMessage](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessage.html) objects.\n", + "\n", + "For example a possible input might be:\n", + "\n", + "```python\n", + "{\n", + " \"messages\": [\n", + " HumanMessage(\"first\"),\n", + " AIMessage(\"a response\"),\n", + " HumanMessage(\"a follow up\")\n", + " ]\n", + "}\n", + "```\n", + "\n", + "Alternatively you can use the shortened form:\n", + "\n", + "```python\n", + "{\n", + " \"messages\": [\n", + " (\"user\", \"first\"),\n", + " (\"ai\", \"a response\"),\n", + " (\"user\", \"a follow up\")\n", + " ]\n", + "}\n", + "```\n", + "\n", + "### Output Interface\n", + "\n", + "All chains use the [LangChain Stream Events (v2) API](https://python.langchain.com/docs/how_to/streaming/#using-stream-events). This API supports various use cases (simple chains, RAG, Agents). This API emits asynchronous events that can be used to stream the chain's output.\n", + "\n", + "LangChain chains (LCEL, LangGraph) automatically implement the `astream_events` API. \n", + "\n", + "We provide examples of emitting `astream_events`-compatible events with custom Python code, allowing implementation with other SDKs (e.g., Vertex AI, LLamaIndex).\n", + "\n", + "### Customizing I/O Interfaces\n", + "\n", + "To modify the Input/Output interface, update `app/server.py` and related unit and integration tests.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "865ba0268d3b" + }, + "source": [ + "## Events supported\n", + "\n", + "The following list defines the events that are captured and supported by the Streamlit frontend." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4ece481555a2" + }, + "outputs": [], + "source": [ + "SUPPORTED_EVENTS = [\n", + " \"on_tool_start\",\n", + " \"on_tool_end\",\n", + " \"on_retriever_start\",\n", + " \"on_retriever_end\",\n", + " \"on_chat_model_stream\",\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "874120e6a4d2" + }, + "source": [ + "### Define the LLM\n", + "We set up the Large Language Model (LLM) for our conversational bot.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0d5c52272898" + }, + "outputs": [], + "source": [ + "llm = ChatVertexAI(model_name=\"gemini-1.5-flash-002\", temperature=0)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "abc296e9da88" + }, + "source": [ + "### Leveraging LangChain LCEL for Efficient Chain Composition\n", + "\n", + "LangChain Expression Language (LCEL) provides a declarative approach to composing chains seamlessly. Key benefits include:\n", + "\n", + "1. Rapid prototyping to production deployment without code changes\n", + "2. Scalability from simple \"prompt + LLM\" chains to complex, multi-step workflows\n", + "3. Enhanced readability and maintainability of chain logic\n", + "\n", + "For comprehensive guidance on LCEL implementation, refer to the [official documentation](https://python.langchain.com/docs/expression_language/get_started).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "25d185f29f42" + }, + "outputs": [], + "source": [ + "template = ChatPromptTemplate.from_messages(\n", + " [\n", + " (\n", + " \"system\",\n", + " \"You are a knowledgeable culinary assistant specializing in providing\"\n", + " \"detailed cooking recipes. Your responses should be informative, engaging, \"\n", + " \"and tailored to the user's specific requests. Include ingredients, \"\n", + " \"step-by-step instructions, cooking times, and any helpful tips or \"\n", + " \"variations. If asked about dietary restrictions or substitutions, offer \"\n", + " \"appropriate alternatives.\",\n", + " ),\n", + " MessagesPlaceholder(variable_name=\"messages\"),\n", + " ]\n", + ")\n", + "\n", + "chain = template | llm" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "216f187e66d7" + }, + "source": [ + "Let's test the chain with a dummy question:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "267a75f53b45" + }, + "outputs": [], + "source": [ + "input_message = {\"messages\": [(\"user\", \"Can you provide me a Lasagne recipe?\")]}\n", + "\n", + "async for event in chain.astream_events(input=input_message, version=\"v2\"):\n", + " if event[\"event\"] in SUPPORTED_EVENTS:\n", + " print(event[\"data\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "b262558c2375" + }, + "source": [ + "This methodology is used for the chain defined in the [`app/chain.py`](../app/chain.py) file.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1fa3c684b527" + }, + "source": [ + "We can also leverage the `invoke` method for synchronous invocation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5fc2bfec1d5b" + }, + "outputs": [], + "source": [ + "response = chain.invoke(input=input_message)\n", + "print(response.content)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "16a9fd5fdf2f" + }, + "source": [ + "### Use LangGraph\n", + "\n", + "LangGraph is a framework for building stateful, multi-actor applications with Large Language Models (LLMs). \n", + "It extends the LangChain library, allowing you to coordinate multiple chains (or actors) across multiple steps of computation in a cyclic manner." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "28b4ad81f8b7" + }, + "outputs": [], + "source": [ + "# 1. Define tools\n", + "@tool\n", + "def search(query: str):\n", + " \"\"\"Simulates a web search. Use it get information on weather. E.g what is the weather like in a region\"\"\"\n", + " if \"sf\" in query.lower() or \"san francisco\" in query.lower():\n", + " return \"It's 60 degrees and foggy.\"\n", + " return \"It's 90 degrees and sunny.\"\n", + "\n", + "\n", + "tools = [search]\n", + "\n", + "# 2. Set up the language model\n", + "llm = llm.bind_tools(tools)\n", + "\n", + "\n", + "# 3. Define workflow components\n", + "def should_continue(state: MessagesState) -> Literal[\"tools\", END]:\n", + " \"\"\"Determines whether to use tools or end the conversation.\"\"\"\n", + " last_message = state[\"messages\"][-1]\n", + " return \"tools\" if last_message.tool_calls else END\n", + "\n", + "\n", + "async def call_model(state: MessagesState, config: RunnableConfig):\n", + " \"\"\"Calls the language model and returns the response.\"\"\"\n", + " response = llm.invoke(state[\"messages\"], config)\n", + " return {\"messages\": response}\n", + "\n", + "\n", + "# 4. Create the workflow graph\n", + "workflow = StateGraph(MessagesState)\n", + "workflow.add_node(\"agent\", call_model)\n", + "workflow.add_node(\"tools\", ToolNode(tools))\n", + "workflow.set_entry_point(\"agent\")\n", + "\n", + "# 5. Define graph edges\n", + "workflow.add_conditional_edges(\"agent\", should_continue)\n", + "workflow.add_edge(\"tools\", \"agent\")\n", + "\n", + "# 6. Compile the workflow\n", + "chain = workflow.compile()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bd27c49717e8" + }, + "source": [ + "Let's test the new chain with a dummy question:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "a33402d68f7b" + }, + "outputs": [], + "source": [ + "input_message = {\"messages\": [(\"user\", \"What is the weather like in NY?\")]}\n", + "\n", + "async for event in chain.astream_events(input=input_message, version=\"v2\"):\n", + " if event[\"event\"] in SUPPORTED_EVENTS:\n", + " print(event[\"data\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2cf0ad148bf8" + }, + "source": [ + "This methodology is used for the chain defined in the [`app/patterns/langgraph_dummy_agent/chain.py`](../app/patterns/langgraph_dummy_agent/chain.py) file.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0ae7dad0966b" + }, + "source": [ + "### Use custom python code\n", + "\n", + "You can also use pure python code to orchestrate the different steps of your chain and emit `astream_events` [API compatible events](https://python.langchain.com/docs/how_to/streaming/#using-stream-events). \n", + "\n", + "This offers full flexibility in how the different steps of a chain are orchestrated and allows you to include other SDK frameworks such as [Vertex AI SDK](https://cloud.google.com/vertex-ai/docs/python-sdk/use-vertex-ai-python-sdk ), [LlamaIndex](https://www.llamaindex.ai/).\n", + "\n", + "We demonstrate this third methodology by implementing a RAG chain. The function `get_vector_store` provides a brute force Vector store (scikit-learn) initialized with data obtained from the [practictioners guide for MLOps](https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ea13644948d2" + }, + "outputs": [], + "source": [ + "llm = ChatVertexAI(model_name=\"gemini-1.5-flash-002\", temperature=0)\n", + "embedding = VertexAIEmbeddings(model_name=\"text-embedding-004\")\n", + "\n", + "\n", + "vector_store = get_vector_store(embedding=embedding)\n", + "retriever = vector_store.as_retriever(search_kwargs={\"k\": 20})\n", + "compressor = VertexAIRank(\n", + " project_id=PROJECT_ID,\n", + " location_id=\"global\",\n", + " ranking_config=\"default_ranking_config\",\n", + " title_field=\"id\",\n", + " top_n=5,\n", + ")\n", + "\n", + "query_gen = query_rewrite_template | llm\n", + "response_chain = rag_template | llm\n", + "\n", + "\n", + "@custom_chain\n", + "def chain(\n", + " input: Dict[str, Any], **kwargs\n", + ") -> Iterator[OnToolEndEvent | OnChatModelStreamEvent]:\n", + " \"\"\"\n", + " Implements a RAG QA chain. Decorated with `custom_chain` to offer LangChain compatible astream_events\n", + " and invoke interface and OpenTelemetry tracing.\n", + " \"\"\"\n", + " # Generate optimized query\n", + " query = query_gen.invoke(input=input).content\n", + "\n", + " # Retrieve and rank documents\n", + " retrieved_docs = retriever.get_relevant_documents(query)\n", + " ranked_docs = compressor.compress_documents(documents=retrieved_docs, query=query)\n", + "\n", + " # Yield tool results metadata\n", + " yield OnToolEndEvent(data={\"input\": {\"query\": query}, \"output\": ranked_docs})\n", + " # Stream LLM response\n", + " for chunk in response_chain.stream(\n", + " input={\"messages\": input[\"messages\"], \"relevant_documents\": ranked_docs}\n", + " ):\n", + " yield OnChatModelStreamEvent(data={\"chunk\": chunk})" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2d59f2641aaf" + }, + "source": [ + "The `@custom_chain` decorator defined in `app/utils/output_types.py`:\n", + "- Enables compatibility with the `astream_events` Langchain API interface by offering a `chain.astream_events` method.\n", + "- Provides an `invoke` method for synchronous invocation. This method can be utilized for evaluation purposes.\n", + "- Adds OpenTelemetry tracing functionality.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1b0d16359361" + }, + "source": [ + "This methodology is used for the chain defined in `app/patterns/custom_rag_qa/chain.py` file.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "82f22f13fde6" + }, + "source": [ + "Let's test the custom chain we just created. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7a1bcac73ff5" + }, + "outputs": [], + "source": [ + "input_message = {\"messages\": [(\"user\", \"What is MLOps?\")]}\n", + "\n", + "async for event in chain.astream_events(input=input_message, version=\"v2\"):\n", + " if event[\"event\"] in SUPPORTED_EVENTS:\n", + " print(event[\"data\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8730b4b6bbff" + }, + "source": [ + "## Evaluation\n", + "\n", + "Evaluation is the activity of assessing the quality of the model's outputs, to gauge its understanding and success in fulfilling the prompt's instructions.\n", + "\n", + "In the context of Generative AI, evaluation extends beyond the evaluation of the model's outputs to include the evaluation of the chain's outputs and in some cases the evaluation of the intermediate steps (for example, the evaluation of the retriever's outputs).\n", + "\n", + "### Vertex AI Evaluation\n", + "To evaluate the chain's outputs, we'll utilize [Vertex AI Evaluation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview) to assess our AI application's performance. \n", + "Vertex AI Evaluation streamlines the evaluation process for generative AI by offering three key features:\n", + "\n", + "- [Pre-built Metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval): It provides a library of ready-to-use metrics for common evaluation tasks, saving you time and effort in defining your own. These metrics cover a range of areas, simplifying the assessment of different aspects of your model's performance.\n", + " \n", + "- [Custom Metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval): Beyond pre-built options, Vertex AI Evaluation allows you to define and implement custom metrics tailored to your specific needs and application requirements. \n", + " \n", + "- Strong Integration with [Vertex AI Experiments](https://cloud.google.com/vertex-ai/docs/experiments/intro-vertex-ai-experiments): Vertex AI Evaluation seamlessly integrates with Vertex AI Experiments, creating a unified workflow for tracking experiments and managing evaluation results.\n", + "\n", + "For a comprehensive list of samples on Vertex AI Evaluation, visit the [official documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-examples)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "e214b9a02547" + }, + "source": [ + "Let's start by defining again a simple chain:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "086298b785fb" + }, + "outputs": [], + "source": [ + "template = ChatPromptTemplate.from_messages(\n", + " [\n", + " (\n", + " \"system\",\n", + " \"You are a knowledgeable culinary assistant specializing in providing\"\n", + " \"detailed cooking recipes. Your responses should be informative, engaging, \"\n", + " \"and tailored to the user's specific requests. Include ingredients, \"\n", + " \"step-by-step instructions, cooking times, and any helpful tips or \"\n", + " \"variations. If asked about dietary restrictions or substitutions, offer \"\n", + " \"appropriate alternatives.\",\n", + " ),\n", + " MessagesPlaceholder(variable_name=\"messages\"),\n", + " ]\n", + ")\n", + "\n", + "chain = template | llm" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "284ec00b23c7" + }, + "source": [ + "We then import the ground truth data we will use for evaluation. Data is stored in [`app/eval/data/chats.yaml`](../app/eval/data/chats.yaml)\n", + "Note: You might need to adjust the path depending on where your Jupyter kernel was initialized.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "f7c05ab676d8" + }, + "outputs": [], + "source": [ + "y = yaml.safe_load(open(\"../app/eval/data/chats.yaml\"))\n", + "df = pd.DataFrame(y)\n", + "df" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "432ce6674c19" + }, + "source": [ + "We leverage the helper functions [`generate_multiturn_history`](../app/eval/utils.py) and [`batch_generate_messages`](../app/eval/utils.py) to prepare the data for evaluation and to generate the responses from the chain.\n", + "\n", + "You can see below the documentation for the two functions." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3a6899faf8b3" + }, + "outputs": [], + "source": [ + "help(generate_multiturn_history)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6afe1b67475f" + }, + "outputs": [], + "source": [ + "help(batch_generate_messages)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "b5e1411cf0f4" + }, + "outputs": [], + "source": [ + "df = generate_multiturn_history(df)\n", + "df" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "9ff1aaed7eb0" + }, + "outputs": [], + "source": [ + "scored_data = batch_generate_messages(df, chain)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "933c797dc79e" + }, + "source": [ + "We extract the user message and the reference (ground truth) message from dataframe so that we can use them for evaluation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5e5ba1acd013" + }, + "outputs": [], + "source": [ + "scored_data[\"user\"] = scored_data[\"human_message\"].apply(lambda x: x[\"content\"])\n", + "scored_data[\"reference\"] = scored_data[\"ai_message\"].apply(lambda x: x[\"content\"])\n", + "scored_data" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dc41cc55703e" + }, + "source": [ + "#### Define a CustomMetric using Gemini model\n", + "\n", + "Define a customized Gemini model-based metric function, with explanations for the score. The registered custom metrics are computed on the client side, without using online evaluation service APIs." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "51c0ffb825e1" + }, + "outputs": [], + "source": [ + "evaluator_llm = ChatVertexAI(\n", + " model_name=\"gemini-1.5-flash-001\",\n", + " temperature=0,\n", + " response_mime_type=\"application/json\",\n", + ")\n", + "\n", + "\n", + "def custom_faithfulness(instance):\n", + " prompt = f\"\"\"You are examining written text content. Here is the text:\n", + "************\n", + "Written content: {instance[\"response\"]}\n", + "************\n", + "Original source data: {instance[\"reference\"]}\n", + "\n", + "Examine the text and determine whether the text is faithful or not.\n", + "Faithfulness refers to how accurately a generated summary reflects the essential information and key concepts present in the original source document.\n", + "A faithful summary stays true to the facts and meaning of the source text, without introducing distortions, hallucinations, or information that wasn't originally there.\n", + "\n", + "Your response must be an explanation of your thinking along with single integer number on a scale of 0-5, 0\n", + "the least faithful and 5 being the most faithful.\n", + "\n", + "Produce results in JSON\n", + "\n", + "Expected format:\n", + "\n", + "```json\n", + "{{\n", + " \"explanation\": \"< your explanation>\",\n", + " \"custom_faithfulness\": \n", + "}}\n", + "```\n", + "\"\"\"\n", + "\n", + " result = evaluator_llm.invoke([(\"human\", prompt)])\n", + " result = json.loads(result.content)\n", + " return result\n", + "\n", + "\n", + "# Register Custom Metric\n", + "custom_faithfulness_metric = CustomMetric(\n", + " name=\"custom_faithfulness\",\n", + " metric_function=custom_faithfulness,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "234f1b868cb9" + }, + "outputs": [], + "source": [ + "experiment_name = \"template-langchain-eval\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f5b15279f9d2" + }, + "source": [ + "We are now ready to run the evaluation. We will use different metrics, combining the custom metric we defined above with some pre-built metrics.\n", + "\n", + "Results of the evaluation will be automatically tagged into the experiment_name we define.\n", + "\n", + "You can click `View Experiment`, to see the experiment in Google Cloud Console." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4d3faf6014f5" + }, + "outputs": [], + "source": [ + "metrics = [\"fluency\", \"safety\", custom_faithfulness_metric]\n", + "\n", + "eval_task = EvalTask(\n", + " dataset=scored_data,\n", + " metrics=metrics,\n", + " experiment=experiment_name,\n", + " metric_column_mapping={\"prompt\": \"user\"},\n", + ")\n", + "eval_result = eval_task.evaluate()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c51deffd590c" + }, + "source": [ + "Once an eval result is produced, we are able to display summary metrics:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "KheOvIvtiRlz" + }, + "outputs": [], + "source": [ + "eval_result.summary_metrics" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JcALGGlwu0p_" + }, + "source": [ + "We are also able to display a pandas dataframe containing a detailed summary of how our eval dataset performed and relative granular metrics." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "9zJ686YYiWJC" + }, + "outputs": [], + "source": [ + "eval_result.metrics_table" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "378025ea68d0" + }, + "source": [ + "## Next Steps\n", + "\n", + "Congratulations on completing the getting started tutorial! You've learned different methodologies to build a chain and how to evaluate it. \n", + "Let's explore the next steps in your journey:\n", + "\n", + "### 1. Prepare for Production\n", + "\n", + "Once you're satisfied with your chain's evaluation results:\n", + "\n", + "1. Write your chain into the [`app/chain.py` file](../app/chain.py).\n", + "2. Remove the `patterns` folder and its associated tests (these are for demonstration only).\n", + "\n", + "### 2. Local Testing\n", + "\n", + "Test your chain using the playground:\n", + "\n", + "```bash\n", + "make playground\n", + "```\n", + "\n", + "This launches af feature-rich playground, including chat curation, user feedback collection, multimodal input, and more!\n", + "\n", + "\n", + "### 3. Production Deployment\n", + "\n", + "Once you are satisfied with the results, you can setup your CI/CD pipelines to deploy your chain to production.\n", + "\n", + "Please refer to the [deployment guide](../deployment/README.md) for more information on how to do that." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7bf6a5b0def0" + }, + "source": [ + "## Cleaning up\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, you can delete the individual resources you created in this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0236a4c1471d" + }, + "outputs": [], + "source": [ + "import os\n", + "\n", + "# Delete Experiments\n", + "delete_experiments = True\n", + "if delete_experiments or os.getenv(\"IS_TESTING\"):\n", + " experiments_list = aiplatform.Experiment.list()\n", + " for experiment in experiments_list:\n", + " experiment.delete()" + ] + } + ], + "metadata": { + "colab": { + "name": "getting_started.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 }