L3_Building_an_Agent_Reasoning_Loop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "64ec2f0f",
   "metadata": {},
   "source": [
    "# Lesson 3: Building an Agent Reasoning Loop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0d7f1cf",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b07baa43-7a51-4c39-91cc-aa0d9619b69f",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from helper import get_openai_api_key\n",
    "OPENAI_API_KEY = get_openai_api_key()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bb245025-2b72-4b86-ae14-7f0a04439965",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import openai\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \" \"\n",
    "openai.api_key = os.environ[\"OPENAI_API_KEY\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dcfa86a3-c7f2-41fa-b8b6-5617659ec36a",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d3af4bb",
   "metadata": {},
   "source": [
    "## Load the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d8bfb34",
   "metadata": {
    "tags": []
   },
   "source": [
    "To download this paper, below is the needed code:\n",
    "\n",
    "#!wget \"https://openreview.net/pdf?id=VtmBAGCN7o\" -O metagpt.pdf\n",
    "\n",
    "**Note**: The pdf file is included with this lesson. To access it, go to the `File` menu and select`Open...`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb741560",
   "metadata": {},
   "source": [
    "## Setup the Query Tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "77464fb2-5ace-4839-9032-a020df8d4259",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils2 import get_doc_tools\n",
    "\n",
    "vector_tool, summary_tool = get_doc_tools(\"metagpt.pdf\", \"metagpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40aae3fc",
   "metadata": {},
   "source": [
    "## Setup Function Calling Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ff4f5199-d02c-47b0-a9ab-cf72c8a506a3",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9365d78d-8e9f-4f22-8d57-35a4c6aa6baf",
   "metadata": {
    "height": 181,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core.agent import FunctionCallingAgentWorker\n",
    "from llama_index.core.agent import AgentRunner\n",
    "\n",
    "agent_worker = FunctionCallingAgentWorker.from_tools(\n",
    "    [vector_tool, summary_tool], \n",
    "    llm=llm, \n",
    "    verbose=True\n",
    ")\n",
    "agent = AgentRunner(agent_worker)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5a9535d7-0baf-4905-ad16-5fb903d33b85",
   "metadata": {
    "height": 96,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Tell me about the agent roles in MetaGPT, and then how they communicate with each other.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"agent roles in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "The agent roles in MetaGPT include the Product Manager, Architect, Project Manager, Engineer, and QA Engineer. The Product Manager is responsible for analyzing user requirements and generating the Product Requirement Document (PRD). The Architect translates these requirements into system design components. The Project Manager distributes tasks based on the system design, while Engineers execute the designated code. The QA Engineer formulates test cases to ensure code quality. Each role has specific responsibilities and contributes to the collaborative software development process within the MetaGPT framework.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"communication between agent roles in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "Communication between agent roles in MetaGPT is structured and formalized, primarily conducted through documents and diagrams rather than dialogue. Agents generate structured outputs like system interface designs and sequence flow diagrams, which are essential deliverables for other roles. The system employs a shared message pool where agents can publish and access messages directly, eliminating the need for one-to-one communication. Additionally, a subscription mechanism allows agents to select and follow information relevant to their role profiles, reducing information overload and improving communication efficiency. This structured approach ensures effective collaboration and streamlined information exchange within the multi-agent system.\n",
      "=== LLM Response ===\n",
      "In MetaGPT, the agent roles include the Product Manager, Architect, Project Manager, Engineer, and QA Engineer. Each role has specific responsibilities in the software development process. The Product Manager analyzes user requirements, the Architect translates requirements into system design, the Project Manager distributes tasks, Engineers execute code, and the QA Engineer ensures code quality through test cases.\n",
      "\n",
      "Communication between agent roles in MetaGPT is structured and formalized. It is primarily conducted through documents and diagrams rather than dialogue. Agents generate outputs like system interface designs and sequence flow diagrams, which are essential for other roles. The system uses a shared message pool for publishing and accessing messages directly, eliminating the need for one-to-one communication. A subscription mechanism allows agents to select and follow relevant information, reducing information overload and improving communication efficiency. This structured approach ensures effective collaboration and streamlined information exchange within the multi-agent system.\n"
     ]
    }
   ],
   "source": [
    "response = agent.query(\n",
    "    \"Tell me about the agent roles in MetaGPT, \"\n",
    "    \"and then how they communicate with each other.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bcf74ec4-559f-4284-9ed0-817d26951c54",
   "metadata": {
    "height": 45,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_label: 1\n",
      "file_name: metagpt.pdf\n",
      "file_path: metagpt.pdf\n",
      "file_type: application/pdf\n",
      "file_size: 16911937\n",
      "creation_date: 2024-05-09\n",
      "last_modified_date: 2024-05-09\n",
      "\n",
      "Preprint\n",
      "METAGPT: M ETA PROGRAMMING FOR A\n",
      "MULTI -AGENT COLLABORATIVE FRAMEWORK\n",
      "Sirui Hong1∗, Mingchen Zhuge2∗, Jonathan Chen1, Xiawu Zheng3, Yuheng Cheng4,\n",
      "Ceyao Zhang4,Jinlin Wang1,Zili Wang ,Steven Ka Shing Yau5,Zijuan Lin4,\n",
      "Liyang Zhou6,Chenyu Ran1,Lingfeng Xiao1,7,Chenglin Wu1†,J¨urgen Schmidhuber2,8\n",
      "1DeepWisdom,2AI Initiative, King Abdullah University of Science and Technology,\n",
      "3Xiamen University,4The Chinese University of Hong Kong, Shenzhen,\n",
      "5Nanjing University,6University of Pennsylvania,\n",
      "7University of California, Berkeley,8The Swiss AI Lab IDSIA/USI/SUPSI\n",
      "ABSTRACT\n",
      "Remarkable progress has been made on automated problem solving through so-\n",
      "cieties of agents based on large language models (LLMs). Existing LLM-based\n",
      "multi-agent systems can already solve simple dialogue tasks. Solutions to more\n",
      "complex tasks, however, are complicated through logic inconsistencies due to\n",
      "cascading hallucinations caused by naively chaining LLMs. Here we introduce\n",
      "MetaGPT, an innovative meta-programming framework incorporating efficient\n",
      "human workflows into LLM-based multi-agent collaborations. MetaGPT en-\n",
      "codes Standardized Operating Procedures (SOPs) into prompt sequences for more\n",
      "streamlined workflows, thus allowing agents with human-like domain expertise\n",
      "to verify intermediate results and reduce errors. MetaGPT utilizes an assembly\n",
      "line paradigm to assign diverse roles to various agents, efficiently breaking down\n",
      "complex tasks into subtasks involving many agents working together. On col-\n",
      "laborative software engineering benchmarks, MetaGPT generates more coherent\n",
      "solutions than previous chat-based multi-agent systems. Our project can be found\n",
      "at https://github.com/geekan/MetaGPT.\n",
      "1 I NTRODUCTION\n",
      "Autonomous agents utilizing Large Language Models (LLMs) offer promising opportunities to en-\n",
      "hance and replicate human workflows. In real-world applications, however, existing systems (Park\n",
      "et al., 2023; Zhuge et al., 2023; Cai et al., 2023; Wang et al., 2023c; Li et al., 2023; Du et al., 2023;\n",
      "Liang et al., 2023; Hao et al., 2023) tend to oversimplify the complexities. They struggle to achieve\n",
      "effective, coherent, and accurate problem-solving processes, particularly when there is a need for\n",
      "meaningful collaborative interaction (Chen et al., 2024; Zhang et al., 2023; Dong et al., 2023; Zhou\n",
      "et al., 2023; Qian et al., 2023).\n",
      "Through extensive collaborative practice, humans have developed widely accepted Standardized\n",
      "Operating Procedures (SOPs) across various domains (Belbin, 2012; Manifesto, 2001; DeMarco &\n",
      "Lister, 2013). These SOPs play a critical role in supporting task decomposition and effective coor-\n",
      "dination. Furthermore, SOPs outline the responsibilities of each team member, while establishing\n",
      "standards for intermediate outputs. Well-defined SOPs improve the consistent and accurate exe-\n",
      "cution of tasks that align with defined roles and quality standards (Belbin, 2012; Manifesto, 2001;\n",
      "DeMarco & Lister, 2013; Wooldridge & Jennings, 1998). For instance, in a software company,\n",
      "Product Managers analyze competition and user needs to create Product Requirements Documents\n",
      "(PRDs) using a standardized structure, to guide the developmental process.\n",
      "Inspired by such ideas, we design a promising GPT -based Meta -Programming framework called\n",
      "MetaGPT that significantly benefits from SOPs. Unlike other works (Li et al., 2023; Qian et al.,\n",
      "2023), MetaGPT requires agents to generate structured outputs, such as high-quality requirements\n",
      "∗These authors contributed equally to this work.\n",
      "†Chenglin Wu (alexanderwu@fuzhi.ai) is the corresponding author, affiliated with DeepWisdom.\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(response.source_nodes[0].get_content(metadata_mode=\"all\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b61b690-f7da-43f6-8608-bc773566e574",
   "metadata": {},
   "source": [
    "实现了Chat效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6b28c184-0b65-4e38-808e-d91a285aaefe",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Tell me about the evaluation datasets used.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"evaluation datasets used in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "The evaluation datasets used in MetaGPT include HumanEval, MBPP, and a self-generated benchmark named SoftwareDev.\n",
      "=== LLM Response ===\n",
      "The evaluation datasets used in MetaGPT include HumanEval, MBPP, and a self-generated benchmark named SoftwareDev.\n"
     ]
    }
   ],
   "source": [
    "response = agent.chat(\n",
    "    \"Tell me about the evaluation datasets used.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d9586cef-21b5-4732-b95d-619462b4aaf6",
   "metadata": {
    "height": 45,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Tell me the results over one of the above datasets.\n",
      "=== Calling Function ===\n",
      "Calling function: vector_tool_metagpt with args: {\"query\": \"results over HumanEval dataset\", \"page_numbers\": [\"7\"]}\n",
      "=== Function Output ===\n",
      "MetaGPT achieved 85.9% and 87.7% Pass rates over the HumanEval dataset.\n",
      "=== LLM Response ===\n",
      "MetaGPT achieved 85.9% and 87.7% Pass rates over the HumanEval dataset.\n"
     ]
    }
   ],
   "source": [
    "response = agent.chat(\"Tell me the results over one of the above datasets.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cc4e983",
   "metadata": {},
   "source": [
    "## Lower-Level: Debuggability and Control"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "55abad72-b189-471a-accc-1621fd19c804",
   "metadata": {
    "height": 130,
    "tags": []
   },
   "outputs": [],
   "source": [
    "agent_worker = FunctionCallingAgentWorker.from_tools(\n",
    "    [vector_tool, summary_tool], \n",
    "    llm=llm, \n",
    "    verbose=True\n",
    ")\n",
    "agent = AgentRunner(agent_worker)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "18e911aa-4640-4f89-99c8-6cdf6aff07c6",
   "metadata": {
    "height": 96,
    "tags": []
   },
   "outputs": [],
   "source": [
    "task = agent.create_task(\n",
    "    \"Tell me about the agent roles in MetaGPT, \"\n",
    "    \"and then how they communicate with each other.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "476998e0-fd0e-4d9f-aa73-72e3d0d91071",
   "metadata": {},
   "source": [
    "逐步分解，完成单项任务么？上了agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5eaf0b88-e03a-4dd9-91f6-f6f0c8758e64",
   "metadata": {
    "height": 45,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Tell me about the agent roles in MetaGPT, and then how they communicate with each other.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"agent roles in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "The agent roles in MetaGPT include Product Manager, Architect, Project Manager, Engineer, and QA Engineer. The Product Manager is responsible for analyzing user requirements and creating detailed Product Requirement Documents (PRDs). The Architect translates requirements into system design components. The Project Manager handles task distribution. Engineers execute designated classes and functions based on the design, while the QA Engineer formulates test cases to ensure code quality. These roles work together in a structured workflow following Standard Operating Procedures (SOPs) to efficiently collaborate and produce software solutions in MetaGPT.\n"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a8e77fac-8734-4071-a672-b3a9f30e2bf1",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num completed for task 793236b2-b4d6-4773-abef-1ccffe895a7f: 1\n",
      "The agent roles in MetaGPT include Product Manager, Architect, Project Manager, Engineer, and QA Engineer. The Product Manager is responsible for analyzing user requirements and creating detailed Product Requirement Documents (PRDs). The Architect translates requirements into system design components. The Project Manager handles task distribution. Engineers execute designated classes and functions based on the design, while the QA Engineer formulates test cases to ensure code quality. These roles work together in a structured workflow following Standard Operating Procedures (SOPs) to efficiently collaborate and produce software solutions in MetaGPT.\n"
     ]
    }
   ],
   "source": [
    "completed_steps = agent.get_completed_steps(task.task_id)\n",
    "print(f\"Num completed for task {task.task_id}: {len(completed_steps)}\")\n",
    "print(completed_steps[0].output.sources[0].raw_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "db8de410-4b82-4daf-93da-28da57cbb0bb",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num upcoming steps for task 793236b2-b4d6-4773-abef-1ccffe895a7f: 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TaskStep(task_id='793236b2-b4d6-4773-abef-1ccffe895a7f', step_id='af519fde-50f8-4351-a133-39a1b6cdf787', input=None, step_state={}, next_steps={}, prev_steps={}, is_ready=True)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "upcoming_steps = agent.get_upcoming_steps(task.task_id)\n",
    "print(f\"Num upcoming steps for task {task.task_id}: {len(upcoming_steps)}\")\n",
    "upcoming_steps[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "dc352582-2c17-46ef-ba80-0f571e920c3c",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: What about how agents share information?\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"how agents share information in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "Agents in MetaGPT share information through a structured communication protocol that includes a shared message pool. This pool allows agents to publish structured messages and subscribe to relevant messages based on their profiles. By utilizing this communication interface, agents can efficiently exchange directional information and public information from the environment. Additionally, agents can access past messages stored in memory to compare them with the Product Requirements Document (PRD), system design, and code files, ensuring effective information sharing and collaboration within the framework.\n"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(\n",
    "    task.task_id, input=\"What about how agents share information?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "be80661f-81b1-45fc-b0ba-33a04dae849b",
   "metadata": {
    "height": 62,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== LLM Response ===\n",
      "In MetaGPT, agents share information through a structured communication protocol that includes a shared message pool. This pool enables agents to publish structured messages and subscribe to relevant messages based on their profiles. By utilizing this communication interface, agents can efficiently exchange directional information and public information from the environment. Additionally, agents can access past messages stored in memory to compare them with the Product Requirements Document (PRD), system design, and code files, ensuring effective information sharing and collaboration within the framework.\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(task.task_id)\n",
    "print(step_output.is_last)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4496328c-e6d5-4722-a8df-78a73a441b3c",
   "metadata": {
    "height": 45,
    "tags": []
   },
   "outputs": [],
   "source": [
    "response = agent.finalize_response(task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "601d1bed-78b2-4512-87ac-aec5ce5d8494",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assistant: In MetaGPT, agents share information through a structured communication protocol that includes a shared message pool. This pool enables agents to publish structured messages and subscribe to relevant messages based on their profiles. By utilizing this communication interface, agents can efficiently exchange directional information and public information from the environment. Additionally, agents can access past messages stored in memory to compare them with the Product Requirements Document (PRD), system design, and code files, ensuring effective information sharing and collaboration within the framework.\n"
     ]
    }
   ],
   "source": [
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d1b4e50-5e2b-4057-b088-dcc511e75157",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}