notebooks/GenAI/AWS_GenAI_Jumpstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cf9eae37-eb34-4afa-89cf-a0b4371978e5",
   "metadata": {},
   "source": [
    "# Deploying a Model from Jumpstart to Sagemaker Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b49b72e7",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "363b6429-2757-4021-b062-a9c2667c2162",
   "metadata": {},
   "source": [
    "Foundational models (FM) are flexible and reusable models that can be used for any use case, these models are extensible allowing the user to customize attributes of the model. Some example for FMs are text generization (e.g., summarizing test), chatbots, and image generation.These model can be accessed through Jumpstart and in this tutorial we will be using [Llama2](https://llama.meta.com/llama2/). To see what other FMs are available in AWS you can go to `Amazon Sagemaker > Jumpstart > Foundational Models`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a2224bf",
   "metadata": {},
   "source": [
    "## Learning Objectives\n",
    "+ Learn about Foundation Models\n",
    "+ Learn how to deploy models to SageMaker via Jumpstart"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67a66b3c",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "You just need access to SageMaker and Model Jumpstart"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be5dd789",
   "metadata": {},
   "source": [
    "## Get Started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d50731d6-20d3-44d7-baf0-98daf19cf487",
   "metadata": {},
   "source": [
    "### Install packages and set up environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ee5a5e2-6302-4ab2-bf4f-c253daa09730",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet sagemaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7780b813-5050-4acb-a3c0-0a961e21958a",
   "metadata": {},
   "source": [
    "Next we will specify the model name and version we want to deploy here we want to deploy the Llama2 chatbot with 7 billion parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cf1429a-314e-49b6-a4f7-16a3e52319af",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id, model_version = \"meta-textgeneration-llama-2-13b-f\", \"2.*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e739c65b-9bd7-4b7f-9ab0-63268b3dda99",
   "metadata": {},
   "source": [
    "Now we will create our endpoint! An endpoint allows us to interact with our model. Sagemaker not only creates the endpoint but at the same time attaches and deploys our model from our endpoint in one step. This will take 1 to 5 mins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c29d29db-3f81-4774-b11f-31d2c2575e74",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "model = JumpStartModel(model_id=model_id, model_version=model_version)\n",
    "predictor = model.deploy()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6285342-c9a7-41bb-aef5-5f7d35966afa",
   "metadata": {},
   "source": [
    "Imports the JsonSerializer which converts .NET objects into their JSON equivalent for Sagemaker to communicate instructions to our endpoint. These instructions are packaged into a payload which we will define in the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecb3c3c1-52da-4357-8665-a3e131c26821",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.serializers import JSONSerializer\n",
    "predictor.serializer = JSONSerializer()\n",
    "predictor.content_type = \"application/json\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca5f0f70-5c77-4025-9fe0-cc91c69d535f",
   "metadata": {},
   "source": [
    "The following function will allows us to pass inputs and parameters so that we can tune our model however we like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf078472-3604-4f42-aa73-95e631dd6acc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_dialog(payload, response):\n",
    "    dialog = payload[\"inputs\"][0]\n",
    "    for msg in dialog:\n",
    "        print(f\"{msg['role'].capitalize()}: {msg['content']}\\n\")\n",
    "    print(\n",
    "        f\"> {response[0]['generation']['role'].capitalize()}: {response[0]['generation']['content']}\"\n",
    "    )\n",
    "    print(\"\\n==================================\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b62d2e4f-66c5-4c1a-bf32-b3993ef10ff7",
   "metadata": {},
   "source": [
    "Now we can define our payload which will hold our input which passes our role as user and the content which will be our question \"what is brain cancer?\". Our parameters allow us to tune our model through max number of new tokens, temperature, and top p:\n",
    "- **Max_New_Tokens:** The size of the output sequence, not including the tokens in the prompt.\n",
    "- **Top_p (nucleus):** The cumulative probability cutoff for token selection. Lower values mean sampling from a smaller, more top-weighted nucleus. Must be a number from 0 to 1.\n",
    "- **Temperature:** Controls randomness, higher values increase diversity meaning a more unique response make the model to think harder. Must be a number from 0 to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59d9b99b-3b74-4e31-b78a-742525330969",
   "metadata": {},
   "outputs": [],
   "source": [
    "payload = {\n",
    "    \"inputs\": [\n",
    "        [\n",
    "            {\"role\": \"user\", \"content\": \"what is brain cancer?\"},\n",
    "        ]\n",
    "    ],\n",
    "    \"parameters\": {\"max_new_tokens\": 512, \"top_p\": 0.9, \"temperature\": 0.6},\n",
    "}\n",
    "try:\n",
    "    response = predictor.predict(payload, custom_attributes=\"accept_eula=true\")\n",
    "    print_dialog(payload, response)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9d6604",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "Congrats! You deployed a model to an endpoint and queried it! "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52572399",
   "metadata": {},
   "source": [
    "## Clean Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec45e77c-113f-4e1c-8297-24e9ac613944",
   "metadata": {},
   "source": [
    "**Warning:** Once you are done don't forget to delete your endpoint, model, buckets, and shutdown or delete your Sagemaker notebook to avoid additional charges!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8ea9087-9d2d-48e1-a9ab-e2b515851930",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint\n",
    "predictor.delete_model()\n",
    "predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}