Dolly2-an-OSS-instruction-LLM.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d2b361bf",
   "metadata": {},
   "source": [
    "Copyright (c) 2023 Graphcore Ltd. All rights reserved."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "af14ce5a",
   "metadata": {},
   "source": [
    "# Dolly 2.0 – The World’s First, Truly Open Instruction-Tuned LLM on IPUs – Inference\n",
    "\n",
    "|  Domain | Tasks | Model | Datasets | Workflow |   Number of IPUs   | Execution time |\n",
    "|---------|-------|-------|----------|----------|--------------|--------------|\n",
    "|   NLP   |  Instruction Fine-tuned Text Generation  | Dolly 2.0 | N/A | Inference | recommended: 16 (minimum 4) |  30 min  |\n",
    "\n",
    "[Dolly 2.0](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) is a 12B parameter language model trained and instruction fine-tuned by [Databricks](https://www.databricks.com). By instruction fine-tuning the large language model (LLM), we obtain an LLM better suited for human interactivity. Crucially, Databricks released all code, model weights, and their fine-tuning dataset with an open-source license that permits commercial use. This makes Dolly 2.0 the world's first, truly open-source instruction-tuned LLM. In this notebook, we will show you how to run Dolly 2.0 using Graphcore IPUs on Paperspace with your own prompts.\n",
    "\n",
    "In this notebook you will:\n",
    "- Create and configure a Dolly inference pipeline.\n",
    "- Run Dolly inference on a text prompts to generate answers to user specified questions.\n",
    "\n",
    "This notebook requires a minimum of 4 IPUs to run. This notebook also supports running on a POD16, resulting in a faster pipeline inference speed.\n",
    "\n",
    "[![Join our Slack Community](https://img.shields.io/badge/Slack-Join%20Graphcore's%20Community-blue?style=flat-square&logo=slack)](https://www.graphcore.ai/join-community)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "20b28534",
   "metadata": {},
   "source": [
    "In order to improve usability and support for future users, Graphcore would like to collect information about the\n",
    "applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore:\n",
    "\n",
    "- User progression through the notebook\n",
    "- Notebook details: number of cells, code being run and the output of the cells\n",
    "- Environment details\n",
    "\n",
    "You can disable logging at any time by running `%unload_ext graphcore_cloud_tools.notebook_logging.gc_logger` from any cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c72d24a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -r requirements.txt\n",
    "%load_ext graphcore_cloud_tools.notebook_logging.gc_logger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "965ca0a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "number_of_ipus = int(os.getenv(\"NUM_AVAILABLE_IPU\", 16))\n",
    "number_of_ipus"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7aa108da",
   "metadata": {},
   "source": [
    "## Dolly inference pipeline\n",
    "\n",
    "Let's begin by loading the inference config for Dolly. A configuration suitable for your instance will automatically be selected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5aa39b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.setup import dolly_config_setup\n",
    "\n",
    "config_name = \"dolly_pod4\" if number_of_ipus == 4 else \"dolly_pod16\"\n",
    "config, *_ = dolly_config_setup(\"config/inference.yml\", \"release\", config_name)\n",
    "config"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7e65adf1",
   "metadata": {},
   "source": [
    "Next, we want to create our inference pipeline. Here we define the maximum\n",
    "sequence length and maximum micro batch size. Before executing a model on IPUs\n",
    "it needs to be turned into an executable format by compiling it. This will\n",
    "happen when the pipeline is created. All input shapes must be known before\n",
    "compiling, so if the maximum sequence length or micro batch size is changed, the\n",
    "pipeline will need to be recompiled.\n",
    "\n",
    "Selecting a longer sequence length or larger batch size will use more IPU\n",
    "memory. This means that increasing one may require you to decrease the other.\n",
    "\n",
    "*This cell will take approximately 18 minutes to complete, which includes downloading the model weights.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21144e73",
   "metadata": {},
   "outputs": [],
   "source": [
    "import api\n",
    "\n",
    "# changing these parameters will trigger a recompile.\n",
    "sequence_length = 512  # max 2048\n",
    "micro_batch_size = 4\n",
    "\n",
    "dolly_pipeline = api.DollyPipeline(\n",
    "    config, sequence_length=sequence_length, micro_batch_size=micro_batch_size\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b5ae263d",
   "metadata": {},
   "source": [
    "And run the pipeline!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9e87e5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = dolly_pipeline(\"How many islands are there in Scotland?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d8e787be",
   "metadata": {},
   "source": [
    "We've just run Dolly with the default parameters. Thanks to instruction fine-tuning, we can give prompts to Dolly as if it were a chatbot, as instruction fine-tuning helps turn a base language model into one more suited for interactive behaviour with humans.\n",
    "\n",
    "There are a few sampling parameters we can use to control the behaviour of Dolly:\n",
    "- `temperature` – Indicates whether you want more creative or more factual\n",
    "outputs. A value of `1.0` corresponds to the model's default behaviour, with relatively higher values generating more creative outputs and lower values \n",
    "generating more factual answers.\n",
    "`temperature` must be at least `0.0`\n",
    "which means the model picks the most probable output at each step.\n",
    "If the model starts repeating itself, try increasing the temperature. If the\n",
    "model starts producing off-topic or nonsensical answers, try decreasing the\n",
    "temperature.\n",
    "- `k` – Indicates that only the highest `k` probable tokens can be sampled. Set\n",
    "to 0 to sample across all possible tokens, which means that top k sampling is\n",
    "disabled. The value for `k` must be between a minimum of 0 and a maximum of\n",
    "`config.model.embedding.vocab_size` which is 50,280.\n",
    "- `output_length` – Indicates the number of tokens to sample before stopping. Sampling can stop early if the model outputs `### END`.\n",
    "- `print_final` – If `True`, prints the total time and throughput.\n",
    "- `print_live` – If `True`, the tokens will be printed as they are being sampled. If `False`, only the answer will be returned as a string.\n",
    "- `prompt` – A string containing the question you wish to generate an answer for.\n",
    "\n",
    "Changing these parameters will not result in any recompilation as the input\n",
    "shape to the model has not changed. These can be freely changed and experimented\n",
    "with in the next cell to produce the behaviour you want from Dolly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9693f108",
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = dolly_pipeline(\n",
    "    \"Who was Dolly the sheep?\",\n",
    "    temperature=0.6,\n",
    "    k=5,\n",
    "    output_length=None,\n",
    "    print_live=True,\n",
    "    print_final=True,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b095e572",
   "metadata": {},
   "source": [
    "If you set the `micro_batch_size` during pipeline creation to be greater than one, you can also use the pipeline on a batch of prompts – simply pass a list of prompts to the pipeline. The number of prompts must be less than or equal to the `micro_batch_size`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3fbc2d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = [\n",
    "    \"Write a Haiku about Dolly the Sheep.\",\n",
    "    \"What makes something a Haiku?\",\n",
    "    \"Can sheep write Haikus?\",\n",
    "    \"Where is the best place to purchase my own Dolly the Sheep?\",\n",
    "]\n",
    "answer = dolly_pipeline(\n",
    "    prompt,\n",
    "    temperature=0.6,\n",
    "    k=5,\n",
    "    output_length=None,\n",
    "    print_live=False,\n",
    "    print_final=True,\n",
    ")\n",
    "\n",
    "for p, a in zip(prompt, answer):\n",
    "    print(f\"Instruction: {p}\")\n",
    "    print(f\"Response: {a}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f0a7a7cc",
   "metadata": {},
   "source": [
    "As Dolly is an instruction fine-tuned model, it was trained with a specific prompt format. Internally, the pipeline will transform your prompts into the correct format. To see the full prompt, you can view the `last_instruction_prompt` attribute on the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b85d0409",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dolly_pipeline.last_instruction_prompt[0])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f2ca473c",
   "metadata": {},
   "source": [
    "Remember to detach your pipeline when you are finished to free up resources:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9e91b9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "dolly_pipeline.detach()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7ece4041",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this notebook, we have demonstrated how you can easily run Dolly 2.0 for inference on Graphcore IPUs on text prompts. Instruction fine-tuning is a powerful method of turning a base LLM into one more suited for human interactivity, such as a question-answer model or a chatbot.\n",
    "\n",
    "Although larger instruction LLMs exist with more world knowledge such as ChatGPT, they are closed-source or are subject to non-commercial licensing. This makes Dolly 2.0 a significant milestone as the first of its kind, with future instruction fine-tuned LLMs no doubt to quickly follow. All of which will be truly open."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "73e9821a",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "Check out the full list of [IPU-powered Jupyter Notebooks](https://www.graphcore.ai/ipu-jupyter-notebooks) to see how IPUs perform on other tasks."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}