docs/quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2e63f667",
   "metadata": {},
   "source": [
    "<p>\n",
    "    <a href=\"https://colab.research.google.com/github/explodinggradients/ragas/blob/main/docs/quickstart.ipynb\">\n",
    "        <img alt=\"Open In Colab\" \n",
    "             align=\"left\"\n",
    "             src=\"https://colab.research.google.com/assets/colab-badge.svg\">\n",
    "    </a>\n",
    "    <br>\n",
    "    <h1> Quickstart </h1>\n",
    "</p>\n",
    "\n",
    "welcome to the ragas quickstart. We're going to get you up and running with ragas as qickly as you can so that you can go back to improving your Retrieval Augmented Generation pipelines while this library makes sure your changes are improving your entire pipeline.\n",
    "\n",
    "to kick things of lets start with the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "57585b55",
   "metadata": {},
   "outputs": [],
   "source": [
    "# if using colab uncomment this\n",
    "#!pip install ragas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c77789bb",
   "metadata": {},
   "source": [
    "Ragas also uses OpenAI for running some metrics so make sure you have your openai key ready and available in your environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0b7179f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"your-openai-key\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06c9fc7d",
   "metadata": {},
   "source": [
    "## The Data\n",
    "\n",
    "Ragas performs a `ground_truth` free evaluation of your RAG pipelines. This is because for most people building a gold labeled dataset which represents in the distribution they get in production is a very expensive process.\n",
    "\n",
    "Hence to work with ragas all you need are the following data\n",
    "- question: `list[str]` - These are the questions you RAG pipeline will be evaluated on. \n",
    "- answer: `list[str]` - The answer generated from the RAG pipeline and give to the user.\n",
    "- contexts: `list[list[str]]` - The contexts which where passed into the LLM to answer the question.\n",
    "- ground_truths: `list[list[str]]` - The ground truth answer to the questions.\n",
    "\n",
    "Ideally your list of questions should reflect the questions your users give, including those that you have been problamatic in the past.\n",
    "\n",
    "Here we're using an example dataset from on of the baselines we created for the [Financial Opinion Mining and Question Answering (fiqa) Dataset](https://sites.google.com/view/fiqa/) we created. If you want to want to know more about the baseline, feel free to check the `experiements/baseline` section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b658e02f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset fiqa (/home/jjmachan/.cache/huggingface/datasets/explodinggradients___fiqa/ragas_eval/1.0.0/3dc7b639f5b4b16509a3299a2ceb78bf5fe98ee6b5fee25e7d5e4d290c88efb8)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c415f76ed4f4c969f87986ee05f2fb1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    baseline: Dataset({\n",
       "        features: ['question', 'ground_truths', 'answer', 'contexts'],\n",
       "        num_rows: 30\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data\n",
    "from datasets import load_dataset\n",
    "\n",
    "fiqa_eval = load_dataset(\"explodinggradients/fiqa\", \"ragas_eval\")\n",
    "fiqa_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84aa640f",
   "metadata": {},
   "source": [
    "## Metrics\n",
    "\n",
    "Ragas provides you with a few metrics to evaluate the different aspects of your RAG systems namely\n",
    "\n",
    "1. metrics to evaluate retrieval: offers `context_relevancy` and `context_recall` which give you the measure of the performance of your retrieval system. \n",
    "2. metrics to evaluate generation: offers `faithfulness` which measures hallucinations and `answer_relevancy` which measures how to the point the answers are to the question.\n",
    "\n",
    "The harmonic mean of these 4 aspects gives you the **ragas score** which is a single measure of the performance of your QA system across all the important aspects.\n",
    "\n",
    "now lets import these metrics and understand more about what they denote"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f17bcf9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.metrics import context_relevancy, answer_relevancy, faithfulness, context_recall\n",
    "from ragas.metrics.critique import harmfulness"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef8c5e60",
   "metadata": {},
   "source": [
    "here you can see that we are using 4 metrics, but what do the represent?\n",
    "\n",
    "1. context_relevancy - a measure of how relevent the retrieved context is to the question. Conveys quality of the retrieval pipeline.\n",
    "2. answer_relevancy - a measure of how relevent the answer is to the question\n",
    "3. faithfulness - the factual consistancy of the answer to the context base on the question.\n",
    "4. context_recall: measures the ability of the retriever to retrieve all the necessary information needed to answer the question. \n",
    "5. harmfulness (AspectCritique) - in general, `AspectCritique` is a metric that can be used to quantify various aspects of the answer. Aspects like harmfulness, maliciousness, coherence, correctness, concisenes are available by default but you can easily define your own. Check the [docs](./metrics.md) for more info.\n",
    "\n",
    "**Note:** *by default these metrics are using OpenAI's API to compute the score. If you using this metric make sure you set the environment key `OPENAI_API_KEY` with your API key. You can also try other LLMs for evaluation, check the [llm guide](./guides/llms.ipynb) to learn more*\n",
    "\n",
    "If you're interested in learning more, feel free to check the [docs](https://github.com/explodinggradients/ragas/blob/main/docs/metrics.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d6ecd5a",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Running the evalutation is as simple as calling evaluate on the `Dataset` with the metrics of your choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "22eb6f97",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [context_ relevancy]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████| 2/2 [05:28<00:00, 164.33s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [faithfulness]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████| 2/2 [09:24<00:00, 282.03s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [answer_relevancy]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████| 2/2 [01:22<00:00, 41.37s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [context_recall]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████| 2/2 [13:02<00:00, 391.15s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [harmfulness]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████| 2/2 [02:10<00:00, 65.37s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'ragas_score': 0.4400, 'context_ relevancy': 0.2339, 'faithfulness': 0.7689, 'answer_relevancy': 0.9260, 'context_recall': 0.4107, 'harmfulness': 0.0000}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ragas import evaluate\n",
    "\n",
    "result = evaluate(\n",
    "    fiqa_eval[\"baseline\"],\n",
    "    metrics=[context_relevancy, faithfulness, answer_relevancy, context_recall, harmfulness],\n",
    ")\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2dc0ec2",
   "metadata": {},
   "source": [
    "and there you have the it, all the scores you need. `ragas_score` gives you a single metric that you can use while the other onces measure the different parts of your pipeline.\n",
    "\n",
    "now if we want to dig into the results and figure out examples where your pipeline performed worse or really good you can easily convert it into a pandas array and use your standard analytics tools too!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8686bf53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>ground_truths</th>\n",
       "      <th>answer</th>\n",
       "      <th>contexts</th>\n",
       "      <th>context_ relevancy</th>\n",
       "      <th>faithfulness</th>\n",
       "      <th>answer_relevancy</th>\n",
       "      <th>context_recall</th>\n",
       "      <th>harmfulness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How to deposit a cheque issued to an associate...</td>\n",
       "      <td>[Have the check reissued to the proper payee.J...</td>\n",
       "      <td>\\nThe best way to deposit a cheque issued to a...</td>\n",
       "      <td>[Just have the associate sign the back and the...</td>\n",
       "      <td>0.088301</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.976247</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Can I send a money order from USPS as a business?</td>\n",
       "      <td>[Sure you can.  You can fill in whatever you w...</td>\n",
       "      <td>\\nYes, you can send a money order from USPS as...</td>\n",
       "      <td>[Sure you can.  You can fill in whatever you w...</td>\n",
       "      <td>0.191611</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.883586</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1 EIN doing business under multiple business n...</td>\n",
       "      <td>[You're confusing a lot of things here. Compan...</td>\n",
       "      <td>\\nYes, it is possible to have one EIN doing bu...</td>\n",
       "      <td>[You're confusing a lot of things here. Compan...</td>\n",
       "      <td>0.069420</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.928548</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Applying for and receiving business credit</td>\n",
       "      <td>[\"I'm afraid the great myth of limited liabili...</td>\n",
       "      <td>\\nApplying for and receiving business credit c...</td>\n",
       "      <td>[Set up a meeting with the bank that handles y...</td>\n",
       "      <td>0.408924</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.906223</td>\n",
       "      <td>0.187500</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>401k Transfer After Business Closure</td>\n",
       "      <td>[You should probably consult an attorney. Howe...</td>\n",
       "      <td>\\nIf your employer has closed and you need to ...</td>\n",
       "      <td>[The time horizon for your 401K/IRA is essenti...</td>\n",
       "      <td>0.064802</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.889312</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  How to deposit a cheque issued to an associate...   \n",
       "1  Can I send a money order from USPS as a business?   \n",
       "2  1 EIN doing business under multiple business n...   \n",
       "3         Applying for and receiving business credit   \n",
       "4               401k Transfer After Business Closure   \n",
       "\n",
       "                                       ground_truths  \\\n",
       "0  [Have the check reissued to the proper payee.J...   \n",
       "1  [Sure you can.  You can fill in whatever you w...   \n",
       "2  [You're confusing a lot of things here. Compan...   \n",
       "3  [\"I'm afraid the great myth of limited liabili...   \n",
       "4  [You should probably consult an attorney. Howe...   \n",
       "\n",
       "                                              answer  \\\n",
       "0  \\nThe best way to deposit a cheque issued to a...   \n",
       "1  \\nYes, you can send a money order from USPS as...   \n",
       "2  \\nYes, it is possible to have one EIN doing bu...   \n",
       "3  \\nApplying for and receiving business credit c...   \n",
       "4  \\nIf your employer has closed and you need to ...   \n",
       "\n",
       "                                            contexts  context_ relevancy  \\\n",
       "0  [Just have the associate sign the back and the...            0.088301   \n",
       "1  [Sure you can.  You can fill in whatever you w...            0.191611   \n",
       "2  [You're confusing a lot of things here. Compan...            0.069420   \n",
       "3  [Set up a meeting with the bank that handles y...            0.408924   \n",
       "4  [The time horizon for your 401K/IRA is essenti...            0.064802   \n",
       "\n",
       "   faithfulness  answer_relevancy  context_recall  harmfulness  \n",
       "0      0.666667          0.976247        0.111111            0  \n",
       "1      1.000000          0.883586        0.800000            0  \n",
       "2      1.000000          0.928548        1.000000            0  \n",
       "3      1.000000          0.906223        0.187500            0  \n",
       "4      0.666667          0.889312        0.000000            0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = result.to_pandas()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f668fce1",
   "metadata": {},
   "source": [
    "And thats it!\n",
    "\n",
    "You can check out the [ragas in action] notebook to get a feel of what is like to use it while trying to improve your pipelines.\n",
    "\n",
    "if you have any suggestion/feedbacks/things your not happy about, please do share it in the [issue section](https://github.com/explodinggradients/ragas/issues). We love hearing from you 😁"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}