L5_exploring-circleci-config-file.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "284bf345-2365-4908-a796-382bcd077ac3",
   "metadata": {},
   "source": [
    "# Optional: Exploring the CircleCI config file\n",
    "\n",
    "This step-by-step tutorial shows you  how to create a full CI/CD pipeline in CircleCI. However, before you start building this pipeline, you need to run the following setup:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9d246136-83c9-4d0f-997b-22cf332ae0f6",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5736b45",
   "metadata": {},
   "source": [
    "## Load API tokens for our 3rd party APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7170f931",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "from utils import get_circle_api_key\n",
    "cci_api_key = get_circle_api_key()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "66c07fcd",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "from utils import get_gh_api_key\n",
    "gh_api_key = get_gh_api_key()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1549072c",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "from utils import get_openai_api_key\n",
    "openai_api_key = get_openai_api_key()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "150d5ae3",
   "metadata": {},
   "source": [
    "### Set up our github branch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "563c7075",
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'CircleCI-Learning/llmops-course'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils import get_repo_name\n",
    "course_repo = get_repo_name()\n",
    "course_repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3535984f",
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'dl-cci-fast-growing-pillow-48'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils import get_branch\n",
    "course_branch = get_branch()\n",
    "course_branch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b9fe8d5-ed21-4a72-9f9d-c8b8b849537e",
   "metadata": {},
   "source": [
    "## Basic CircleCI configuration structure\n",
    "\n",
    "CircleCI uses configuration as code, so everything your pipeline does is stored in a [CircleCI configuration file](https://circleci.com/docs/config-intro/).\n",
    "\n",
    "All CI/CD pipelines in CircleCI are defined in a single YAML file at the path `.circleci/config.yml` in your project's Git repository. This tutorial explains the example configuration file used throughout the course.\n",
    "\n",
    "There are three key components to every CircleCI config file: **jobs**, **commands**, and **workflows**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cfe3f83-e35e-4071-9bd1-faaebbbe1587",
   "metadata": {},
   "source": [
    "### Jobs\n",
    "\n",
    "A [job](https://circleci.com/docs/jobs-steps/) is the basic unit of automation in a CircleCI CI/CD pipeline. Each job represents one of the  high-level tasks you want to automate, containing the commands you want to execute to complete the task.\n",
    "\n",
    "Each job  is defined in its own configuration section. The example snippet below defines a job, including the [Docker image](https://circleci.com/docs/using-docker/) that will be used to execute this job. When CircleCI executes this job, a cloud execution environment is created on CircleCI’s infrastructure, running the specified Docker image.\n",
    "\n",
    "Any commands defined in the job will be run in this environment (including bash commands, Python scripts, and any other command or application executed inside the job). There are many other execution environments — we’ll explore these later in this lesson.\n",
    "\n",
    "Here is an example job named `run-hello-world` that spins up an execution environment based on the `cimg/python` Docker image and then prints “Hello World!” to the console. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "38e3c7d5-2b5a-48de-8e38-b63006fd068e",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jobs:\r\n",
      "  run-hello-world:  # This is the name of the job - feel free to change it to better match what you're trying to do!\r\n",
      "    docker:\r\n",
      "      - image: cimg/python:3.10.5\r\n",
      "    steps:\r\n",
      "      - run:\r\n",
      "          name: Print Hello World\r\n",
      "          command: echo \"Hello World!\""
     ]
    }
   ],
   "source": [
    "!cat hello_world.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d6a33b0-0f1e-4683-90e9-bf24661e013b",
   "metadata": {},
   "source": [
    "Below, we have an entire config file that runs this job. This file uses other CircleCI configuration options that will be discussed later. For now, you can just focus on the `jobs` section at the bottom of this file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d634a8fd-0d8a-48ac-9d7c-c611dda5b698",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "version: 2.1\r\n",
      "orbs:\r\n",
      "  python: circleci/python@2.1.1\r\n",
      "\r\n",
      "parameters:\r\n",
      "  eval-mode:\r\n",
      "    type: string\r\n",
      "    default: \"commit\"\r\n",
      "\r\n",
      "workflows:\r\n",
      "  evaluate-app:  # This is the name of the workflow - feel free to change it to better match your workflow.\r\n",
      "    # Inside the workflow, you define the jobs you want to run.\r\n",
      "    # For more details on extending your workflow, see the configuration docs: https://circleci.com/docs/2.0/configuration-reference/#workflows\r\n",
      "    jobs:\r\n",
      "      - run-hello-world:\r\n",
      "          context:\r\n",
      "            - dl-ai-courses\r\n",
      "\r\n",
      "jobs:\r\n",
      "  run-hello-world:  # This is the name of the job - feel free to change it to better match what you're trying to do!\r\n",
      "    docker:\r\n",
      "      - image: cimg/python:3.10.5\r\n",
      "    steps:\r\n",
      "      - run:\r\n",
      "          name: Print Hello World\r\n",
      "          command: echo \"Hello World!\"\r\n",
      "          "
     ]
    }
   ],
   "source": [
    "!cat circle_config_v1.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7718d0f9-4a9a-41e5-8f3e-e2260a450fbb",
   "metadata": {},
   "source": [
    "You can trigger this job by interacting with the [CircleCI API](https://circleci.com/docs/api-intro/). This is done by calling the helper functions below defined in `utils.py`. After the workflow begins running successfully, a link will be provided to view the results. Note that it may take a few seconds for the APIs to finish executing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e7a21324-0c91-4c31-bfae-d5ddf55cc759",
   "metadata": {
    "height": 47
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CircleCI-Learning/llmops-course\n",
      "dl-cci-fast-growing-pillow-48\n"
     ]
    }
   ],
   "source": [
    "print(course_repo)\n",
    "print(course_branch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9c469f57-502d-4428-b379-f755dd1c4a18",
   "metadata": {
    "height": 115
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "uploading test_assistant.py\n",
      "uploading app.py\n"
     ]
    }
   ],
   "source": [
    "from utils import push_files\n",
    "push_files(course_repo, \n",
    "           course_branch, \n",
    "           [\"app.py\", \"test_assistant.py\"],\n",
    "           \"circle_config_v1.yml\"\n",
    "          )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "97c4a639-52fd-4866-a8af-0236f7667dc5",
   "metadata": {
    "height": 81
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please visit https://app.circleci.com/pipelines/github/CircleCI-Learning/llmops-course/1074\n"
     ]
    }
   ],
   "source": [
    "from utils import trigger_commit_evals\n",
    "trigger_commit_evals(course_repo, \n",
    "                     course_branch, \n",
    "                     cci_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e91def7-7183-4ab9-9c59-80269a8c0ce3",
   "metadata": {},
   "source": [
    "### Commands\n",
    "\n",
    "[Commands](https://circleci.com/docs/configuration-reference/#commands) are the individual steps executed in sequence inside of jobs. Commands can either be defined inline with the `run` keyword as we have done above, or created and named outside of jobs so that they can be used in multiple jobs. This second method of defining commands mimics the principle of DRY (don't repeat yourself) development, in which we create functions once and use them over and over.\n",
    "\n",
    "When using the `run` keyword to define inline commands, the `name` field determines what you will see to identify the command in the CircleCI dashboard and the `command` field determines which command will be executed on the command line of the executor.\n",
    "\n",
    "The example code below reworks the \"Hello World\" job to run the first type of eval that was covered in Lesson 2. Some additional required steps have been added as well: the [checkout](https://circleci.com/docs/configuration-reference/#checkout) step is a built-in CircleCI command that checks out the code from the repository containing the config file, while the `python/install-packages` step installs any Python packages included in `requirements.txt`.\n",
    "\n",
    "The `python/install-packages` step is defined in the [Python orb](https://circleci.com/developer/orbs/orb/circleci/python) image we are using in this configuration. We'll go into more depth on orbs at the end of the tutorial.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7ba084ce-1b9f-4fad-a565-3cab400664d7",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "version: 2.1\r\n",
      "orbs:\r\n",
      "  python: circleci/python@2.1.1\r\n",
      "\r\n",
      "parameters:\r\n",
      "  eval-mode:\r\n",
      "    type: string\r\n",
      "    default: \"commit\"\r\n",
      "\r\n",
      "workflows:\r\n",
      "  evaluate-app:  # This is the name of the workflow - feel free to change it to better match your workflow.\r\n",
      "    # Inside the workflow, you define the jobs you want to run.\r\n",
      "    # For more details on extending your workflow, see the configuration docs: https://circleci.com/docs/2.0/configuration-reference/#workflows\r\n",
      "    jobs:\r\n",
      "      - run-commit-evals:\r\n",
      "          context:\r\n",
      "            - dl-ai-courses\r\n",
      "\r\n",
      "jobs:\r\n",
      "  run-commit-evals:  # This is the name of the job - feel free to change it to better match what you're trying to do!\r\n",
      "    # These next lines define the Docker executor: https://circleci.com/docs/2.0/executor-types/\r\n",
      "    # You can specify an image from Docker Hub or use one of the convenience images from CircleCI's Developer Hub\r\n",
      "    # A list of available CircleCI Docker convenience images is available here: https://circleci.com/developer/images/image/cimg/python\r\n",
      "    # The executor is the environment in which the steps below will be executed - below will use a Python 3.10 container\r\n",
      "    # Change the version below to your required version of Python\r\n",
      "    docker:\r\n",
      "      - image: cimg/python:3.10.5\r\n",
      "    # Check out the code as the first step. This is a dedicated CircleCI step.\r\n",
      "    # The Python orb's install-packages step will install the dependencies from a Pipfile via Pipenv by default.\r\n",
      "    # Here we're making sure we use just use the system-wide pip. By default it uses the project root's requirements.txt.\r\n",
      "    # Then run your tests!\r\n",
      "    # CircleCI will report the results back to your VCS provider.\r\n",
      "    steps:\r\n",
      "      - checkout\r\n",
      "      - python/install-packages:\r\n",
      "          pkg-manager: pip\r\n",
      "          # app-dir: ~/project/package-directory/  # If your requirements.txt isn't in the root directory.\r\n",
      "          # pip-dependency-file: test-requirements.txt  # if you have a different name for your requirements file, maybe one that combines your runtime and test requirements.\r\n",
      "      - run:\r\n",
      "          name: Run assistant evals.\r\n",
      "          command: python -m pytest test_assistant.py\r\n",
      "          "
     ]
    }
   ],
   "source": [
    "!cat circle_config_v2.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75b8be56-8a09-40b8-b383-944f0435a2e3",
   "metadata": {},
   "source": [
    "As you can see, this is running a Python unit test file `test_assistant.py` through Pytest instead of just echoing \"Hello World\". You can use the same `app.py` and `test_assistant.py` files that you used in the first lesson.\n",
    "\n",
    "As in the previous example, this job can be triggered by interacting with the CircleCI API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "48b590ad-5ab8-49e1-9fca-267dd5c91f61",
   "metadata": {
    "height": 115
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "uploading test_assistant.py\n",
      "uploading app.py\n",
      "dl-cci-fast-growing-pillow-48 already exists in the repository pushing updated changes\n"
     ]
    }
   ],
   "source": [
    "from utils import push_files\n",
    "push_files(course_repo, \n",
    "           course_branch, \n",
    "           [\"app.py\", \"test_assistant.py\"],\n",
    "           \"circle_config_v2.yml\"\n",
    "          )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "59cc721a-04ca-4273-bb48-d696f94d9df2",
   "metadata": {
    "height": 47
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please visit https://app.circleci.com/pipelines/github/CircleCI-Learning/llmops-course/1075\n"
     ]
    }
   ],
   "source": [
    "from utils import trigger_commit_evals\n",
    "trigger_commit_evals(course_repo, course_branch, cci_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3088b9ac-1ccc-4c0f-a07b-68c77889bcd3",
   "metadata": {},
   "source": [
    "### Workflows\n",
    "\n",
    "To run multiple types of evals, you need to define multiple jobs in your config file. You can use workflows in CircleCI to orchestrate these.\n",
    "\n",
    "Put simply, a [workflow](https://circleci.com/docs/workflows/#overview) is used to orchestrate jobs. You can define multiple workflows that will run when you push to specific branches in your repository or run them on a schedule.  Workflows log their output to the [CircleCI dashboard](https://circleci.com/docs/introduction-to-the-circleci-web-app/), stopping when a job fails so that the output can be inspected.\n",
    "\n",
    "Here's how you can add jobs for the other eval types into your workflow. Now, when the `evaluate-app` workflow is triggered, the `run-commit-evals` job **and** the new `run-pre-release-evals` job will be run. **Any number of jobs, containing any number of commands, can be defined and added to this workflow.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "464b06e3-7887-4dce-a31d-20bbcb46ede1",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "version: 2.1\r\n",
      "orbs:\r\n",
      "  # The Python orb contains a set of prepackaged CircleCI configurations you can use repeatedly in your configuration files\r\n",
      "  python: circleci/python@2.1.1\r\n",
      "\r\n",
      "parameters:\r\n",
      "  eval-mode:\r\n",
      "    type: string\r\n",
      "    default: \"commit\"\r\n",
      "\r\n",
      "\r\n",
      "workflows:\r\n",
      "  evaluate-app:  # This is the name of the workflow - feel free to change it to better match your workflow.\r\n",
      "    # Inside the workflow, you define the jobs you want to run.\r\n",
      "    # For more details on extending your workflow, see the configuration docs: https://circleci.com/docs/2.0/configuration-reference/#workflows\r\n",
      "    jobs:\r\n",
      "      - run-commit-evals:\r\n",
      "          context:\r\n",
      "            - dl-ai-courses\r\n",
      "      # Note: In a real application, you'd run these on commits to the main branch\r\n",
      "      # for this course, we're using a parameter instead.\r\n",
      "      - run-pre-release-evals:\r\n",
      "          context:\r\n",
      "            - dl-ai-courses\r\n",
      "\r\n",
      "jobs:\r\n",
      "  run-commit-evals:  # This is the name of the job - feel free to change it to better match what you're trying to do!\r\n",
      "    # These next lines define the Docker executor: https://circleci.com/docs/2.0/executor-types/\r\n",
      "    # You can specify an image from Docker Hub or use one of the convenience images from CircleCI's Developer Hub\r\n",
      "    # A list of available CircleCI Docker convenience images is available here: https://circleci.com/developer/images/image/cimg/python\r\n",
      "    # The executor is the environment in which the steps below will be executed - below will use a Python 3.10 container\r\n",
      "    # Change the version below to your required version of Python\r\n",
      "    docker:\r\n",
      "      - image: cimg/python:3.10.5\r\n",
      "    # Check out the code as the first step. This is a dedicated CircleCI step.\r\n",
      "    # The Python orb's install-packages step will install the dependencies from a Pipfile via Pipenv by default.\r\n",
      "    # Here we're making sure we use just use the system-wide pip. By default it uses the project root's requirements.txt.\r\n",
      "    # Then run your tests!\r\n",
      "    # CircleCI will report the results back to your VCS provider.\r\n",
      "    steps:\r\n",
      "      - checkout\r\n",
      "      - python/install-packages:\r\n",
      "          pkg-manager: pip\r\n",
      "          # app-dir: ~/project/package-directory/  # If your requirements.txt isn't in the root directory.\r\n",
      "          # pip-dependency-file: test-requirements.txt  # if you have a different name for your requirements file, maybe one that combines your runtime and test requirements.\r\n",
      "      - run:\r\n",
      "          name: Run assistant evals.\r\n",
      "          command: python -m pytest test_assistant.py\r\n",
      "  run-pre-release-evals:  # This is the name of the job - feel free to change it to better match what you're trying to do!\r\n",
      "    # These next lines define the Docker executor: https://circleci.com/docs/2.0/executor-types/\r\n",
      "    # You can specify an image from Docker Hub or use one of the convenience images from CircleCI's Developer Hub\r\n",
      "    # A list of available CircleCI Docker convenience images is available here: https://circleci.com/developer/images/image/cimg/python\r\n",
      "    # The executor is the environment in which the steps below will be executed - below will use a Python 3.10 container\r\n",
      "    # Change the version below to your required version of Python\r\n",
      "    docker:\r\n",
      "      - image: cimg/python:3.10.5\r\n",
      "    # Check out the code as the first step. This is a dedicated CircleCI step.\r\n",
      "    # The Python orb's install-packages step will install the dependencies from a Pipfile via Pipenv by default.\r\n",
      "    # Here we're making sure we use just use the system-wide pip. By default it uses the project root's requirements.txt.\r\n",
      "    # Then run your tests!\r\n",
      "    # CircleCI will report the results back to your VCS provider.\r\n",
      "\r\n",
      "    steps:\r\n",
      "      - checkout\r\n",
      "      - python/install-packages:\r\n",
      "          pkg-manager: pip\r\n",
      "          # app-dir: ~/project/package-directory/  # If your requirements.txt isn't in the root directory.\r\n",
      "          # pip-dependency-file: test-requirements.txt  # if you have a different name for your requirements file, maybe one that combines your runtime and test requirements.\r\n",
      "      - run:\r\n",
      "          name: Run release evals.\r\n",
      "          command: python -m pytest test_release_evals.py"
     ]
    }
   ],
   "source": [
    "!cat circle_config_v3.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35b97d98-9649-4c8e-abc8-0f3a997b900e",
   "metadata": {},
   "source": [
    "You can trigger the workflow again with the code below. By default the two jobs will run in parallel. Later we will show how you can adjust the workflow to make the jobs run in series or based on certain conditions.\n",
    "\n",
    "**We have intentionally inserted a bug into `test_release_evals.py` so you can see what the output looks like when one of your tests fails.** You can see the output from the error by clicking on the failed job and scrolling down to the step that generated the error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a62bb795-70c0-46c7-bbd1-34025e62218c",
   "metadata": {
    "height": 115
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "uploading test_release_evals.py\n"
     ]
    }
   ],
   "source": [
    "from utils import push_files\n",
    "push_files(course_repo, \n",
    "           course_branch, \n",
    "           [\"app.py\", \"test_assistant.py\", \"test_release_evals.py\"],\n",
    "           config=\"circle_config_v3.yml\"\n",
    "          )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b9bbf5f-8ad1-4685-a993-48abb99f182f",
   "metadata": {
    "height": 81
   },
   "outputs": [],
   "source": [
    "from utils import trigger_commit_evals\n",
    "trigger_commit_evals(course_repo, \n",
    "                     course_branch, \n",
    "                     cci_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97b4a9b2-b3f8-419a-a25e-36a58d20f3d0",
   "metadata": {},
   "source": [
    "## Enhancing your CircleCI configuration file\n",
    "\n",
    "Jobs, commands, and workflows are key components of every CircleCI pipeline. However, there are many additional features within CircleCI that you can take advantage of to better orchestrate the flow of jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80e5ac33-a967-4805-a064-57feda76b0af",
   "metadata": {},
   "source": [
    "### Conditional workflows\n",
    "\n",
    "You can also execute different workflows on different conditions with [conditional workflows](https://circleci.com/docs/pipeline-variables/#conditional-workflows). Conditional workflows allow you to use if-statement logic in your CircleCI configuration.\n",
    "\n",
    "For example, you might want to run the pre-commit evals whenever there is a push to your dev branches and the pre-release evals when there is a push to your main branch. In this configuration, we’ll show you how to conditionally execute different workflows by passing in [pipeline parameters](https://circleci.com/docs/pipeline-variables/#pipeline-parameters-in-configuration) to our workflows. However, you could also execute different workflows based on [pipeline values](https://circleci.com/docs/pipeline-variables/#pipeline-values) like `pipeline.git.branch`.\n",
    "\n",
    "With pipeline parameters, you can define parameters for your configuration and change them in your application through [CircleCI’s API](https://circleci.com/docs/pipeline-variables/#passing-parameters-when-triggering-pipelines-via-the-api). The `utils.py` file in this example is used to pass in different `eval-mode` parameters based on this.\n",
    "\n",
    "The CircleCI configuration snippet below runs different workflows based on the current `eval-mode`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a813e42c-dd9f-44f4-85cf-9dbb60b89ea6",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "!cat circle_config_v4.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be940c60-c017-4e70-81a0-fd26b4d1bb59",
   "metadata": {},
   "source": [
    "This conditional workflow can be triggered with the commands below. We expect the `commit-workflow` to pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83cf00cd-ef8a-4c66-af8b-a3592c4c6b76",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "from utils import push_files\n",
    "push_files(course_repo, \n",
    "           course_branch, \n",
    "           [\"app.py\", \"test_assistant.py\", \"test_release_evals.py\"],\n",
    "           config=\"circle_config_v4.yml\"\n",
    "          )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07ccd2a4",
   "metadata": {
    "height": 81
   },
   "outputs": [],
   "source": [
    "from utils import trigger_commit_evals\n",
    "trigger_commit_evals(course_repo, \n",
    "                     course_branch, \n",
    "                     cci_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34cd94bd",
   "metadata": {},
   "source": [
    "For this optional lesson, we have implemented placeholder commands for pre-release and manual evals. We are including them to show an example of how to trigger different conditional behavior in workflows using the API.\n",
    "\n",
    "To learn how to implement model graded evals and run them directly as part of the CI pipeline, please visit lesson 3 of this course. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63496c74",
   "metadata": {
    "height": 81
   },
   "outputs": [],
   "source": [
    "from utils import trigger_release_evals\n",
    "trigger_release_evals(course_repo, \n",
    "                      course_branch, \n",
    "                      cci_api_key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "766dc40e",
   "metadata": {
    "height": 81
   },
   "outputs": [],
   "source": [
    "from utils import trigger_full_evals\n",
    "trigger_full_evals(course_repo, \n",
    "                   course_branch, \n",
    "                   cci_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf739f0c-b5f6-4f26-9dbf-7c6141762fa2",
   "metadata": {},
   "source": [
    "### Scheduled workflows\n",
    "\n",
    "So far, all of these workflows have been triggered whenever a commit is made to a Git repository, which is typical for continuous integration. However, you might want to schedule your more comprehensive evals to run on a regular schedule for continuous delivery or deployment. This can be done with [scheduled workflows](https://circleci.com/docs/workflows/#scheduling-a-workflow).\n",
    "\n",
    "For example, you could set up a nightly trigger to run a certain workflow by providing a standard `cron` syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0638208-7179-49c8-8598-8788a62c898f",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "!cat circle_config_v5.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd157cfd-0d51-42ff-9a26-87d5fedb6cfe",
   "metadata": {},
   "source": [
    "## Some other features we've used\n",
    "\n",
    "Throughout this tutorial, we have used other features of CircleCI as well, although we didn't focus on them as much. Despite this, each of the following features are key to building a functional CI/CD pipeline.\n",
    "\n",
    "### Execution environments\n",
    "\n",
    "CircleCI provides many different options for [execution environments](https://circleci.com/docs/executor-intro/), including Docker images, Linux VMs, MacOS VMs, Windows VMs, GPU executors, and even [self-hosted runners](https://circleci.com/docs/runner-overview/) to run jobs on your own infrastructure.\n",
    "\n",
    "You can also run different jobs in the same workflow **on different machines**. Some example use cases for this would be using more expensive and specialized cloud-based GPU executors to train ML models or deploying applications to your own infrastructure after testing them on cloud infrastructure.\n",
    "\n",
    "### Orbs\n",
    "\n",
    "[Orbs](https://circleci.com/developer/orbs) are shareable packages of CircleCI configurations, similar to libraries or packages in conventional software development. CircleCI supports certified orbs, community orbs, and private orbs to bring the same advantages of libraries for product development to configuration development.\n",
    "\n",
    "### Contexts\n",
    "\n",
    "Security is paramount for any application. You shouldn't include credentials and other sensitive information (known as secrets) in your CircleCI configuration file, as it will be committed to your code repositories where it can potentially be exposed.\n",
    "\n",
    "[Contexts](https://circleci.com/docs/contexts/#create-and-use-a-context) allow you to securely store certain credentials in one centralized location on CircleCI's infrastructure for access during certain workflows.\n",
    "\n",
    "For this tutorial, you can see that we've been using the `dl-ai-courses` context, which contains various API keys for LangSmith, OpenAI, and other tools required for this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23bd46dc-fe43-4a27-bce8-3e914b9a7096",
   "metadata": {},
   "source": [
    "## Building out your AI application and workflows\n",
    "\n",
    "We've covered the major features of CircleCI that you can use to build this pipeline and similar ones as well. However, we've only scratched the surface. You can check out [CircleCI's documentation](https://circleci.com/docs/configuration-reference/) to see more advanced features and use cases.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5edb613d-647e-460f-b835-d2a64feae84b",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4e862ca",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34a0a0db",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}