CapybaraHermes_2_5_Mistral_7B_GGUF.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Troyanovsky/Local-LLM-Comparison-Colab-UI/blob/main/CapybaraHermes_2_5_Mistral_7B_GGUF.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XNkpBtLvuTOp"
      },
      "source": [
        "## CapybaraHermes-2.5-Mistral-7B-GGUF WebUI\n",
        "\n",
        "1. Run the following cell, takes ~5 min\n",
        "(You may need to confirm to proceed by typing \"Y\")\n",
        "2. Pick the version you need from one of the last two cells and only run that cell.\n",
        "3. Click the gradio link at the bottom\n",
        "4. In Chat settings - Instruction Template: ChatML\n",
        "```\n",
        "<|im_start|>system\n",
        "{system_message}<|im_end|>\n",
        "<|im_start|>user\n",
        "{prompt}<|im_end|>\n",
        "<|im_start|>assistant\n",
        "```\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true,
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "D-MiHp_bveP6",
        "outputId": "1aa189ae-9409-45c4-9bf3-391541405689"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n",
            "Selecting previously unselected package libc-ares2:amd64.\n",
            "(Reading database ... 121747 files and directories currently installed.)\n",
            "Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.2_amd64.deb ...\n",
            "Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.2) ...\n",
            "Selecting previously unselected package libaria2-0:amd64.\n",
            "Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ...\n",
            "Unpacking libaria2-0:amd64 (1.36.0-1) ...\n",
            "Selecting previously unselected package aria2.\n",
            "Preparing to unpack .../aria2_1.36.0-1_amd64.deb ...\n",
            "Unpacking aria2 (1.36.0-1) ...\n",
            "Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.2) ...\n",
            "Setting up libaria2-0:amd64 (1.36.0-1) ...\n",
            "Setting up aria2 (1.36.0-1) ...\n",
            "Processing triggers for man-db (2.10.2-1) ...\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.4) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link\n",
            "\n",
            "Cloning into 'text-generation-webui'...\n",
            "remote: Enumerating objects: 16095, done.\u001b[K\n",
            "remote: Counting objects: 100% (3083/3083), done.\u001b[K\n",
            "remote: Compressing objects: 100% (193/193), done.\u001b[K\n",
            "remote: Total 16095 (delta 2974), reused 2890 (delta 2890), pack-reused 13012\u001b[K\n",
            "Receiving objects: 100% (16095/16095), 26.13 MiB | 10.93 MiB/s, done.\n",
            "Resolving deltas: 100% (11276/11276), done.\n",
            "Note: switching to '4f3fdf1b5ff6884b9899a3630b3ed9aae27decbf'.\n",
            "\n",
            "You are in 'detached HEAD' state. You can look around, make experimental\n",
            "changes and commit them, and you can discard any commits you make in this\n",
            "state without impacting any branches by switching back to a branch.\n",
            "\n",
            "If you want to create a new branch to retain commits you create, you may\n",
            "do so (now or later) by using -c with the switch command. Example:\n",
            "\n",
            "  git switch -c <new-branch-name>\n",
            "\n",
            "Or undo this operation with:\n",
            "\n",
            "  git switch -\n",
            "\n",
            "Turn off this advice by setting config variable advice.detachedHead to false\n",
            "\n",
            "/content/text-generation-webui\n",
            "Ignoring exllamav2: markers 'platform_system != \"Darwin\" and platform_machine != \"x86_64\"' don't match your environment\n",
            "Ignoring bitsandbytes: markers 'platform_system == \"Windows\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python==0.2.36+cpuavx2 (from -r requirements.txt (line 33))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/cpu/llama_cpp_python-0.2.36+cpuavx2-cp310-cp310-manylinux_2_31_x86_64.whl (2.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.4/2.4 MB\u001b[0m \u001b[31m33.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python-cuda==0.2.36+cu121 (from -r requirements.txt (line 41))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda-0.2.36+cu121-cp310-cp310-manylinux_2_31_x86_64.whl (30.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m30.4/30.4 MB\u001b[0m \u001b[31m33.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring llama-cpp-python-cuda-tensorcores: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting llama-cpp-python-cuda-tensorcores==0.2.36+cu121 (from -r requirements.txt (line 47))\n",
            "  Downloading https://github.com/oobabooga/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda_tensorcores-0.2.36+cu121-cp310-cp310-manylinux_2_31_x86_64.whl (17.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.8/17.8 MB\u001b[0m \u001b[31m39.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring auto-gptq: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring auto-gptq: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring auto-gptq: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting auto-gptq==0.6.0+cu121 (from -r requirements.txt (line 53))\n",
            "  Downloading https://github.com/jllllll/AutoGPTQ/releases/download/v0.6.0/auto_gptq-0.6.0+cu121-cp310-cp310-linux_x86_64.whl (4.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.8/4.8 MB\u001b[0m \u001b[31m98.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring exllamav2: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting exllamav2==0.0.12+cu121 (from -r requirements.txt (line 57))\n",
            "  Downloading https://github.com/turboderp/exllamav2/releases/download/v0.0.12/exllamav2-0.0.12+cu121-cp310-cp310-linux_x86_64.whl (27.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m27.7/27.7 MB\u001b[0m \u001b[31m24.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring flash-attn: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting flash-attn==2.3.4+cu122torch2.1cxx11abiFALSE (from -r requirements.txt (line 61))\n",
            "  Downloading https://github.com/Dao-AILab/flash-attention/releases/download/v2.3.4/flash_attn-2.3.4+cu122torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl (57.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.4/57.4 MB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hIgnoring gptq-for-llama: markers 'platform_system == \"Windows\" and python_version == \"3.11\"' don't match your environment\n",
            "Ignoring gptq-for-llama: markers 'platform_system == \"Windows\" and python_version == \"3.10\"' don't match your environment\n",
            "Ignoring gptq-for-llama: markers 'platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version == \"3.11\"' don't match your environment\n",
            "Collecting gptq-for-llama==0.1.1+cu121 (from -r requirements.txt (line 65))\n",
            "  Downloading https://github.com/jllllll/GPTQ-for-LLaMa-CUDA/releases/download/0.1.1/gptq_for_llama-0.1.1+cu121-cp310-cp310-linux_x86_64.whl (739 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m739.5/739.5 kB\u001b[0m \u001b[31m52.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ctransformers==0.2.27+cu121 (from -r requirements.txt (line 66))\n",
            "  Downloading https://github.com/jllllll/ctransformers-cuBLAS-wheels/releases/download/AVX2/ctransformers-0.2.27+cu121-py3-none-any.whl (15.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.5/15.5 MB\u001b[0m \u001b[31m76.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting accelerate==0.25.* (from -r requirements.txt (line 1))\n",
            "  Downloading accelerate-0.25.0-py3-none-any.whl (265 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m265.7/265.7 kB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting colorama (from -r requirements.txt (line 2))\n",
            "  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
            "Collecting datasets (from -r requirements.txt (line 3))\n",
            "  Downloading datasets-2.16.1-py3-none-any.whl (507 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m507.1/507.1 kB\u001b[0m \u001b[31m43.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting einops (from -r requirements.txt (line 4))\n",
            "  Downloading einops-0.7.0-py3-none-any.whl (44 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.6/44.6 kB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gradio==3.50.* (from -r requirements.txt (line 6))\n",
            "  Downloading gradio-3.50.2-py3-none-any.whl (20.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.3/20.3 MB\u001b[0m \u001b[31m72.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting hqq==0.1.2.post1 (from -r requirements.txt (line 7))\n",
            "  Downloading hqq-0.1.2.post1.tar.gz (26 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting jinja2==3.1.2 (from -r requirements.txt (line 8))\n",
            "  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m133.1/133.1 kB\u001b[0m \u001b[31m19.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting lm_eval==0.3.0 (from -r requirements.txt (line 9))\n",
            "  Downloading lm_eval-0.3.0-py3-none-any.whl (178 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m178.7/178.7 kB\u001b[0m \u001b[31m18.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 10)) (3.5.2)\n",
            "Collecting numpy==1.24.* (from -r requirements.txt (line 11))\n",
            "  Downloading numpy-1.24.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.3/17.3 MB\u001b[0m \u001b[31m70.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting optimum==1.16.* (from -r requirements.txt (line 12))\n",
            "  Downloading optimum-1.16.2-py3-none-any.whl (402 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m402.5/402.5 kB\u001b[0m \u001b[31m39.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 13)) (1.5.3)\n",
            "Collecting peft==0.7.* (from -r requirements.txt (line 14))\n",
            "  Downloading peft-0.7.1-py3-none-any.whl (168 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m168.3/168.3 kB\u001b[0m \u001b[31m22.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting Pillow>=9.5.0 (from -r requirements.txt (line 15))\n",
            "  Downloading pillow-10.2.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.5/4.5 MB\u001b[0m \u001b[31m109.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 16)) (6.0.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 17)) (2.31.0)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 18)) (13.7.0)\n",
            "Collecting safetensors==0.4.1 (from -r requirements.txt (line 19))\n",
            "  Downloading safetensors-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m82.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 20)) (1.11.4)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 21)) (0.1.99)\n",
            "Requirement already satisfied: tensorboard in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 22)) (2.15.1)\n",
            "Collecting transformers==4.37.* (from -r requirements.txt (line 23))\n",
            "  Downloading transformers-4.37.2-py3-none-any.whl (8.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.4/8.4 MB\u001b[0m \u001b[31m84.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 24)) (4.66.1)\n",
            "Collecting wandb (from -r requirements.txt (line 25))\n",
            "  Downloading wandb-0.16.2-py3-none-any.whl (2.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.2/2.2 MB\u001b[0m \u001b[31m95.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting bitsandbytes==0.41.1 (from -r requirements.txt (line 28))\n",
            "  Downloading bitsandbytes-0.41.1-py3-none-any.whl (92.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.6/92.6 MB\u001b[0m \u001b[31m9.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting autoawq==0.1.8 (from -r requirements.txt (line 67))\n",
            "  Downloading autoawq-0.1.8-cp310-cp310-manylinux2014_x86_64.whl (20.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.5/20.5 MB\u001b[0m \u001b[31m71.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.*->-r requirements.txt (line 1)) (23.2)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.*->-r requirements.txt (line 1)) (5.9.5)\n",
            "Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.*->-r requirements.txt (line 1)) (2.1.0+cu121)\n",
            "Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.*->-r requirements.txt (line 1)) (0.20.3)\n",
            "Collecting aiofiles<24.0,>=22.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading aiofiles-23.2.1-py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (4.2.2)\n",
            "Collecting fastapi (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading fastapi-0.109.2-py3-none-any.whl (92 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.1/92.1 kB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ffmpy (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading ffmpy-0.3.1.tar.gz (5.5 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting gradio-client==0.6.1 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading gradio_client-0.6.1-py3-none-any.whl (299 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m299.2/299.2 kB\u001b[0m \u001b[31m30.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpx (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading httpx-0.26.0-py3-none-any.whl (75 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.9/75.9 kB\u001b[0m \u001b[31m10.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (6.1.1)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (2.1.4)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (3.7.1)\n",
            "Collecting orjson~=3.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading orjson-3.9.13-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.7/138.7 kB\u001b[0m \u001b[31m19.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (1.10.14)\n",
            "Collecting pydub (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Collecting python-multipart (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading python_multipart-0.0.7-py3-none-any.whl (22 kB)\n",
            "Collecting semantic-version~=2.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.*->-r requirements.txt (line 6)) (4.5.0)\n",
            "Collecting uvicorn>=0.14.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading uvicorn-0.27.0.post1-py3-none-any.whl (60 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.7/60.7 kB\u001b[0m \u001b[31m10.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting websockets<12.0,>=10.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m17.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting timm (from hqq==0.1.2.post1->-r requirements.txt (line 7))\n",
            "  Downloading timm-0.9.12-py3-none-any.whl (2.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.2/2.2 MB\u001b[0m \u001b[31m91.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from hqq==0.1.2.post1->-r requirements.txt (line 7)) (2.4.0)\n",
            "Collecting jsonlines (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading jsonlines-4.0.0-py3-none-any.whl (8.7 kB)\n",
            "Requirement already satisfied: numexpr in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.3.0->-r requirements.txt (line 9)) (2.9.0)\n",
            "Collecting openai>=0.6.4 (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading openai-1.11.1-py3-none-any.whl (226 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m226.1/226.1 kB\u001b[0m \u001b[31m29.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pybind11>=2.6.2 (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading pybind11-2.11.1-py3-none-any.whl (227 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.7/227.7 kB\u001b[0m \u001b[31m29.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pycountry (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading pycountry-23.12.11-py3-none-any.whl (6.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.2/6.2 MB\u001b[0m \u001b[31m78.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytablewriter (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading pytablewriter-1.2.0-py3-none-any.whl (111 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.1/111.1 kB\u001b[0m \u001b[31m15.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting rouge-score>=0.0.4 (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading rouge_score-0.1.2.tar.gz (17 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting sacrebleu==1.5.0 (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading sacrebleu-1.5.0-py3-none-any.whl (65 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m65.6/65.6 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: scikit-learn>=0.24.1 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.3.0->-r requirements.txt (line 9)) (1.2.2)\n",
            "Collecting sqlitedict (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading sqlitedict-2.1.0.tar.gz (21 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting tqdm-multiprocess (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading tqdm_multiprocess-0.0.11-py3-none-any.whl (9.8 kB)\n",
            "Collecting zstandard (from lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading zstandard-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.4/5.4 MB\u001b[0m \u001b[31m102.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting coloredlogs (from optimum==1.16.*->-r requirements.txt (line 12))\n",
            "  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from optimum==1.16.*->-r requirements.txt (line 12)) (1.12)\n",
            "Requirement already satisfied: transformers[sentencepiece]>=4.26.0 in /usr/local/lib/python3.10/dist-packages (from optimum==1.16.*->-r requirements.txt (line 12)) (4.35.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.37.*->-r requirements.txt (line 23)) (3.13.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.37.*->-r requirements.txt (line 23)) (2023.12.25)\n",
            "Requirement already satisfied: tokenizers<0.19,>=0.14 in /usr/local/lib/python3.10/dist-packages (from transformers==4.37.*->-r requirements.txt (line 23)) (0.15.1)\n",
            "Collecting texttable (from autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading texttable-1.7.0-py2.py3-none-any.whl (10 kB)\n",
            "Requirement already satisfied: toml in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.8->-r requirements.txt (line 67)) (0.10.2)\n",
            "Collecting attributedict (from autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading attributedict-0.3.0-py3-none-any.whl (14 kB)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.8->-r requirements.txt (line 67)) (3.20.3)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.8->-r requirements.txt (line 67)) (0.16.0+cu121)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.10/dist-packages (from autoawq==0.1.8->-r requirements.txt (line 67)) (0.9.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.6.1->gradio==3.50.*->-r requirements.txt (line 6)) (2023.6.0)\n",
            "Collecting portalocker (from sacrebleu==1.5.0->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading portalocker-2.8.2-py3-none-any.whl (17 kB)\n",
            "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (10.0.1)\n",
            "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (0.6)\n",
            "Collecting dill<0.3.8,>=0.3.0 (from datasets->-r requirements.txt (line 3))\n",
            "  Downloading dill-0.3.7-py3-none-any.whl (115 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.3/115.3 kB\u001b[0m \u001b[31m17.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (3.4.1)\n",
            "Collecting multiprocess (from datasets->-r requirements.txt (line 3))\n",
            "  Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m20.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (3.9.3)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 13)) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 13)) (2023.4)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 17)) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 17)) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 17)) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 17)) (2023.11.17)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->-r requirements.txt (line 18)) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->-r requirements.txt (line 18)) (2.16.1)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 22)) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 22)) (1.60.0)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 22)) (2.17.3)\n",
            "Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 22)) (1.2.0)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 22)) (67.7.2)\n",
            "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 22)) (1.16.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 22)) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 22)) (3.0.1)\n",
            "Requirement already satisfied: Click!=8.0.0,>=7.1 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 25)) (8.1.7)\n",
            "Collecting GitPython!=3.1.29,>=1.0.0 (from wandb->-r requirements.txt (line 25))\n",
            "  Downloading GitPython-3.1.41-py3-none-any.whl (196 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m196.4/196.4 kB\u001b[0m \u001b[31m26.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sentry-sdk>=1.0.0 (from wandb->-r requirements.txt (line 25))\n",
            "  Downloading sentry_sdk-1.40.0-py2.py3-none-any.whl (257 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m257.5/257.5 kB\u001b[0m \u001b[31m33.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting docker-pycreds>=0.4.0 (from wandb->-r requirements.txt (line 25))\n",
            "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
            "Collecting setproctitle (from wandb->-r requirements.txt (line 25))\n",
            "  Downloading setproctitle-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n",
            "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 25)) (1.4.4)\n",
            "Requirement already satisfied: diskcache>=5.6.1 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python==0.2.36+cpuavx2->-r requirements.txt (line 33)) (5.6.3)\n",
            "Collecting rouge (from auto-gptq==0.6.0+cu121->-r requirements.txt (line 53))\n",
            "  Downloading rouge-1.0.1-py3-none-any.whl (13 kB)\n",
            "Collecting gekko (from auto-gptq==0.6.0+cu121->-r requirements.txt (line 53))\n",
            "  Downloading gekko-1.0.6-py3-none-any.whl (12.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.2/12.2 MB\u001b[0m \u001b[31m30.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ninja (from exllamav2==0.0.12+cu121->-r requirements.txt (line 57))\n",
            "  Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m307.2/307.2 kB\u001b[0m \u001b[31m32.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting fastparquet (from exllamav2==0.0.12+cu121->-r requirements.txt (line 57))\n",
            "  Downloading fastparquet-2023.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m72.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: py-cpuinfo<10.0.0,>=9.0.0 in /usr/local/lib/python3.10/dist-packages (from ctransformers==0.2.27+cu121->-r requirements.txt (line 66)) (9.0.0)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (4.19.2)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (0.12.1)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (6.0.4)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (1.9.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (4.0.3)\n",
            "Collecting gitdb<5,>=4.0.1 (from GitPython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 25))\n",
            "  Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 22)) (5.3.2)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 22)) (0.3.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 22)) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard->-r requirements.txt (line 22)) (1.3.1)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->-r requirements.txt (line 18)) (0.1.2)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.*->-r requirements.txt (line 6)) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.*->-r requirements.txt (line 6)) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.*->-r requirements.txt (line 6)) (4.47.2)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.*->-r requirements.txt (line 6)) (1.4.5)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.*->-r requirements.txt (line 6)) (3.1.1)\n",
            "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai>=0.6.4->lm_eval==0.3.0->-r requirements.txt (line 9)) (3.7.1)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai>=0.6.4->lm_eval==0.3.0->-r requirements.txt (line 9)) (1.7.0)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai>=0.6.4->lm_eval==0.3.0->-r requirements.txt (line 9)) (1.3.0)\n",
            "Collecting typing-extensions~=4.0 (from gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading typing_extensions-4.9.0-py3-none-any.whl (32 kB)\n",
            "Collecting httpcore==1.* (from httpx->gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m12.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting h11<0.15,>=0.13 (from httpcore==1.*->httpx->gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm_eval==0.3.0->-r requirements.txt (line 9)) (3.8.1)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.3.0->-r requirements.txt (line 9)) (1.3.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.3.0->-r requirements.txt (line 9)) (3.2.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.*->-r requirements.txt (line 1)) (3.2.1)\n",
            "Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.*->-r requirements.txt (line 1)) (2.1.0)\n",
            "INFO: pip is looking at multiple versions of transformers[sentencepiece] to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting rootpath>=0.1.0 (from attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading rootpath-0.1.1-py3-none-any.whl (15 kB)\n",
            "Collecting inspecta>=0.1.0 (from attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading inspecta-0.1.3-py3-none-any.whl (9.2 kB)\n",
            "Collecting colour-runner>=0.0.5 (from attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading colour_runner-0.1.1-py2.py3-none-any.whl (3.7 kB)\n",
            "Collecting deepdiff>=3.3.0 (from attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading deepdiff-6.7.1-py3-none-any.whl (76 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.6/76.6 kB\u001b[0m \u001b[31m11.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tox>=3.0.0 (from attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading tox-4.12.1-py3-none-any.whl (154 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m154.4/154.4 kB\u001b[0m \u001b[31m22.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting coverage>=4.5.2 (from attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading coverage-7.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (233 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m234.0/234.0 kB\u001b[0m \u001b[31m24.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting codecov>=2.0.15 (from attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading codecov-2.1.13-py2.py3-none-any.whl (16 kB)\n",
            "Collecting humanfriendly>=9.1 (from coloredlogs->optimum==1.16.*->-r requirements.txt (line 12))\n",
            "  Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m12.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting starlette<0.37.0,>=0.36.3 (from fastapi->gradio==3.50.*->-r requirements.txt (line 6))\n",
            "  Downloading starlette-0.36.3-py3-none-any.whl (71 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m71.5/71.5 kB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting cramjam>=2.3 (from fastparquet->exllamav2==0.0.12+cu121->-r requirements.txt (line 57))\n",
            "  Downloading cramjam-2.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m35.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hINFO: pip is looking at multiple versions of multiprocess to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting multiprocess (from datasets->-r requirements.txt (line 3))\n",
            "  Downloading multiprocess-0.70.15-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m16.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting DataProperty<2,>=1.0.1 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading DataProperty-1.0.1-py3-none-any.whl (27 kB)\n",
            "Collecting mbstrdecoder<2,>=1.0.0 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading mbstrdecoder-1.1.3-py3-none-any.whl (7.8 kB)\n",
            "Collecting pathvalidate<4,>=2.3.0 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading pathvalidate-3.2.0-py3-none-any.whl (23 kB)\n",
            "Collecting tabledata<2,>=1.3.1 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading tabledata-1.3.3-py3-none-any.whl (11 kB)\n",
            "Collecting tcolorpy<1,>=0.0.5 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading tcolorpy-0.1.4-py3-none-any.whl (7.9 kB)\n",
            "Collecting typepy[datetime]<2,>=1.3.2 (from pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9))\n",
            "  Downloading typepy-1.3.2-py3-none-any.whl (31 kB)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->optimum==1.16.*->-r requirements.txt (line 12)) (1.3.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai>=0.6.4->lm_eval==0.3.0->-r requirements.txt (line 9)) (1.2.0)\n",
            "Collecting blessings (from colour-runner>=0.0.5->attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading blessings-1.7-py3-none-any.whl (18 kB)\n",
            "Collecting ordered-set<4.2.0,>=4.0.2 (from deepdiff>=3.3.0->attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading ordered_set-4.1.0-py3-none-any.whl (7.6 kB)\n",
            "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 25))\n",
            "  Downloading smmap-5.0.1-py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (2023.12.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (0.33.0)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.*->-r requirements.txt (line 6)) (0.17.1)\n",
            "Requirement already satisfied: chardet<6,>=3.0.4 in /usr/local/lib/python3.10/dist-packages (from mbstrdecoder<2,>=1.0.0->pytablewriter->lm_eval==0.3.0->-r requirements.txt (line 9)) (5.2.0)\n",
            "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 22)) (0.5.1)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard->-r requirements.txt (line 22)) (3.2.2)\n",
            "Requirement already satisfied: platformdirs>=4.1 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.8->-r requirements.txt (line 67)) (4.2.0)\n",
            "Requirement already satisfied: pluggy>=1.3 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.8->-r requirements.txt (line 67)) (1.4.0)\n",
            "Collecting pyproject-api>=1.6.1 (from tox>=3.0.0->attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading pyproject_api-1.6.1-py3-none-any.whl (12 kB)\n",
            "Requirement already satisfied: tomli>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from tox>=3.0.0->attributedict->autoawq==0.1.8->-r requirements.txt (line 67)) (2.0.1)\n",
            "Collecting virtualenv>=20.25 (from tox>=3.0.0->attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading virtualenv-20.25.0-py3-none-any.whl (3.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.8/3.8 MB\u001b[0m \u001b[31m78.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting distlib<1,>=0.3.7 (from virtualenv>=20.25->tox>=3.0.0->attributedict->autoawq==0.1.8->-r requirements.txt (line 67))\n",
            "  Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m468.9/468.9 kB\u001b[0m \u001b[31m48.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hBuilding wheels for collected packages: hqq, rouge-score, ffmpy, sqlitedict\n",
            "  Building wheel for hqq (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for hqq: filename=hqq-0.1.2.post1-py3-none-any.whl size=33636 sha256=82794db1c4d3b062ff2d653f8b90ece0ffa332e57420cba208419f7d4b33ee85\n",
            "  Stored in directory: /root/.cache/pip/wheels/94/e7/47/a3a4cbc595d16c7197965c01015666ca31aee73cb5ea43b9c0\n",
            "  Building wheel for rouge-score (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for rouge-score: filename=rouge_score-0.1.2-py3-none-any.whl size=24933 sha256=c6d83d1768eca159bb4e1b6b51aed2da1c935e7e5854a78bfa33e8c172720f61\n",
            "  Stored in directory: /root/.cache/pip/wheels/5f/dd/89/461065a73be61a532ff8599a28e9beef17985c9e9c31e541b4\n",
            "  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ffmpy: filename=ffmpy-0.3.1-py3-none-any.whl size=5579 sha256=27205ffe35e31e56b63c44c32c529d664f73fbb8ab87bc8c5fd53c07f0f28ae3\n",
            "  Stored in directory: /root/.cache/pip/wheels/01/a6/d1/1c0828c304a4283b2c1639a09ad86f83d7c487ef34c6b4a1bf\n",
            "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sqlitedict: filename=sqlitedict-2.1.0-py3-none-any.whl size=16863 sha256=c0921c505ab45fdec00b8fcf065610db80e480e2f809920c2e707de291d85288\n",
            "  Stored in directory: /root/.cache/pip/wheels/79/d6/e7/304e0e6cb2221022c26d8161f7c23cd4f259a9e41e8bbcfabd\n",
            "Successfully built hqq rouge-score ffmpy sqlitedict\n",
            "Installing collected packages: texttable, sqlitedict, pydub, ninja, gptq-for-llama, ffmpy, distlib, bitsandbytes, zstandard, websockets, virtualenv, typing-extensions, tcolorpy, smmap, setproctitle, sentry-sdk, semantic-version, safetensors, rouge, python-multipart, pyproject-api, pycountry, pybind11, portalocker, Pillow, pathvalidate, orjson, ordered-set, numpy, mbstrdecoder, jsonlines, jinja2, humanfriendly, h11, einops, docker-pycreds, dill, cramjam, coverage, colorama, blessings, aiofiles, uvicorn, typepy, tqdm-multiprocess, tox, starlette, sacrebleu, rouge-score, multiprocess, llama-cpp-python-cuda-tensorcores, llama-cpp-python-cuda, llama-cpp-python, httpcore, gitdb, gekko, deepdiff, colour-runner, coloredlogs, codecov, rootpath, httpx, GitPython, flash-attn, fastparquet, fastapi, ctransformers, accelerate, wandb, transformers, timm, openai, inspecta, gradio-client, exllamav2, datasets, DataProperty, tabledata, peft, hqq, gradio, attributedict, pytablewriter, optimum, auto-gptq, lm_eval, autoawq\n",
            "  Attempting uninstall: typing-extensions\n",
            "    Found existing installation: typing_extensions 4.5.0\n",
            "    Uninstalling typing_extensions-4.5.0:\n",
            "      Successfully uninstalled typing_extensions-4.5.0\n",
            "  Attempting uninstall: safetensors\n",
            "    Found existing installation: safetensors 0.4.2\n",
            "    Uninstalling safetensors-0.4.2:\n",
            "      Successfully uninstalled safetensors-0.4.2\n",
            "  Attempting uninstall: Pillow\n",
            "    Found existing installation: Pillow 9.4.0\n",
            "    Uninstalling Pillow-9.4.0:\n",
            "      Successfully uninstalled Pillow-9.4.0\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.23.5\n",
            "    Uninstalling numpy-1.23.5:\n",
            "      Successfully uninstalled numpy-1.23.5\n",
            "  Attempting uninstall: jinja2\n",
            "    Found existing installation: Jinja2 3.1.3\n",
            "    Uninstalling Jinja2-3.1.3:\n",
            "      Successfully uninstalled Jinja2-3.1.3\n",
            "  Attempting uninstall: transformers\n",
            "    Found existing installation: transformers 4.35.2\n",
            "    Uninstalling transformers-4.35.2:\n",
            "      Successfully uninstalled transformers-4.35.2\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "lida 0.0.10 requires kaleido, which is not installed.\n",
            "llmx 0.0.15a0 requires cohere, which is not installed.\n",
            "llmx 0.0.15a0 requires tiktoken, which is not installed.\n",
            "imageio 2.31.6 requires pillow<10.1.0,>=8.3.2, but you have pillow 10.2.0 which is incompatible.\n",
            "tensorflow-probability 0.22.0 requires typing-extensions<4.6.0, but you have typing-extensions 4.9.0 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed DataProperty-1.0.1 GitPython-3.1.41 Pillow-10.2.0 accelerate-0.25.0 aiofiles-23.2.1 attributedict-0.3.0 auto-gptq-0.6.0+cu121 autoawq-0.1.8 bitsandbytes-0.41.1 blessings-1.7 codecov-2.1.13 colorama-0.4.6 coloredlogs-15.0.1 colour-runner-0.1.1 coverage-7.4.1 cramjam-2.8.1 ctransformers-0.2.27+cu121 datasets-2.16.1 deepdiff-6.7.1 dill-0.3.7 distlib-0.3.8 docker-pycreds-0.4.0 einops-0.7.0 exllamav2-0.0.12+cu121 fastapi-0.109.2 fastparquet-2023.10.1 ffmpy-0.3.1 flash-attn-2.3.4 gekko-1.0.6 gitdb-4.0.11 gptq-for-llama-0.1.1+cu121 gradio-3.50.2 gradio-client-0.6.1 h11-0.14.0 hqq-0.1.2.post1 httpcore-1.0.2 httpx-0.26.0 humanfriendly-10.0 inspecta-0.1.3 jinja2-3.1.2 jsonlines-4.0.0 llama-cpp-python-0.2.36+cpuavx2 llama-cpp-python-cuda-0.2.36+cu121 llama-cpp-python-cuda-tensorcores-0.2.36+cu121 lm_eval-0.3.0 mbstrdecoder-1.1.3 multiprocess-0.70.15 ninja-1.11.1.1 numpy-1.24.4 openai-1.11.1 optimum-1.16.2 ordered-set-4.1.0 orjson-3.9.13 pathvalidate-3.2.0 peft-0.7.1 portalocker-2.8.2 pybind11-2.11.1 pycountry-23.12.11 pydub-0.25.1 pyproject-api-1.6.1 pytablewriter-1.2.0 python-multipart-0.0.7 rootpath-0.1.1 rouge-1.0.1 rouge-score-0.1.2 sacrebleu-1.5.0 safetensors-0.4.1 semantic-version-2.10.0 sentry-sdk-1.40.0 setproctitle-1.3.3 smmap-5.0.1 sqlitedict-2.1.0 starlette-0.36.3 tabledata-1.3.3 tcolorpy-0.1.4 texttable-1.7.0 timm-0.9.12 tox-4.12.1 tqdm-multiprocess-0.0.11 transformers-4.37.2 typepy-1.3.2 typing-extensions-4.9.0 uvicorn-0.27.0.post1 virtualenv-20.25.0 wandb-0.16.2 websockets-11.0.3 zstandard-0.22.0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "PIL",
                  "numpy"
                ]
              }
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: gradio==3.50.2 in /usr/local/lib/python3.10/dist-packages (3.50.2)\n",
            "Requirement already satisfied: aiofiles<24.0,>=22.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (23.2.1)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (4.2.2)\n",
            "Requirement already satisfied: fastapi in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.109.2)\n",
            "Requirement already satisfied: ffmpy in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.3.1)\n",
            "Requirement already satisfied: gradio-client==0.6.1 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.6.1)\n",
            "Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.26.0)\n",
            "Requirement already satisfied: huggingface-hub>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.20.3)\n",
            "Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (6.1.1)\n",
            "Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (3.1.2)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (2.1.4)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (3.7.1)\n",
            "Requirement already satisfied: numpy~=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (1.24.4)\n",
            "Requirement already satisfied: orjson~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (3.9.13)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (23.2)\n",
            "Requirement already satisfied: pandas<3.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (1.5.3)\n",
            "Requirement already satisfied: pillow<11.0,>=8.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (10.2.0)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (1.10.14)\n",
            "Requirement already satisfied: pydub in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.25.1)\n",
            "Requirement already satisfied: python-multipart in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.0.7)\n",
            "Requirement already satisfied: pyyaml<7.0,>=5.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (6.0.1)\n",
            "Requirement already satisfied: requests~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (2.31.0)\n",
            "Requirement already satisfied: semantic-version~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (2.10.0)\n",
            "Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (4.9.0)\n",
            "Requirement already satisfied: uvicorn>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (0.27.0.post1)\n",
            "Requirement already satisfied: websockets<12.0,>=10.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.50.2) (11.0.3)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.6.1->gradio==3.50.2) (2023.6.0)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.2) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.2) (4.19.2)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.2) (0.12.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.14.0->gradio==3.50.2) (3.13.1)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.14.0->gradio==3.50.2) (4.66.1)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (4.47.2)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (1.4.5)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (3.1.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas<3.0,>=1.0->gradio==3.50.2) (2023.4)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio==3.50.2) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio==3.50.2) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio==3.50.2) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio==3.50.2) (2023.11.17)\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.14.0->gradio==3.50.2) (8.1.7)\n",
            "Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.14.0->gradio==3.50.2) (0.14.0)\n",
            "Requirement already satisfied: starlette<0.37.0,>=0.36.3 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio==3.50.2) (0.36.3)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.50.2) (3.7.1)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.50.2) (1.0.2)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.50.2) (1.3.0)\n",
            "Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.2) (23.2.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.2) (2023.12.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.2) (0.33.0)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.2) (0.17.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio==3.50.2) (1.16.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx->gradio==3.50.2) (1.2.0)\n",
            "Found existing installation: llama_cpp_python 0.2.36+cpuavx2\n",
            "Uninstalling llama_cpp_python-0.2.36+cpuavx2:\n",
            "  Successfully uninstalled llama_cpp_python-0.2.36+cpuavx2\n",
            "Collecting llama-cpp-python\n",
            "  Downloading llama_cpp_python-0.2.38.tar.gz (10.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.7/10.7 MB\u001b[0m \u001b[31m194.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing backend dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (4.9.0)\n",
            "Requirement already satisfied: numpy>=1.20.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (1.24.4)\n",
            "Requirement already satisfied: diskcache>=5.6.1 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (5.6.3)\n",
            "Requirement already satisfied: jinja2>=2.11.3 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (3.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2>=2.11.3->llama-cpp-python) (2.1.4)\n",
            "Building wheels for collected packages: llama-cpp-python\n",
            "  Building wheel for llama-cpp-python (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.2.38-cp310-cp310-manylinux_2_35_x86_64.whl size=9697722 sha256=6d1efcdf334cdaf27645aa0aa39de757a99e2f0b015a2ff4bf0f22512262fbc8\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-hxe0ria0/wheels/eb/58/77/20d3d9a235b4930050fbcde1ad4f0a4d054644269e801b08aa\n",
            "Successfully built llama-cpp-python\n",
            "Installing collected packages: llama-cpp-python\n",
            "Successfully installed llama-cpp-python-0.2.38\n",
            "Found existing installation: flash-attn 2.3.4\n",
            "Uninstalling flash-attn-2.3.4:\n",
            "  Successfully uninstalled flash-attn-2.3.4\n",
            "Collecting flash-attn==2.3.4\n",
            "  Downloading flash_attn-2.3.4.tar.gz (2.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m34.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.3.4) (2.1.0+cu121)\n",
            "Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.3.4) (0.7.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.3.4) (23.2)\n",
            "Requirement already satisfied: ninja in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.3.4) (1.11.1.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (3.13.1)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (4.9.0)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (3.2.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (3.1.2)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (2023.6.0)\n",
            "Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.3.4) (2.1.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->flash-attn==2.3.4) (2.1.4)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->flash-attn==2.3.4) (1.3.0)\n",
            "Building wheels for collected packages: flash-attn\n",
            "  Building wheel for flash-attn (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for flash-attn: filename=flash_attn-2.3.4-cp310-cp310-linux_x86_64.whl size=57390576 sha256=a4f74d7c46a5b6c86d3f7539b07d11a76e7bfe8089cbc1de8ee79f0cf984bb13\n",
            "  Stored in directory: /root/.cache/pip/wheels/32/19/1b/bf8147431c43f8b8d2b2e5d66be1ebf586f2fca96d72203f53\n",
            "Successfully built flash-attn\n",
            "Installing collected packages: flash-attn\n",
            "Successfully installed flash-attn-2.3.4\n",
            "\u001b[0m\n",
            "Download Results:\n",
            "gid   |stat|avg speed  |path/URI\n",
            "======+====+===========+=======================================================\n",
            "d9cf78|\u001b[1;32mOK\u001b[0m  |   171MiB/s|/content/text-generation-webui/models//capybarahermes-2.5-mistral-7b.Q5_K_M.gguf\n",
            "\n",
            "Status Legend:\n",
            "(OK):download completed.\n",
            "/content/text-generation-webui\n",
            "\u001b[2;36m06:20:26-913637\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Starting Text generation web UI                                            \n",
            "\u001b[2;36m06:20:26-999058\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loading capybarahermes-\u001b[1;36m2.5\u001b[0m-mistral-7b.Q5_K_M.gguf                          \n",
            "\u001b[2;36m06:20:27-124464\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m llama.cpp weights detected:                                                \n",
            "\u001b[2;36m                \u001b[0m         models/capybarahermes-\u001b[1;36m2.5\u001b[0m-mistral-7b.Q5_K_M.gguf                           \n",
            "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   yes\n",
            "ggml_init_cublas: CUDA_USE_TENSOR_CORES: no\n",
            "ggml_init_cublas: found 1 CUDA devices:\n",
            "  Device 0: Tesla T4, compute capability 7.5, VMM: yes\n",
            "llama_model_loader: loaded meta data with 24 key-value pairs and 291 tensors from models/capybarahermes-2.5-mistral-7b.Q5_K_M.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
            "llama_model_loader: - kv   1:                               general.name str              = argilla_capybarahermes-2.5-mistral-7b\n",
            "llama_model_loader: - kv   2:                       llama.context_length u32              = 32768\n",
            "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
            "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
            "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
            "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
            "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
            "llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000\n",
            "llama_model_loader: - kv  11:                          general.file_type u32              = 17\n",
            "llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama\n",
            "llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32002]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
            "llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32002]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
            "llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32002]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
            "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
            "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 32000\n",
            "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
            "llama_model_loader: - kv  19:            tokenizer.ggml.padding_token_id u32              = 0\n",
            "llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  22:                    tokenizer.chat_template str              = {% for message in messages %}{{'<|im_...\n",
            "llama_model_loader: - kv  23:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:   65 tensors\n",
            "llama_model_loader: - type q5_K:  193 tensors\n",
            "llama_model_loader: - type q6_K:   33 tensors\n",
            "llm_load_vocab: special tokens definition check successful ( 261/32002 ).\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = llama\n",
            "llm_load_print_meta: vocab type       = SPM\n",
            "llm_load_print_meta: n_vocab          = 32002\n",
            "llm_load_print_meta: n_merges         = 0\n",
            "llm_load_print_meta: n_ctx_train      = 32768\n",
            "llm_load_print_meta: n_embd           = 4096\n",
            "llm_load_print_meta: n_head           = 32\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_layer          = 32\n",
            "llm_load_print_meta: n_rot            = 128\n",
            "llm_load_print_meta: n_embd_head_k    = 128\n",
            "llm_load_print_meta: n_embd_head_v    = 128\n",
            "llm_load_print_meta: n_gqa            = 4\n",
            "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
            "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 14336\n",
            "llm_load_print_meta: n_expert         = 0\n",
            "llm_load_print_meta: n_expert_used    = 0\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 10000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_yarn_orig_ctx  = 32768\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: model type       = 7B\n",
            "llm_load_print_meta: model ftype      = Q5_K - Medium\n",
            "llm_load_print_meta: model params     = 7.24 B\n",
            "llm_load_print_meta: model size       = 4.78 GiB (5.67 BPW) \n",
            "llm_load_print_meta: general.name     = argilla_capybarahermes-2.5-mistral-7b\n",
            "llm_load_print_meta: BOS token        = 1 '<s>'\n",
            "llm_load_print_meta: EOS token        = 32000 '<|im_end|>'\n",
            "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
            "llm_load_print_meta: PAD token        = 0 '<unk>'\n",
            "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
            "llm_load_tensors: ggml ctx size =    0.22 MiB\n",
            "llm_load_tensors: offloading 32 repeating layers to GPU\n",
            "llm_load_tensors: offloading non-repeating layers to GPU\n",
            "llm_load_tensors: offloaded 33/33 layers to GPU\n",
            "llm_load_tensors:        CPU buffer size =    85.94 MiB\n",
            "llm_load_tensors:      CUDA0 buffer size =  4807.06 MiB\n",
            "...................................................................................................\n",
            "llama_new_context_with_model: n_ctx      = 32768\n",
            "llama_new_context_with_model: freq_base  = 10000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_kv_cache_init:      CUDA0 KV buffer size =  4096.00 MiB\n",
            "llama_new_context_with_model: KV self size  = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host input buffer size   =    72.13 MiB\n",
            "llama_new_context_with_model:      CUDA0 compute buffer size =  2349.60 MiB\n",
            "llama_new_context_with_model:  CUDA_Host compute buffer size =     8.80 MiB\n",
            "llama_new_context_with_model: graph splits (measure): 3\n",
            "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
            "Model metadata: {'tokenizer.chat_template': \"{% for message in messages %}{{'<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\\n' }}{% endif %}\", 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.padding_token_id': '0', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '32000', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '32768', 'general.name': 'argilla_capybarahermes-2.5-mistral-7b', 'tokenizer.ggml.add_bos_token': 'true', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '17'}\n",
            "\u001b[2;36m06:20:45-654259\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m LOADER: llama.cpp                                                          \n",
            "\u001b[2;36m06:20:45-656262\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m TRUNCATION LENGTH: \u001b[1;36m32768\u001b[0m                                                   \n",
            "\u001b[2;36m06:20:45-657948\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m INSTRUCTION TEMPLATE: Custom \u001b[1m(\u001b[0mobtained from model metadata\u001b[1m)\u001b[0m                \n",
            "\u001b[2;36m06:20:45-659137\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loaded the model in \u001b[1;36m18.66\u001b[0m seconds.                                         \n",
            "\u001b[2;36m06:20:45-660186\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loading the extension \u001b[32m\"gallery\"\u001b[0m                                            \n",
            "Running on local URL:  http://127.0.0.1:7860\n",
            "Running on public URL: https://89c27278bcae164463.gradio.live\n",
            "\n",
            "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n",
            "\n",
            "llama_print_timings:        load time =     426.09 ms\n",
            "llama_print_timings:      sample time =     154.72 ms /   264 runs   (    0.59 ms per token,  1706.29 tokens per second)\n",
            "llama_print_timings: prompt eval time =     425.83 ms /    33 tokens (   12.90 ms per token,    77.50 tokens per second)\n",
            "llama_print_timings:        eval time =    7576.35 ms /   263 runs   (   28.81 ms per token,    34.71 tokens per second)\n",
            "llama_print_timings:       total time =    9713.58 ms /   296 tokens\n",
            "Output generated in 10.45 seconds (25.16 tokens/s, 263 tokens, context 63, seed 1637599030)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =     426.09 ms\n",
            "llama_print_timings:      sample time =     106.57 ms /   175 runs   (    0.61 ms per token,  1642.16 tokens per second)\n",
            "llama_print_timings: prompt eval time =     309.98 ms /    32 tokens (    9.69 ms per token,   103.23 tokens per second)\n",
            "llama_print_timings:        eval time =    5104.41 ms /   174 runs   (   29.34 ms per token,    34.09 tokens per second)\n",
            "llama_print_timings:       total time =    6830.69 ms /   206 tokens\n",
            "Output generated in 7.40 seconds (23.50 tokens/s, 174 tokens, context 380, seed 1142723284)\n",
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =     426.09 ms\n",
            "llama_print_timings:      sample time =      59.73 ms /    88 runs   (    0.68 ms per token,  1473.40 tokens per second)\n",
            "llama_print_timings: prompt eval time =     303.79 ms /    26 tokens (   11.68 ms per token,    85.59 tokens per second)\n",
            "llama_print_timings:        eval time =    2601.33 ms /    87 runs   (   29.90 ms per token,    33.44 tokens per second)\n",
            "llama_print_timings:       total time =    3882.98 ms /   113 tokens\n",
            "Output generated in 4.67 seconds (18.63 tokens/s, 87 tokens, context 602, seed 1636743163)\n"
          ]
        }
      ],
      "source": [
        "%cd /content\n",
        "!apt-get -y install -qq aria2\n",
        "\n",
        "!git clone -b V20240206 https://github.com/Troyanovsky/text-generation-webui\n",
        "%cd /content/text-generation-webui\n",
        "!pip install -r requirements.txt\n",
        "!pip install -U gradio==3.50.2\n",
        "\n",
        "!pip uninstall -y llama-cpp-python -y\n",
        "!CMAKE_ARGS=\"-DLLAMA_CUDA=on\" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir\n",
        "\n",
        "!pip uninstall flash-attn -y\n",
        "!pip install --no-build-isolation flash-attn==2.3.4\n",
        "\n",
        "!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/TheBloke/CapybaraHermes-2.5-Mistral-7B-GGUF/resolve/main/capybarahermes-2.5-mistral-7b.Q5_K_M.gguf?download=true -d /content/text-generation-webui/models/ -o capybarahermes-2.5-mistral-7b.Q5_K_M.gguf\n",
        "\n",
        "%cd /content/text-generation-webui\n",
        "!python server.py --share --n-gpu-layers 100000 --model capybarahermes-2.5-mistral-7b.Q5_K_M.gguf"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOGmV4o4xDkl6Pv5iCCLuDF",
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}