phase3_1_2.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "YEYU9YCUhGDh"
      },
      "source": [
        "`Adaptive Transformers for Learning Multimodal Representations`\\\n",
        "Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we used pre-trained the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. Then fine-tunned this Model on our dataset in this project. \n",
        "\n",
        "<img src=\"https://drive.google.com/file/d/1_k94nRsBYbe5F1qeirTOYF3wROEtGEP_/view?usp=share_link\" alt=\": The LXMERT model for learning vision-and-language cross-modality representations.\" />"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fLms3tNca5BK"
      },
      "source": [
        "# install and import packages "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "76uP5yUSbDIr"
      },
      "source": [
        "in this part for fast and robust training we extract faces from images\n",
        "ans store `bbox` ,`embedding feature`,`sentences`,`sentiment cls` as json file \n",
        "for trian ,test ,validation \\\n",
        "extraction and storing are in appendix section of this notebook"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vEb44p3X3GAI",
        "outputId": "c8f9ee2f-3645-4189-ac72-8ea985795a2e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive/\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive/')\n",
        "! cp -r /content/drive/MyDrive/Transfomer/* .\n",
        "# ! cp /content/drive/MyDrive/multidataset.zip .\n",
        "# !unzip ./multidataset.zip "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xeClh1f7SEtJ",
        "outputId": "13107cf2-b5de-4848-a85d-92779988291a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Archive:  ./multidataset.zip\n",
            "  inflating: content/drive/MyDrive/train_dataset.json  \n",
            "  inflating: content/drive/MyDrive/test_dataset.json  \n",
            "  inflating: content/drive/MyDrive/validation_dataset.json  \n"
          ]
        }
      ],
      "source": [
        "! cp /content/drive/MyDrive/multidataset.zip .\n",
        "!unzip ./multidataset.zip "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3R51cHYcLp_O"
      },
      "outputs": [],
      "source": [
        "# ! pip install transformers\n",
        "!pip install git+https://github.com/huggingface/transformers\n",
        "!pip install boto3"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xaB71irkwtWr"
      },
      "outputs": [],
      "source": [
        "# download pretrained transfomer\n",
        "!mkdir -p snap/pretrained \n",
        "!wget https://nlp.cs.unc.edu/data/model_LXRT.pth -P snap/pretrained"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g3cuhPLt3GAL"
      },
      "source": [
        "# Download and Load dataset,Loader"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "Vq2JvGNr3GAN"
      },
      "outputs": [],
      "source": [
        "import gdown\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "from torchvision.transforms import ToTensor, Resize\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torchvision.transforms import ToTensor, Resize, Compose\n",
        "from torchvision.models import resnet50, ResNet50_Weights\n",
        "from sklearn.metrics import confusion_matrix\n",
        "from sklearn.metrics import classification_report\n",
        "import seaborn as sb\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import PIL \n",
        "import os\n",
        "import torch.utils.data as data\n",
        "import json\n",
        "from models.lxmert_adaptive import VQAModel_Adaptive\n",
        "from learner import Learner\n",
        "# gdown.download(id=\"16TKyvmy3JQfcJPo7oHHD4q8VS-ybHrRs\" ,output=\"multidataset.zip\", quiet=False, fuzzy=True, use_cookies=False)\n",
        "# gdown.download(id=\"1-7lKSz6bDLcBpBAYn7SIJzYgOLogAROW\" ,output=\"validation_dataset.json\", quiet=False, fuzzy=True, use_cookies=False)\n",
        "# gdown.download(id=\"1-7wBuOYhgQG5rR9BtW1nzxG_qUO9vf2q\" ,output=\"test_dataset.json\", quiet=False, fuzzy=True, use_cookies=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "-R2gY94XGasN"
      },
      "outputs": [],
      "source": [
        "class Transfomer_Dataset(data.Dataset):\n",
        "    def __init__(self,root_dir='.',mode='train') -> None:\n",
        "        super(Transfomer_Dataset,self).__init__()\n",
        "        f = open(os.path.join(root_dir,mode+'_dataset.json'))\n",
        "        self.Dataset = json.load(f)\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "\n",
        "        data=self.Dataset[str(index)]\n",
        "        \n",
        "        # Normalize the boxes (to 0 ~ 1)\n",
        "        img_h,img_w=data['size'][0:2]\n",
        "        boxes = np.array(data['pos'])\n",
        "        boxes = boxes.copy()\n",
        "        boxes[:, (0, 2)] /= img_w\n",
        "        boxes[:, (1, 3)] /= img_h\n",
        "        \n",
        "        feats = np.array(data['embedding']).copy()\n",
        "        dialog = data['text']\n",
        "        sentiment = data['sentiment_cls']\n",
        "        y_true=np.zeros(shape=(3,))\n",
        "        y_true[int(sentiment[0])]=1.0\n",
        "\n",
        "        if len(boxes) == 1:\n",
        "          pad_feats=np.repeat(feats,4,axis=0)\n",
        "          pad_boxes=np.repeat(boxes,4,axis =0)\n",
        "          return pad_boxes,pad_feats,dialog[0],y_true\n",
        "        if len(boxes) == 2:\n",
        "          pad_feats=np.repeat(feats,2,axis=0)\n",
        "          pad_boxes=np.repeat(boxes,2,axis =0)\n",
        "          return pad_boxes,pad_feats,dialog[0],y_true\n",
        "        \n",
        "        if len(boxes) == 3:\n",
        "          pad_feats=np.concatenate((feats[0:1],feats),axis=0)\n",
        "          pad_boxes=np.concatenate((boxes[0:1],boxes),axis =0)\n",
        "          return pad_boxes,pad_feats,dialog[0],y_true\n",
        "        else :\n",
        "          return boxes,feats,dialog[0],y_true\n",
        "\n",
        "       \n",
        "    def __len__(self):\n",
        "        return len(self.Dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "VzufcF8p3GAO"
      },
      "outputs": [],
      "source": [
        "# from dataset.Dataset_transfomer import Transfomer_Dataset\n",
        "\n",
        "\n",
        "ValidationDataset=Transfomer_Dataset(root_dir='content/drive/MyDrive/',mode='validation')\n",
        "TrainDataset=Transfomer_Dataset(root_dir='content/drive/MyDrive/',mode='train')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "aPmmhI4iHzBA"
      },
      "outputs": [],
      "source": [
        "Train_loader = DataLoader(TrainDataset, shuffle=True, batch_size=128)\n",
        "Valid_loader =  DataLoader(ValidationDataset, shuffle=True, batch_size=128)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xDPXUmKpH9Sc"
      },
      "outputs": [],
      "source": [
        "next(iter(Valid_loader))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ciwVYZ5GTIhX"
      },
      "source": [
        "# Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2CKJPNayTX_N"
      },
      "source": [
        "## preparation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SJSjw_1uPtUD",
        "outputId": "c0f905c7-8a14-4ae6-f2ed-35b84c473c25"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "LXRT encoder with 9 l_layers, 5 x_layers, and 5 r_layers.\n",
            "Using Adaptive Variant\n"
          ]
        }
      ],
      "source": [
        "# config parameter for training\n",
        "params = {\n",
        "    \"adapt_span_enabled\": True,\n",
        "    \"attn_span\": 1024,\n",
        "    \"adapt_span_loss_coeff\": 0.000005,\n",
        "    \"adapt_span_ramp\": 32,\n",
        "    \"adapt_span_init\": 0.002,\n",
        "    \"adapt_span_cache\": True,\n",
        "    \"nb_heads\": 12,\n",
        "    \"bs\": 1,\n",
        "    \"mask_size\": [20, 36],\n",
        "    \"sparse_enabled\": False,\n",
        "    \"num_attention_heads\": 4,\n",
        "    \"layer_sizes\": {\"lang\": 9, \"cross\": 5, \"vision\": 5},\n",
        "    \"from_scratch\": False,\n",
        "    \"layerdrop_enabled\": False,\n",
        "    \"layerdrop_num_layers\": 1,\n",
        "}\n",
        "model=VQAModel_Adaptive(3,params)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7mIrO9tFQuX7",
        "outputId": "bc9af920-6309-48be-d947-71c5aac13cf3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Load QA pre-trained LXMERT from /content/snap/pretrained/model \n",
            "Loaded 0 answers from LXRTQA pre-training and 3 not\n",
            "\n",
            "Keys don't exactly match\n"
          ]
        }
      ],
      "source": [
        "data_tuple_dict = {\n",
        "    \"train\": Train_loader,\n",
        "    \"validation\": Valid_loader,\n",
        "    \"test\": None,\n",
        "}\n",
        "config = {\n",
        "    \"adaptive_enable\": True,\n",
        "    \"sparse_enable\": False,\n",
        "    \"measure_flops\": False,\n",
        "    \"load_model\": True,\n",
        "}\n",
        "learn = Learner(model, data_tuple_dict, config)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wkH76UW5TSHN"
      },
      "source": [
        "## Training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kwN2vTjrR5YK",
        "outputId": "98b44536-b5a9-4894-8da8-cc7292471805"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\r  0%|          | 0/147 [00:00<?, ?it/s]/content/optimizers/lamb.py:88: UserWarning: This overload of add_ is deprecated:\n",
            "\tadd_(Number alpha, Tensor other)\n",
            "Consider using one of the following signatures instead:\n",
            "\tadd_(Tensor other, *, Number alpha) (Triggered internally at ../torch/csrc/utils/python_arg_parser.cpp:1420.)\n",
            "  exp_avg.mul_(beta1).add_(1 - beta1, grad)\n",
            "100%|██████████| 147/147 [02:10<00:00,  1.12it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0: Loss 1.8198037861845089 Accuracy : 49.25508623911999\n",
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 37/37 [00:09<00:00,  4.00it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "143.23917365074158\n",
            "Epoch 0: Valid 60.12\n",
            "Epoch 0: Best 60.12\n",
            "Time elpased for epoch 143.239174\n",
            "Epoch 0: Valid 60.12\n",
            "Epoch 0: Best 60.12\n",
            "Time elpased for epoch 143.239174\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 147/147 [02:09<00:00,  1.13it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1: Loss 1.521507061650669 Accuracy : 64.9543439953009\n",
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 37/37 [00:09<00:00,  4.10it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "142.40326714515686\n",
            "Epoch 1: Valid 61.14\n",
            "Epoch 1: Best 61.14\n",
            "Time elpased for epoch 142.403267\n",
            "Epoch 1: Valid 61.14\n",
            "Epoch 1: Best 61.14\n",
            "Time elpased for epoch 142.403267\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 147/147 [02:09<00:00,  1.14it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 2: Loss 1.3843761036987718 Accuracy : 69.0446948256528\n",
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 37/37 [00:09<00:00,  3.94it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "141.6281509399414\n",
            "Epoch 2: Valid 61.31\n",
            "Epoch 2: Best 61.31\n",
            "Time elpased for epoch 141.628151\n",
            "Epoch 2: Valid 61.31\n",
            "Epoch 2: Best 61.31\n",
            "Time elpased for epoch 141.628151\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 147/147 [02:12<00:00,  1.11it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 3: Loss 1.2928924459804676 Accuracy : 71.74667592246489\n",
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 37/37 [00:09<00:00,  4.07it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "141.2658772468567\n",
            "Epoch 3: Valid 60.88\n",
            "Epoch 3: Best 61.31\n",
            "Time elpased for epoch 141.265877\n",
            "Epoch 3: Valid 60.88\n",
            "Epoch 3: Best 61.31\n",
            "Time elpased for epoch 141.265877\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 147/147 [02:08<00:00,  1.15it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 4: Loss 1.1985344674253962 Accuracy : 74.78506968548086\n",
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 37/37 [00:09<00:00,  3.96it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "137.46756553649902\n",
            "Epoch 4: Valid 60.46\n",
            "Epoch 4: Best 61.31\n",
            "Time elpased for epoch 137.467566\n",
            "Epoch 4: Valid 60.46\n",
            "Epoch 4: Best 61.31\n",
            "Time elpased for epoch 137.467566\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 147/147 [02:11<00:00,  1.12it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 5: Loss 1.1058701280680667 Accuracy : 77.56180915256047\n",
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 37/37 [00:09<00:00,  3.88it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "140.78752446174622\n",
            "Epoch 5: Valid 61.22\n",
            "Epoch 5: Best 61.31\n",
            "Time elpased for epoch 140.787524\n",
            "Epoch 5: Valid 61.22\n",
            "Epoch 5: Best 61.31\n",
            "Time elpased for epoch 140.787524\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 147/147 [02:08<00:00,  1.15it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 6: Loss 1.0190482680436734 Accuracy : 80.41330698990762\n",
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 37/37 [00:09<00:00,  3.75it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "138.0096080303192\n",
            "Epoch 6: Valid 60.57\n",
            "Epoch 6: Best 61.31\n",
            "Time elpased for epoch 138.009608\n",
            "Epoch 6: Valid 60.57\n",
            "Epoch 6: Best 61.31\n",
            "Time elpased for epoch 138.009608\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 147/147 [02:08<00:00,  1.14it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 7: Loss 0.9319076577162487 Accuracy : 82.56528007689433\n",
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 37/37 [00:08<00:00,  4.17it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "137.44195890426636\n",
            "Epoch 7: Valid 60.19\n",
            "Epoch 7: Best 61.31\n",
            "Time elpased for epoch 137.441959\n",
            "Epoch 7: Valid 60.19\n",
            "Epoch 7: Best 61.31\n",
            "Time elpased for epoch 137.441959\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 147/147 [02:07<00:00,  1.16it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 8: Loss 0.8473401736658895 Accuracy : 84.73861269824317\n",
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 37/37 [00:09<00:00,  3.97it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "136.57204627990723\n",
            "Epoch 8: Valid 58.73\n",
            "Epoch 8: Best 61.31\n",
            "Time elpased for epoch 136.572046\n",
            "Epoch 8: Valid 58.73\n",
            "Epoch 8: Best 61.31\n",
            "Time elpased for epoch 136.572046\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 147/147 [02:07<00:00,  1.15it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 9: Loss 0.7731219899180723 Accuracy : 86.44737544721525\n",
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 37/37 [00:09<00:00,  4.09it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "136.6813678741455\n",
            "Epoch 9: Valid 59.38\n",
            "Epoch 9: Best 61.31\n",
            "Time elpased for epoch 136.681368\n",
            "Epoch 9: Valid 59.38\n",
            "Epoch 9: Best 61.31\n",
            "Time elpased for epoch 136.681368\n",
            "\n"
          ]
        }
      ],
      "source": [
        "learn.train(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jw6k5wnVgNVz"
      },
      "source": [
        "# Test model on test dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "81qUGHfKnw_3"
      },
      "outputs": [],
      "source": [
        "testDataset=Transfomer_Dataset(root_dir='content/drive/MyDrive/',mode='test')\n",
        "test_loader =  DataLoader(testDataset, shuffle=True, batch_size=128)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OFLgdydfq6Df",
        "outputId": "82746d07-5a0f-434d-9daa-a21fbb8ba6eb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "LXRT encoder with 9 l_layers, 5 x_layers, and 5 r_layers.\n",
            "Using Adaptive Variant\n"
          ]
        }
      ],
      "source": [
        "# config parameter for training\n",
        "params = {\n",
        "    \"adapt_span_enabled\": True,\n",
        "    \"attn_span\": 1024,\n",
        "    \"adapt_span_loss_coeff\": 0.000005,\n",
        "    \"adapt_span_ramp\": 32,\n",
        "    \"adapt_span_init\": 0.002,\n",
        "    \"adapt_span_cache\": True,\n",
        "    \"nb_heads\": 12,\n",
        "    \"bs\": 1,\n",
        "    \"mask_size\": [20, 36],\n",
        "    \"sparse_enabled\": False,\n",
        "    \"num_attention_heads\": 4,\n",
        "    \"layer_sizes\": {\"lang\": 9, \"cross\": 5, \"vision\": 5},\n",
        "    \"from_scratch\": False,\n",
        "    \"layerdrop_enabled\": False,\n",
        "    \"layerdrop_num_layers\": 1,\n",
        "}\n",
        "model=VQAModel_Adaptive(3,params)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IIXWAdtAoFt6",
        "outputId": "18692480-e2f6-4705-a4b7-fbf10bdca7cb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "loading best model\n"
          ]
        }
      ],
      "source": [
        "data_tuple_dict = {\n",
        "    \"train\": None,\n",
        "    \"validation\": None,\n",
        "    \"test\": test_loader,\n",
        "}\n",
        "config = {\n",
        "    \"adaptive_enable\": True,\n",
        "    \"sparse_enable\": False,\n",
        "    \"measure_flops\": False,\n",
        "    \"load_model\": False,\n",
        "    \"load_best\" : True\n",
        "}\n",
        "learn = Learner(model, data_tuple_dict, config)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 334
        },
        "id": "GHzLtMsHrOjB",
        "outputId": "625705e4-f294-4c12-d51e-e7a3069a819a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Predict in progress\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 38/38 [00:08<00:00,  4.35it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "test acc is 61.586508432229856%\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEGCAYAAACaSwWnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dZ5QVVfb38e+vaUCCkmUQUBBQBh0DooBpVIwYMEcUHUYG8xjG/IiOCUcx/Y0oKEbMGQNiAhVRUckqioAIknPu3s+LOg0X6HC7+1bf25f9WatWV51Kp2rB7updp86RmeGccy475KS7As4551LHg7pzzmURD+rOOZdFPKg751wW8aDunHNZJDfdFShK2z73eLOcmLV4Y0G6q5D18sZMTHcVNgtD819SeY+RP2uHpGNOzl9+Kvf54uJP6s45l0Uy9kndOecqUj75SW+byU/DHtSdcw5YY3lJb5vJgTOT6+accxWmNE/qmcyDunPOAXlZ0mWKB3XnnAPy8aDunHNZI8+DunPOZQ9/UnfOuSyyxnPqzjmXPbIl/ZLJbeidc67C5FnyU0kkDZQ0W9K4QtZdLskkNQzLknS/pMmSxkhqn7BtD0k/h6lHMtfhQd0554D8UkxJeBI4fONCSc2BQ4FpCcVHAG3C1At4OGxbH+gDdAT2AvpIqlfSiT2oO+cckIeSnkpiZp8B8wtZdQ9wJWyQ6+kGPGWRkUBdSU2Aw4ChZjbfzBYAQynkF8XGPKfunHPAGou340VJ3YAZZvaDtMG5mgLTE5Z/D2VFlRfLg7pzzkFST+AFJPUiSpUU6G9m/YvZviZwLVHqJVYe1J1zDsgvxZN6COBFBvFCtAJaAgVP6c2A0ZL2AmYAzRO2bRbKZgAHbFT+SUkn8py6c86R2pz6xsxsrJltbWYtzKwFUSqlvZnNAt4EzgqtYDoBi8xsJvA+cKikeuEF6aGhrFj+pO6cc0BeCp9xJT1P9JTdUNLvQB8zG1DE5kOArsBkYDlwDoCZzZd0M/B12O6/ZlbYy9cNeFB3zjlKl34piZmdVsL6FgnzBlxQxHYDgYGlObcHdeecA1ZblXRXISU8qDvnHJCfJa8YYwnqkpZAoR0piOivja3iOK9zzpVVWV6AZqJYgrqZbRnHcZ1zLi555k/qSZO0NbBFwbKZTStmc+ecq3D5/qReMknHAP2AbYDZwHbARGCnOM/rnHOltdqy4xVj3H9v3Ax0An4ys5ZAF2BkzOd0zrlSyycn6SmTxV27NWY2D8iRlGNmHwMdYj6nc86VWp4p6SmTxf33xkJJtYHPgGclzQaWxXxO55wrtVR+UZpOcQf1bsAK4FLgDKAO8N+Yz+mcc6WW761fiiepCvC2mR1INFjIoLjO5Zxz5eVP6iUwszxJ+ZLqmNmiuM7jnHOpsMa7CUjKUmCspKEk5NLN7OKYz5tyLRvU4+6Tuq5bbl6vDvd//CV/Ll7KhQd2plXD+pz82POM++NPAKpWyeGmow9m520ak2/Gbe9+wqjffk9X9SuFqtVyuWvgP6haNZcquTkM/3A8zzz8MZf26Uabdk2R4Pep8+h3w2usXLGaQ47ZjZ7/Pox5cxYD8Nbgr3jvtdFpvorMd/mA8+h45B4snL2IXrtcDsCZfU6i6z8PZlG4lwOve45R7363bp9GzRsyYPw9PHXTi7zc76201Dtu/vFRcl4NU6IkxuLOPFPmLeC4R54FIEfi08vP5cOJk9mialUuHvwWNx3dZYPtT9rjbwAc89DT1K9Vg8e6H8eJ/Z/DKuXVV4w1q9dy1blPsnLFaqrk5tDviX/yzYifefSu91i+bBUAvS4/nGNO7ciLTwwH4LMPxvFQ33fSWOvK54MnP+GNB97jykEXblD+yr1vFxmwe/frwdcJQT4b+cdHyalrZvclFki6JOZzxq7z9s2ZvmARfyxaUuQ2rRrVZ+Sv0fCC85etYPHKVey8TWPGzvizoqpZKa1csRqA3Nwq5ObmYMa6gA5QrXou5r8Zy2Xs8Ik03q5R0tvv3W1PZv02m5XLVsZYq/TLlif1uK+iRyFlZ8d8zth13XlH3hk7qdhtfpw1l4Pabk+VHNG07lbs1GRrmmzlXeKUJCdHPPjCeQz+6EpGj/yFH8dFKavLbjqW54ddSfOWjXhz8Ffrtt+3SzsefvF8rrvzFBo29n7iyqPbBYfz6Pd3cfmA86hdtxYAW9TaglOuPJanb3opzbWLXx45SU+ZLJbaSTpN0ltAS0lvJkwfAyWO3JHJqlbJ4aAdW/He+J+L3e6V78Yxa/FSXu51OtcecQDfTZ9Jnj9hlig/37jglIfpflg/dty5Gdu12hqAu/u8zhmH3Mm0KXPY/7CdARj56Y/06Ho35538EN+N/IUrbj4+nVWv1N56+AN6tL6I3rv/h/kzF/KvfmcBcNaNJ/HKvW9n/VM6RINkJDtlsrjSL18AM4GGRH2/FFgCjClqp8QRuhsfeRJ19+gcU/XKbr/WLZgwczbzli0vdru8fKPve5+uW36+5yn8Nm9B3NXLGsuWrOSHr6fQYZ82TP1lNhAF/E/fG8tJZ+/L0De+Y8miFeu2f++1b+n579gHas9aC2evb6A25LEPufmtqwFou1cb9juhE+fe0Z3adWuRn2+sWbmGNx58L11Vjc2aLOn7Ja6ud6cCU4FSReXEEbrb9rknIx9rj/xb2xJTLwBbVM1FwIo1a9l7+21Zm5/PL3Mq9R8psatTryZr1+azbMlKqlXPpX2nVrz05AiaNK/PzOnRvev097ZMnzIXgPoNazN/7tJ15dOmzElb3Su7+n+py/xZCwHY57i9+G1c9D7osr/fsG6bM/ucxIqlK7MyoIP3p56UjQbLqAZUBZZV1kEyalTNZZ9W29LnrQ/XlR3cthXXdz2Q+rVq8MgZ3Zg0aw7/fPo1GtSqyeNnHke+GX8uXsZVr2bnf4RUqt9wSy6/+Xiq5AjliM8+GM+o4T9x1xM9qVmrOhL8+tMsHrj1bQC6ndaJTge0JW9tPksWr6DfDa+l+Qoqh2ufvYRdDtiJOg235Llpj/DUjS+y6993otVuLTAz/vxtDvf2fjTd1axw2fJFqSqqJYEkEXUb0MnMri5p+0x9Us8mLd7wdFDc8sZMTHcVNgtD818q92P27RO6Jh1zrmk3JGMf6yvsV5NFXgcOq6hzOudcsvItJ+kpk8WdfklsjpBD1O1u9r9Gd85VOt5NQHKOTphfC/xGlIJxzrmMki0fH8Ua1M3snDiP75xzqZLK9ueSBgJHAbPNbOdQdifRg+5q4BfgHDNbGNZdA/QE8oCLzez9UH44cB9QBXjczPqWdO5YfzVJ2kHSMEnjwvIukq6P85zOOVcWKf6i9Eng8I3KhgI7m9kuwE/ANQCS2gGnEo3dfDjwkKQqofvyB4EjgHbAaWHbYsX998ZjRBVfA2BmY4gq75xzGSWVX5Sa2Wds9PW8mX1gZmvD4kigWZjvBgw2s1VmNgWYDOwVpslm9quZrQYGk0T6Ou6gXtPMRm1UtrbQLZ1zLo1KM/C0pF6SvkmYepXydP8A3g3zTYHpCet+D2VFlRcr7helcyW1InyAJOlEou4DnHMuo6zJT/4ZN/Hr99KSdB3Rw+2zZdm/JHEH9QuILrytpBnAFKKxSp1zLqNURPtzSWcTvUDtYuu//JwBNE/YrFkoo5jyIsUd1GcATwAfA/WBxUTd8frg0865jBJ33y+hJcuVwN/NLLFHwDeB5yTdDWwDtAFGAQLaSGpJFEtPBU4v6TxxB/U3gIXAaOCPmM/lnHNlluImjc8DBwANJf0O9CFqNFIdGBr1msJIM+ttZuMlvQhMIErLXGBmeeE4FwLvEzVpHGhm40s6d9xBvZmZbdysxznnMk4q0y9mdlohxQOK2f5W4NZCyocAQ0pz7riTSF9I+lvM53DOuXLLR0lPmSzuJ/V9gbMlTQFWEeWILDS+d865jLEm3/t+ScYRMR/fOedSItOHqUtW3H2/TI3z+M45lyqZnlZJVnYMyuecc+XkT+rOOZdFMn3wi2R5UHfOOWCtB3XnnMsenn5xzrks4kHdOeeyiAd155zLIh7UnXMui3g7deecyyJrSzFIRibzoO6cc3j6xTnnsooHdeecyyLmQd0557KHvyh1zrks4ukX55zLInne+iVezYctL3kjVy5D3huc7ipkvSNad053FVySPKfunHNZxNMvzjmXRczSXYPU8KDunHN46xfnnMsq2fKiNDuuwjnnysks+akkkgZKmi1pXEJZfUlDJf0cftYL5ZJ0v6TJksZIap+wT4+w/c+SeiRzHR7UnXOOqPVLslMSngQO36jsamCYmbUBhoVlgCOANmHqBTwM0S8BoA/QEdgL6FPwi6A4HtSdc47UBnUz+wyYv1FxN2BQmB8EHJtQ/pRFRgJ1JTUBDgOGmtl8M1sADGXTXxSb8Jy6c85RuiaNknoRPVUX6G9m/UvYrbGZzQzzs4DGYb4pMD1hu99DWVHlxfKg7pxzlK5JYwjgJQXx4vY3SbE0ovT0i3POAfn5OUlPZfRnSKsQfs4O5TOA5gnbNQtlRZUXy4O6c84BVoqpjN4EClqw9ADeSCg/K7SC6QQsCmma94FDJdULL0gPDWXF8vSLc86R2r5fJD0PHAA0lPQ7USuWvsCLknoCU4GTw+ZDgK7AZGA5cE5UH5sv6Wbg67Ddf81s45evm/Cg7pxzUK5H8E0OZXZaEau6FLKtARcUcZyBwMDSnNuDunPOsRn00ijp/yjmd5eZXRxLjZxzLg3y87M8qAPfVFgtnHMu3bL9Sd3MBiUuS6ppZj5yhXMuK2VL17slNmmU1FnSBGBSWN5V0kPJnkBSDUk7lqOOzjkXvwpo01gRkmmnfi9RHwTzAMzsB2D/ZA4u6Wjge+C9sLybpDfLVlXnnItPijv0SpukPj4ys+kbFeUlefwbiXoXWxiO8z3QMtnKOedchcmSJ/VkmjROl7Q3YJKqApcAE5M8/hozWyRt8Jstw2+Jc25zZFnS+iWZJ/XeRA3jmwJ/ALtRREP5QoyXdDpQRVKb0EzyizLV1DnnYqVSTJmrxCd1M5sLnFHG418EXAesAp4j6rfgljIeyznn4pMlOYQSg7qk7YH7gE5El/0lcKmZ/ZrE8dua2XVEgd055zJXlgT1ZNIvzwEvAk2AbYCXgOeTPH4/SRMl3Sxp5zLW0Tnn4mdKfspgyQT1mmb2tJmtDdMzwBbJHNzMDgQOBOYAj0oaK+n6ctTXOedikcqBp9OpyKAeRr6uD7wr6WpJLSRtJ+lKoq4ik2Jms8zsfqIXrt8DN5S71s45l2r5Sn7KYMXl1L8lyjIVXMG/EtYZcE1JB5f0V+AU4ASij5deAC4vU02dcy5G8QwuV/GK6/slFR8JDSQK5IeZ2R8pOJ5zzsUj24N6ovCSsx0JuXQze6qk/cysc9mr5pxzFSjDX4AmK5kmjX2IhmVqR5RLPwIYARQZ1CW9aGYnSxrLhr//RDTQxy7lqbRzzqXcZvSkfiKwK/CdmZ0jqTHwTAn7XBJ+HlWeyjnnXIXJT3cFUiOZoL7CzPIlrZW0FTAbaF7cDmEkbIDzzeyqxHWS7gCu2nSvzFa1WhXufuAsqlarQpUqOQz/eBJPDfyM3fdowbnndyEnR6xYsZo7b32LP2YsoFHjrbjyuqOpXXsLcnLEgEc+ZtTIX9J9GRnpur7wyZdQvx689eSG6554Af73kPjiDaNeXViyFK68BWbOhrV58I9T4Piu8NVo6Pvg+v1+nQb9boCD96vQS6kULnuoJx2P2I2Fcxbzr72i7wK337k5F913NjVqV+fPqXO5o+cjLF+yct0+jZrV57FvbueZ217n5fvfTVfV45Ul6Zdk2ql/I6ku8BhRi5jRRF+VJuOQQsqOSHLfjLJmdR7/ueQZep/9OL3PfpwOnbbnrzttw8VXHE7f/75O73Me56Oh4zmjxz4AnNFjXz79aCLn/WMAt974OhddfniaryBzHXsE9L9z0/KZs+Hzr6FJ4/V/Fz/3GrRqAa8PhKfug/89BKvXQMf28NqAaHriHqhRHfbZs+KuoTL54NkRXHfsXRuU/fvBfzCwz4v07ng9n7/1LSf+u+sG6//V93S+HjqmIqtZ4WTJT5msxKBuZueb2UIze4QoSPcws3OK20fSeSGfvqOkMQnTFKDS/stYuWINALm5OeRWqbLuQ4SataoDUKtWdebNXQqAmVGrkHK3qT13hbpbblre9wG4ojckdvIpwbLl0X1fvgLqbAW5VTbc74NPYL+OUCOpT+Q2P+M+/5ElC5ZtUNas9V8YO+JHAL77aDz7duuwbl3no9oz67c5TJ04o0LrWeGyvetdSe2LW2dmo4s57nPAu8DtwNUJ5UvMbH6pa5khcnLEQwN6sk3Terz52jdMmvAHd/d9h1vvPIVVq9ayfNkqLv7XkwA8PXA4fe8+jW4ndGCLGlW56t/PpbfylcywEdC4IbRtvWH5GcfD+dfA/sdHQb1fH8jZ6NFkyEfQ4+SKq2s2mDpxBp2Pas+Xb49mv+P2pFHT+gBsUas6J196JNcc8z9OvKRS/pG92Skup96vmHUGHFTkSrNFwCLgNABJWxM1h6wtqbaZTStsP0m9gF4AbVt1o9lfMuvv5/x8o/c5j1OrdnVuvO1EWrRsxAmn7MV1/3mBSRP+4KTTOtH7okO4+453OPDgdnzw7hheHvwVf92pKVddfwznntU/4z8xzgQrVkL/Z+DxuzZdN2IUtG0DT94L02ZAz8uhwy5Qu1a0fvY8+OlX2Heviq1zZXf3+QM4787unHFVN74c8h1rV0fj4Jx57XG89uD7rFy2Ks01jF8q0yqSLgX+SRQrxwLnEPWfNRhoQJTKPtPMVkuqTtSacA+ijzRPMbPfynru4j4+OrCsBy0QhrO7m6gjsNnAdkQDbOxUxDn7A/0BDtn31owNf8uWruKH0VPZs1Mrtm/dmEkTou+qPvloArffdSoAhx+1G9deHvV7NnH8DKpVz6VOnZosXOhjd5dk+gz4fSYc2zNa/nMOnHAuvPAIvPounHt6lIbZrhk0axK9FN3lr9G2730cvRytmtQXGK7A9J9mcm236MVG09aN6XjYrgC03XN79j22Az1vPpnadWpi+cbqVWt489EP01ndeKTo839JTYGLgXZmtkLSi8CpQFfgHjMbLOkRoCfwcPi5wMxaSzoVuIPoS/wySWo4u3K4hajL3p/CF6pdgJExnzMWderWpFbtKEderVou7fdsybSpc6lVqzpNm0d/qu7RoSXTps4DYPafi9l9j+ij3G23a0C1arke0JO0Qyv4/A0Y9kI0NW4ErzwGjRpAk61hZEj8zZ0PU6ZD8ybr931nGBzZJT31rszqNIpeakji9Cu78faAjwC4/NDb6LHTFfTY6Qpee+gDBt/1dnYGdEh1Tj0XqCEpF6gJzCTKbrwc1g8Cjg3z3cIyYX0XbTRcXGnE/TyzxszmScqRlGNmH0u6N+ZzxqJ+g9pced3R5OQI5YjPPprIV19M5p7/DaHPLSeQb8bSJSu56/a3AXj0gQ+57MquHH/KXmBw561vpfkKMtflN8Go72HhIjjgRLjwHDjxyMK3Pb8HXHM7HHN29H/r8n9BvbrRuhkzYdZs2HO3iqp55XT1E+exy35tqdOgNs/8eA9P3/oaNWpX5+hzDwbg8ze/4YOnh6e5lhWvNOmXxFRx0D9kGjCzGZLuAqYBK4APiNItC81sbdj+d6LR5Ag/p4d910paRJSimVu264gxySvpQ6LfRrcDDYlSMHua2d4l7ZvJ6Zds8f7Lg0reyJXLEa29p4yK8P7SQeXOnbTqd3fSMeeXyy8r8nyS6gGvEKVQFhKNQfEycKOZtQ7bNAfeNbOdJY0DDjez38O6X4COYdS5Uisx/aJId0k3hOVtJSX7Gqob0W+qS4H3gF+Ao8tSUeeci1Xq0i8HA1PMbI6ZrQFeBfYB6oZ0DEAzoKCN6AzCB51hfR2iF6ZlkkxO/SGgM6ElC7AEeLDozdczs2VmlhcG1xhkZvebWZkr65xzcUnhx0fTgE6SaobceBdgAvAxUbcrAD2AN8L8m2GZsP4jK0cKJZmcekczay/pOwAzWyCpWjIHl7SETX+vLQK+AS5PcpxT55yLX4pav5jZV5JeJvr6fi3wHVGrvneAwZJuCWUDwi4DgKclTQbmE7WUKbNkgvoaSVUIwVlSI5Lv+uZeohcCzxH10Hgq0IroYgcS9f7onHNpl8p26mbWB+izUfGvwCapazNbCZyUqnMnk365H3gN2FrSrUTd7t6W5PGPMbNHzWyJmS0Ob4cPM7MXgHplq7JzzsUg27sJKGBmz0r6ligvJOBYM5uY5PGXSzqZ9W0zTwQKun7L8FvjnNucZHpHXclKZpCMbYHlwFuJZUV96r+RM4D7iF62GtGHR90l1QAuLFONnXMuDptLUCdK7hcMQL0F0BL4kSI+9U8UXoQW1YRxRJJ1dM652ClLBslIpuvdv5nZLuFnG6JEf1L9qUvaQdKw0LgeSbtIur58VXbOOVeUUvf9Errc7Zjk5o8B1wBrwr5jKGdzHeeci8Xm8qJU0mUJizlAe+CPJI9f08xGbdQ3zdqiNnbOuXTZbF6UAolj0qwlyrG/kuTx50pqxfo27icS9VbmnHOZZXMI6uGjoy3N7IoyHv8Coi+p2kqaAUwhahHjnHOZJduDuqTc0A3kPuU4/gzgCaI+D+oDi4n6OPhvOY7pnHMply2tX4p7Uh9FlD//XtKbRN1Hrhut1sxeTeL4bxB1PTma5PPwzjlX4TannPoWRN1AHsT69upG1J1kSZqZ2eFlr55zzlWQzSCobx1avoxjfTAvkOzlfyHpb2Y2tqwVdM65CrEZBPUqQG02DOYFkr38fYGzJU0BVoVjmZntUqpaOudczDaH9MtMMyvvC80jyrm/c85VjM0gqJe7x3gzm1reYzjnXEXYHFq/dKmwWjjnXLpl+5O6mc2vyIo451w6bQ45deec23x4UHfOuSziQd0557KHp1+ccy6LeFB3zrls4kHdOeeySJYE9VIPZ+ecc9lIlvxU4rGkupJeljRJ0kRJnSXVlzRU0s/hZ72wrSTdL2mypDGS2pfnOjyoO+ccpHqM0vuA98ysLbArMBG4GhhmZm2AYWEZou5U2oSpF/BweS7Dg7pzzhF1E5DsVOxxpDrA/sAAADNbbWYLgW7AoLDZIODYMN8NeMoiI4G6kpqU9ToyNqe+qkG1dFch6x154AnprkLWm3L11umugktSaVq/SOpF9FRdoL+Z9Q/zLYE5wBOSdgW+BS4BGptZwRjNs4DGYb4pMD3hWL+HsjKN55yxQd055ypUKYJ6COD9i1idSzRq3EVm9pWk+1ifainY36R4GlF6+sU55yCVOfXfgd/N7Kuw/DJRkP+zIK0Sfs4O62cAzRP2bxbKysSDunPOkbrWL2Y2C5guacdQ1AWYALwJ9AhlPYjGcCaUnxVawXQCFiWkaUrN0y/OOQcoP6XZkIuAZyVVA34FziF6iH5RUk9gKnBy2HYI0BWYDCwP25aZB3XnnIOUfnxkZt8DHQpZtck4FWZmwAWpOrcHdeecw/t+cc657OJB3Tnnsoc/qTvnXDbxoO6cc9mjpM//KwsP6s45h6dfnHMuu1h2RHUP6s45hz+pO+dcdvGg7pxz2cNflDrnXBbxoO6cc9nEX5Q651z28BelzjmXTbIkqMc6SEbo9L27pBvC8raS9orznM45VxapGiQj3eIe+eghoDNwWlheAjwY8zmdc67UlG9JT5ks7vRLRzNrL+k7ADNbEEYCcc65zJLZsTppcQf1NZKqEG6XpEZAljQccs5lk0xPqyQr7qB+P/AasLWkW4ETgetjPqdzzpVehqdVkhVrUDezZyV9SzQun4BjzWxinOd0zrkyyY6YHm9Ql3Q/MNjM/OWocy6jZUv6Je7WL98C10v6RdJdkgobXds559IuW1q/xBrUzWyQmXUF9gR+BO6Q9HOc53TOuTKxUkwZrKK+KG0NtAW2Azyn7pzLOMqSvl/i/qL0f+HJ/L/AOKCDmR0d5zmdc65M8ksxJUFSFUnfSXo7LLeU9JWkyZJeKPhmR1L1sDw5rG9RnsuIO6f+C9DZzA43syfMbGHM53POuTKRWdJTki5hw8zEHcA9ZtYaWAD0DOU9gQWh/J6wXZnFkn6R1NbMJgFfA9tK2jZxvZmNjuO8FSEnR/TvdyZz5y3l6lte5f9uO42aNaKPZOvVrcnEn2Zy3e2vA3DxuQfRaY/tWbVqLbffN4Sffp2dzqpnvKrVcrnz6V5UrZZLldwcRrw/jmce+JDGTetxdb/T2KpuTX6eMIO7rnqRtWvy1u23zyE7cf393bn4xAf4efyMNF5B5dCyfj3uPb7ruuXm9epw36dfstUW1Tl5t78xf/lyAO7++HM+/eU39m65LVcctC9Vq1RhTV4e/xs2nJG/TU9X9eOTwuyLpGbAkcCtwGWSBBwEnB42GQTcCDwMdAvzAC8DD0iSWdnyQXHl1C8DegH9CllnRBdXKZ141B5MnT6PWjWrA3DRtc+vW3fzVd0YMWoyAJ32aEmzJvU4vffjtNuhCZeddwi9//NsWupcWaxZvZarz3mclctXUyU3h7ue6c03w3/kuB778vpTI/h0yBgu7HMsh53QgXcGfwVAjZrV6HbWPkz6YVqaa195TJm/gG6PR/8WcySGX3IuQ3+czAm77sQTo0YzcOS3G2y/YPkKer/wBrOXLqNNowYMPO149rv/sXRUPValadUiqRdRjCvQ38z6JyzfC1wJbBmWGwALzWxtWP4daBrmmwLTAcxsraRFYfu5pb0GiCn9YmYFF3uEmR2YOAFdi9s3kzVqUJvOHbbnnaFjN1lXs0Y12u+yLcNHRo179t2rDe9/PB6ACT/NpHatLWhQr1aF1rcyWrl8NQC5uVXIrZqDGezaqRXD3x8HwIdvjKZzl3brtj/rkkN56fFPWb1qbaHHc8Xr3LI50xYs4o9FS4rcZuKfc5i9dBkAP8+ZR/WquVStUqWiqlhxzJKezKy/mXVImNYFdElHAbPN7NtizhabuHPqXyRZVilc9M+DeHjQp+QX8lfRfp3a8O2YqSxfEQWlhg1qM3vu+v8oc+YuoQ+2K4QAAA6bSURBVGGD2hVW18oqJ0c88OpFPD/iOr77YjIzp81j2eKV5OdFb6fmzlpEg8ZbAdCq3TY0/Esdvv70x3RWuVI7st2OvDN+0rrl7h125c1zu3PbUYew1RbVN9n+sLZtmDBrNmvy8jZZV9kpP/mpBPsAx0j6DRhMlJm4D6grqSA70gwoyBXOAJoDhPV1gHllvY5Ygrqkv0jaA6ghaXdJ7cN0AFAzjnPGrXOH7VmwcDk//fJnoeu77NeWYZ9NKnSdS15+vnHh8f/HmQf2ZYe/NaP59o0K3U4Sva46ksfueKeCa5g9qubk0GWHVrw7Mfrr8rlvx3Dwg0/Q7bFnmLN0GVcfvP8G27du2ID/dNmX/zfkw3RUN36leFIv/jB2jZk1M7MWwKnAR2Z2BvAxUf9XAD2AN8L8m2GZsP6jsubTIb6c+mHA2US/je5OKF8CXFvUTol5qta7HE+TFp1iql7p/e2vTdlnr9Z02mN7qlXLpVbNalx/6ZHccs871NmyBn9t04TrwwtSgLnzlrJ1wy3XLTdquCVz5y1NR9UrpWVLVjJm1K+03W1bam21BTlVcsjPy6fhX+ow78/F1KhVje3aNOZ/T0WZvnoNa9PnobO46fyn/GVpkvZv3YLxs2Yzb1n0YrTgJ8CL343j0VO6rVtuvGVtHjzpaK58432mL1hU4XWtEPE3U78KGCzpFuA7YEAoHwA8LWkyMJ/oF0GZxRLUzWwQMEjSCWb2Sin26w/0B9i/250Z9SVA/6eH0//p4QDstnNzTj12T265J3pK/Ps+O/DlN7+wOqFFxohRkzn+yN0ZNnwS7XZowrJlq5i3YFla6l5Z1KlXi7Vr81i2ZCXVqueye+fWvDTgM8Z89Sv7HbYznw4Zw8Hd2vPlRxNZvnQVp+59y7p97xh0Lo//b4gH9FI4aqe2vJ2QemlUuxZzQu78kB1b8fOcKAOwZfXqPHbqsfT7aASjf/8jLXWtCMpPfa/gZvYJ8EmY/xXYZOQ3M1sJnJSqc8bVpLG7mT0DtJB02cbrzezuQnartLrs25ZnX/lqg7KR3/5K5w7b8/wj57Jq1Rpu/79301S7yqNeoy254vaTyKkilCOGvzeWUZ9MYtrkP7m632mcdfGh/DLxDz54+et0V7XSq1E1l71bbrtBKuXKLvvRtnEjzIwZixZzw5BhAHTfc1e2rVeXC/bryAX7dQTgnOdeZf7yFWmpe2yyZKQHlSN1U/RBpX+Z2aOS+hS23sxuKukYmfakno1q/TQ/3VXIer+csXW6q7BZ+On6S1XeYxy2501Jx5z3v+5T7vPFJa70y6PhZ4nB2znnMoL3/VKy0PfLVpKqShomaY6k7nGe0znnyiRFrV/SLe526oea2WLgKOA3ot4a/xPzOZ1zrvRS3KFXusTd9W7B8Y8EXjKzRVEXCM45l1niaP2SDnEH9bclTQJWAOdJagSsjPmczjlXehmeVklW3CMfXQ3sTdSP+hpgGVGPZM45l1myJKce98DTVYHuwP4h7fIp8Eic53TOuTLJjuxL7OmXh4GqwENh+cxQ9s+Yz+ucc6WSLcPZxR3U9zSzXROWP5L0Q8zndM650vOgnpQ8Sa3M7BcASdsD2ddnp3Ou8svLjvxL3EH9P8DHkn4Nyy2Ac2I+p3POlV6WPKnH/fHR58CjRK8g5of5L2M+p3POlZ63fknKU8Bi4OawfDrwNCnsZtI551KiFGOUZrK4g/rOZtYuYfljSRNiPqdzzpWeZUdOPe70y2hJ64YvktQR+CbmczrnXOnl5Sc/ZbC4n9T3AL6QNC0sbwv8KGksYGa2S8znd8655GR4rjxZcQf1w2M+vnPOpYYH9ZKZ2dQ4j++ccynjQd0557KId73rnHNZxJ/UnXMui2R4q5Zkxd2k0TnnKgWz/KSn4khqLuljSRMkjZd0SSivL2mopJ/Dz3qhXJLulzRZ0hhJ7ctzHR7UnXMOoi9Kk52Ktxa4PHx42Qm4QFI74GpgmJm1AYaFZYAjgDZh6kXUPXmZeVB3zjlIWd8vZjbTzEaH+SXARKAp0ahvg8Jmg4Bjw3w34CmLjATqSmpS1svwoO6ccxC1fklyktRL0jcJU6/CDimpBbA78BXQ2MxmhlWzgMZhvikwPWG330NZmfiLUuecg1K1fjGz/kD/4raRVBt4Bfi3mS0OQ3oW7G+SYmlu40HdOecAy0vd+D1hfOZXgGfN7NVQ/KekJmY2M6RXZofyGUDzhN2bhbIy8fSLc85Byl6UKnokHwBMNLO7E1a9CfQI8z2ANxLKzwqtYDoBixLSNKXmT+rOOQep7Hp3H+BMYKyk70PZtUBf4EVJPYGpwMlh3RCgKzAZWE45R4fzoO6cc4ClaJAMMxsBqIjVXQrZ3oALUnJyPKg751wkSwbJ8KDunHOk9kVpOsmypBObTCCpV2jq5GLi9zh+fo8rN2/9klqFfoDgUsrvcfz8HldiHtSdcy6LeFB3zrks4kE9tTwPGT+/x/Hze1yJ+YtS55zLIv6k7pxzWcSDunPOZREP6ikmqYWk08u479JU1yebSaor6fyE5W0kvZzOOlV2knpLOivMny1pm4R1j4cRfFwG85x6ikk6ALjCzI4qZF2uma0tZt+lZlY7zvplkzAAwdtmtnOaq5KVJH1C9G/5m3TXxSXPn9SD8IQ9UdJjYbDYDyTVkNRK0nuSvpU0XFLbsP2Tkk5M2L/gKbsvsJ+k7yVdGp523pT0ETBMUm1JwySNljRWUrc0XG6FKMM9bSVpZLgvtxTc02LuWV+gVbjXd4bzjQv7jJS0U0JdPpHUQVItSQMljZL0XTbd/3D9kyQ9G+77y5JqSuoSrnVsuPbqYfu+igZHHiPprlB2o6Qrwr/tDsCz4f7WSLiHvSXdmXDesyU9EOa7h3v7vaRHJVVJx73YrJmZT9FfKy2IBozdLSy/CHQnGiC2TSjrCHwU5p8ETkzYf2n4eQDR02NB+dlEw1PVD8u5wFZhviFRd5tKPEa2TGW4p28Dp4X53gn3tNB7Fo4/bqPzjQvzlwI3hfkmwI9h/jage5ivC/wE1Er3vUrh/TZgn7A8ELieaKi0HULZU8C/gQbAjwn/9uqGnzcSPZ0DfAJ0SDj+J0SBvhEwOaH8XWBf4K/AW0DVUP4QcFa678vmNvmT+oammFlB/8ffEv0n2Rt4KfSL/ChRgCitoWY2P8wLuE3SGOBDorEIGxe5Z+VXmnvaGXgpzD+XcIyy3LMXgYK/pE4GCnLthwJXh3N/AmwBbFvqq8pc083s8zD/DFFXr1PM7KdQNgjYH1gErAQGSDqeqB/vpJjZHOBXSZ0kNQDaAp+Hc+0BfB3ubxdg+xRckysF76VxQ6sS5vOIAsdCM9utkG3XEtJXknKAasUcd1nC/BlETzp7mNkaSb8RBZZsVZp7WpRS3zMzmyFpnqRdgFOInvwh+gVxgpn9WIrzVyYbvyRbSPRUvuFGZmsl7UUUeE8ELgQOKsV5BhP9spwEvGZmJknAIDO7pkw1dynhT+rFWwxMkXQSRMNUSdo1rPuN6KkE4BigaphfAmxZzDHrALNDcDoQ2C7ltc5sxd3TkcAJYf7UhH2Kumcl3esXgCuBOmY2JpS9D1wUAhCSdi/vBWWYbSV1DvOnA98ALSS1DmVnAp8qGhS5jpkNIUpV7brpoYq9v68B3YDTiAI8RGm1EyVtDSCpvqTN7d932nlQL9kZQE9JPwDjif4hAzwG/D2Ud2b90/gYIE/SD5IuLeR4zwIdJI0FziJ60tncFHVP/w1cFtIsrYlSBFDEPTOzecDnksYlvrhL8DLRL4cXE8puJvoFPEbS+LCcTX4ELpA0EagH3EM0PNpL4f7lA48QBeu3w70eAVxWyLGeBB4peFGauMLMFgATge3MbFQom0CUw/8gHHcoZUtXunLwJo0uY0iqCawIf8qfSvTSNGtap8RN3sTT4Tl1l1n2AB4IqZGFwD/SXB/nKh1/UnfOuSziOXXnnMsiHtSdcy6LeFB3zrks4kHdFUtSXmjSNk7SS6GFSlmPta6/HJXQ45+kAyTtXYZz/CapYbLlG21Tql4yC/pJKW0dnYuTB3VXkhVmtltoJrea9V9mAlHPk2U5qJn9M7RrLsoBRN0JOOdKwYO6K43hQOvwFD1c0pvABElVFPWS+HXo8e9fsO5r0Qck/SjpQ2DrggMV9PgX5g9X1APjD4p6Y2xB9Mvj0vBXwn6SGkl6JZzja0n7hH0bKOr9cbykx4m6ASiWpNcV9RA5XlKvjdbdE8qHSWoUygrtVdK5TOTt1F1SwhP5EcB7oag9sLOZTQmBcZGZ7amoW9fPJX0A7A7sCLQj6vNlAlHPgYnHbUT0de7+4Vj1zWy+pEeIemks6BL2OeAeMxshaVuiz/3/CvQBRpjZfyUdCfRM4nL+Ec5Rg6jzqVfC16m1gG/M7FJJN4RjX0g0EHNvM/tZUkei3gdL00+KcxXGg7orSQ1FPe5B9KQ+gCgtMsrMpoTyQ4FdtL5/+TpAG6LeAJ83szzgD0V9ym+sE/BZwbESerPc2MFAu9BlC8BWof+S/YHjw77vSFqQxDVdLOm4MN881HUe0Sf0L4TyZ4BXwzkKepUs2L96EudwLi08qLuSrNi4R8UQ3BJ7nhRwkZm9v9F2XVNYjxygk5mtLKQuSVM0MtXBQGczW65odJ+ieny0cN7S9irpXNp4Tt2lwvvAeZKqAkjaQVIt4DPglJBzbwIcWMi+I4H9JbUM+9YP5Rv3EPgBcFHBgqSCIPsZUW+ESDqCqBOr4tQBFoSA3pboL4UCOazvg/10orROcb1KOpdxPKi7VHicKF8+WtFwco8S/RX4GvBzWPcU8OXGO4YBF3oRpTp+YH364y3guIIXpcDFRD01jpE0gfWtcG4i+qUwnigNM62Eur4H5IZeDPsS/VIpsAzYK1zDQcB/Q3lRvUo6l3G87xfnnMsi/qTunHNZxIO6c85lEQ/qzjmXRTyoO+dcFvGg7pxzWcSDunPOZREP6s45l0X+P9LSrXAbuyxWAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "acc,y_true,y_pred=learn.test()\n",
        "print(f\"test acc is {acc*100}%\")\n",
        "cf_matrix = confusion_matrix(y_true=y_true, y_pred=y_pred)\n",
        "\n",
        "sb.heatmap(cf_matrix,\n",
        "           annot=True,\n",
        "           fmt='d',\n",
        "           cmap='viridis',\n",
        "           xticklabels=['neutral','negative','positive'],\n",
        "           yticklabels=['neutral','negative','positive'])\n",
        "plt.xlabel('Predicted label')\n",
        "plt.ylabel('True label')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "erusOLymdjWv"
      },
      "source": [
        "# Appendix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Wlbu9AHudmNH"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import cv2\n",
        "import insightface\n",
        "from insightface.app import FaceAnalysis\n",
        "from insightface.data import get_image as ins_get_image\n",
        "import os\n",
        "from tqdm import tqdm\n",
        "from PIL import Image\n",
        "import json\n",
        "class faceExtractor():\n",
        "    def __init__(self,rootDir,mode=\"train\",max_num=4) -> None:\n",
        "        self.root=rootDir\n",
        "        self.mode=mode\n",
        "        self.max_num=max_num\n",
        "        if not os.path.exists(f\"{self.root}/FaceDataset\"):\n",
        "          os.mkdir(f\"{self.root}/FaceDataset\")\n",
        "        if not os.path.exists(f\"{self.root}/FaceDataset/faceImage\"):\n",
        "          os.mkdir(f\"{self.root}/FaceDataset/faceImage\")\n",
        "        if not os.path.exists(f\"{self.root}/FaceDataset/faceImage/\"+mode):\n",
        "          os.mkdir(f\"{self.root}/FaceDataset/faceImage/\"+mode)\n",
        "        # if not os.path.exists(f\"{self.root}/FaceDataset/labels\"):\n",
        "        #   os.mkdir(f\"{self.root}/FaceDataset/labels\")\n",
        "        self.fault=0\n",
        "        self.idx_fault=[]\n",
        "        self.max_len=0\n",
        "        self.app = FaceAnalysis(providers=['CUDAExecutionProvider', 'CPUExecutionProvider']) # ,allowed_modules=['detection']\n",
        "        self.Dataset={}\n",
        "    def run(self,Dataset):\n",
        "        self.app.prepare(ctx_id=0,det_thresh=0.25,det_size=(640,960))\n",
        "        index = 0\n",
        "        for img,dialog,sentiment,idx in tqdm(Dataset):\n",
        "            faces = self.app.get(img[0])\n",
        "            img_PIL=Image.fromarray(img[0].astype(np.uint8))\n",
        "          \n",
        "            face_count=len(faces)\n",
        "\n",
        "            if face_count > self.max_len:\n",
        "              self.max_len = face_count\n",
        "\n",
        "            if face_count==0:\n",
        "                self.fault+=1\n",
        "                self.idx_fault.append(idx)\n",
        "\n",
        "            else:\n",
        "\n",
        "\n",
        "              if face_count>self.max_num:\n",
        "                a = self.max_num\n",
        "              else :\n",
        "                a = face_count\n",
        "              pos=[]\n",
        "              embedding=[]\n",
        "              \n",
        "              for i,f in zip(range(a),faces):\n",
        "                  pos .append(f['bbox'].tolist()) \n",
        "                  embedding.append(f['embedding'].tolist())\n",
        "              \n",
        "              self.Dataset[index]={'pos':pos,'embedding':embedding,'text':dialog,'sentiment_cls':sentiment,'size':img[0].shape}\n",
        "                  # face=img_PIL.crop(f['bbox'])\n",
        "                  # face.save(f\"{self.root}/FaceDataset/faceImage/{self.mode}/\"+f'{idx}_{i}.jpg')\n",
        "              index+=1\n",
        "\n",
        "        return self.Dataset  \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TdtOs-Vgdoi0"
      },
      "outputs": [],
      "source": [
        "face_train=faceExtractor(rootDir=\".\",mode=\"train\")\n",
        "Dataset_dict=face_train.run(training_data)\n",
        "\n",
        "with open('/content/drive/MyDrive/train_dataset.json', 'w') as f:\n",
        "  json.dump(Dataset_dict,f)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2JZerP9Edskv"
      },
      "outputs": [],
      "source": [
        "face_test=faceExtractor(rootDir=\".\",mode=\"test\")\n",
        "Dataset_dict=face_test.run(test_data)\n",
        "with open('/content/drive/MyDrive/test_dataset.json', 'w') as f:\n",
        "  json.dump(Dataset_dict,f)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "fLms3tNca5BK",
        "g3cuhPLt3GAL",
        "2CKJPNayTX_N",
        "wkH76UW5TSHN",
        "jw6k5wnVgNVz",
        "erusOLymdjWv"
      ],
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    },
    "vscode": {
      "interpreter": {
        "hash": "570feb405e2e27c949193ac68f46852414290d515b0ba6e5d90d076ed2284471"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}