adapters/question-answering/question_answering_adapter_fusion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rEJBSTyZIrIb"
   },
   "source": [
    "# Fine-tuning BERT (base or large) on a Question-Answering task by using the library adapter-transformers (notebook version) and its Fusion mode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Credit**: [Hugging Face](https://huggingface.co/) and [adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)\n",
    "- **Author**: [Pierre GUILLOU](https://www.linkedin.com/in/pierreguillou/)\n",
    "- **Date**: 08/02/2021\n",
    "- **Blog post**: [NLP nas empresas | Como criar um modelo BERT de Question-Answering (QA) de desempenho aprimorado com AdapterFusion?](https://medium.com/@pierre_guillou/nlp-nas-empresas-como-criar-um-modelo-bert-de-question-answering-qa-de-desempenho-aprimorado-21b699326c82)\n",
    "- **Link to the folder in github with this notebook and all necessary scripts**: [question-answering with adapters](https://github.com/piegu/language-models/tree/master/adapters/question-answering/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objective"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective here is to **fine-tune a Masked Language Model (MLM) like BERT (base or large) for a QA task by training adapters (library [adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)) on its Fusion mode, not the embeddings and transformers layers of the MLM model**, and to compare results with BERT model fully fine-tune for the same task.\n",
    "\n",
    "The interest is obvious: if you need models for different NLP tasks, instead of fine-tuning and storing one model by NLP task, **you store only one MLM model and the trained tasks adapters which sizes are between 1% (QA adapter) and 8% (Lang+QA adapters) of the MLM model one** (it depends of the choosen adapter configuration). **With the fusion mode, there is one more adapter by layer (the fusion adapter) that raises the number of adapter parameters from 23.5% (QA+NER adapters) to 31% (LANG+QA+NER adapters).** More, the loading of these adapters in production is very easy.\n",
    "\n",
    "The difference here with our others notebooks about Question-Answering (QA) with Adapters (see our folder [question-answering](https://github.com/piegu/language-models/tree/master/adapters/question-answering) in github) is **the use of the Fusion mode**: instead of training only a task adapter on QA (with or without a lang adapter), we've trained a fusion adapter in QA that will make a weighted composite of the parameters of the adapters added to our model. **The idea here is not just to kepp all knowledge from other tasks, but to compose with it.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Adapter Fusion architecture](images/adapterfusion1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will see how to fine-tune one of the [🤗 Transformers](https://github.com/huggingface/transformers) model to a question answering task, which is the task of extracting the answer to a question from a given context. We will use the library [adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers) and see how to easily load a dataset for these kinds of tasks and use the `Trainer` API to fine-tune a model on it.\n",
    "\n",
    "![Widget inference representing the QA task](images/qa_adapter_fusion.png)\n",
    "\n",
    "**Note:** This notebook finetunes models that answer question by taking a substring of a context, not by generating new text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4RRkXuteIrIh"
   },
   "source": [
    "This notebook is built to run on any question answering task with the same format as SQUAD (version 1 or 2), with any model checkpoint from the [Model Hub](https://huggingface.co/models) as long as that model has a version with a token classification head and a fast tokenizer (check on [this table](https://huggingface.co/transformers/index.html#bigtable) if this is the case). It might just need some small adjustments if you decide to use a different dataset than the one used here. Depending on you model and the GPU you are using, you might need to adjust the batch size to avoid out-of-memory errors. Set those three parameters, then the rest of the notebook should run smoothly:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### History and Credit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is an adaptation of the following notebooks and scripts for **fine-tuning a (transformer) Masked Language Model (MLM) like BERT (base or large) on the QA task with any QA dataset** (we use here the [Portuguese Squad 1.1 dataset](https://forum.ailab.unb.br/t/datasets-em-portugues/251/4)):\n",
    "- **from [adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** | notebook [03_Adapter_Fusion.ipynb](https://github.com/Adapter-Hub/adapter-transformers/blob/master/notebooks/03_Adapter_Fusion.ipynb), notebook [04_Cross_Lingual_Transfer.ipynb](https://github.com/Adapter-Hub/adapter-transformers/blob/master/notebooks/04_Cross_Lingual_Transfer.ipynb) and script [run_qa.py](https://github.com/Adapter-Hub/adapter-transformers/blob/master/examples/question-answering/run_qa.py) (this script was adapted from the script [run_qa.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py) of HF)\n",
    "- **from [transformers](https://github.com/huggingface/transformers) of Hugging Face** | notebook [question_answering.ipynb](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb) and script [run_qa.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py) \n",
    "\n",
    "In order to speed up the fine-tuning of the model on only one GPU, the library [DeepSpeed](https://www.deepspeed.ai/) could be used by applying the configuration provided by HF in the notebook [transformers + deepspeed CLI](https://github.com/stas00/porting/blob/master/transformers/deepspeed/DeepSpeed_on_colab_CLI.ipynb) but as the library adapter-transformers is not synchronized with the last version of the library transformers of HF, we keep that option for the future."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Major changes from original notebooks and scripts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script [run_qa.py](https://github.com/Adapter-Hub/adapter-transformers/blob/master/examples/question-answering/run_qa.py) allows to evaluate the model performance against f1 metric at the end of each epoch, and not against validation loss as done in the HF notebook [question_answering.ipynb](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb). This is very important as we consider the metric when selecting a model, not the loss. Therefore, we decided to launch this script inside the notebook [question_answering_adapter_script.ipynb](https://github.com/piegu/language-models/blob/master/adapters/question-answering/question_answering_adapter_script.ipynb) (by simulating terminal command line) instead of running code in cells as done in the HF notebook. \n",
    "\n",
    "However, to provide an HF-like notebook, we also updated the HF notebook [question_answering.ipynb](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb) on this notebook ([question_answering_adapter.ipynb](https://github.com/piegu/language-models/blob/master/adapters/question-answering/question_answering_adapter.ipynb)), but without the f1 metric as the evaluation metric during model training (ie, the f1 metric is used only after the model training)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More, we updated the script [run_qa.py](https://github.com/Adapter-Hub/adapter-transformers/blob/master/examples/question-answering/run_qa.py) to [run_qa_adapter.py](https://github.com/piegu/language-models/blob/master/adapters/question-answering/run_qa_adapter.py) with the following changes:\n",
    "- **EarlyStopping** by selecting the model with the highest eval f1 (patience of 3 before ending the training)\n",
    "- **MAD-X 2.0** that allows not to train adapters in the last transformer layer (read page 6 of [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://arxiv.org/pdf/2012.15562.pdf))\n",
    "- **Stack method** for the lang and task adapters when a lang adapter is loaded ([doc](https://docs.adapterhub.ml/adapter_composition.html?highlight=stack#stack))\n",
    "- **Fusion method** of 2 task adapters (see [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://arxiv.org/pdf/2005.00247.pdf)).\n",
    "- **Houlsby MHA last layer** that allows no to train the task adapter after the Feed Fordward but only after the MHA (Multi-Head Attention) in the last layer for the Houlsby configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pathlib\n",
    "from pathlib import Path\n",
    "\n",
    "#root path\n",
    "root = Path.cwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python: 3.8.10 (default, Jun  4 2021, 15:09:15) \n",
      "[GCC 7.5.0]\n",
      "Pytorch: 1.9.0\n",
      "adapter-transformers: 2.1.1\n",
      "HF transformers: 4.8.2\n",
      "tokenizers: 0.10.3\n",
      "datasets: 1.9.0\n"
     ]
    }
   ],
   "source": [
    "import sys; print('python:',sys.version)\n",
    "\n",
    "import torch; print('Pytorch:',torch.__version__)\n",
    "\n",
    "import transformers; print('adapter-transformers:',transformers.__version__)\n",
    "import transformers; print('HF transformers:',transformers.__hf_version__)\n",
    "import tokenizers; print('tokenizers:',tokenizers.__version__)\n",
    "import datasets; print('datasets:',datasets.__version__)\n",
    "\n",
    "# import deepspeed; print('deepspeed:',deepspeed.__version__)\n",
    "\n",
    "# Versions used in the virtuel environment of this notebook:\n",
    "\n",
    "# python: 3.8.10 (default, Jun  4 2021, 15:09:15) \n",
    "# [GCC 7.5.0]\n",
    "# Pytorch: 1.9.0\n",
    "# adapter-transformers: 2.1.1\n",
    "# HF transformers: 4.8.2\n",
    "# tokenizers: 0.10.3\n",
    "# datasets: 1.9.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Model & dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select a MLM BERT base or large in the dataset language\n",
    "# model_checkpoint = \"neuralmind/bert-base-portuguese-cased\"\n",
    "model_checkpoint = \"neuralmind/bert-large-portuguese-cased\"\n",
    "model_checkpoint_original = \"neuralmind/bert-large-portuguese-cased\"\n",
    "model_checkpoint_original_local = root.parent/'neuralmind-bert-large-portuguese-cased-lenerbr/mlm_base'\n",
    "\n",
    "# SQuAD 1.1 in Portuguese\n",
    "dataset_name = \"squad11pt\"\n",
    "\n",
    "# datasets of task adapters already trained\n",
    "dataset_name_ad1 = \"squad11pt\"\n",
    "dataset_name_ad2 = \"lenerbr\"\n",
    "\n",
    "# This flag is the difference between SQUAD v1 or 2 (if you're using another dataset, it indicates if impossible\n",
    "# answers are allowed or not).\n",
    "squad_v2 = False # If true, some of the examples do not have an answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Main hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "task = \"qa_fusion\"\n",
    "\n",
    "# tasks of task adapters already trained\n",
    "task_ad1 = \"qa\"\n",
    "task_ad2 = \"ner\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training arguments\n",
    "batch_size = 16\n",
    "gradient_accumulation_steps = 2\n",
    "\n",
    "learning_rate = 3e-5\n",
    "num_train_epochs = 5.\n",
    "early_stopping_patience = 3\n",
    "\n",
    "weight_decay = 0.01\n",
    "\n",
    "adam_epsilon = 1e-6\n",
    "\n",
    "fp16 = True\n",
    "ds = False # If True, we use DeepSpeed\n",
    "if ds:\n",
    "    zero2 = True\n",
    "\n",
    "# best model\n",
    "load_best_model_at_end = True \n",
    "metric_for_best_model = \"loss\"\n",
    "greater_is_better = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train adapter\n",
    "train_adapter = True # we want to train an adapter\n",
    "load_adapter = None # we do not upload an existing adapter \n",
    "\n",
    "# lang adapter\n",
    "with_adapters_mlm = True # if False, we do not upload an existing lang adapter\n",
    "\n",
    "if with_adapters_mlm:\n",
    "    adapter_composition = \"stack\" # we will stack the lang and task adapters\n",
    "else:\n",
    "    adapter_composition = None\n",
    "\n",
    "# if True, do not put adapter in the last transformer layer (Pfeiffer configuration)\n",
    "madx2 = True\n",
    "\n",
    "# if True, put only an adapter after the MHA but not after the FF in the last layer (Houlsby configuration)\n",
    "houlsby_MHA_lastlayer = False\n",
    "if madx2:\n",
    "    houlsby_MHA_lastlayer = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# gpu\n",
    "n_gpu = 1 # train on just one GPU\n",
    "gpu = 1 # select the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select GPU 0\n",
    "import os\n",
    "os.environ['MASTER_ADDR'] = 'localhost'\n",
    "if gpu == 0:\n",
    "    os.environ['MASTER_PORT'] = '9993' # modify if RuntimeError: Address already in use # GPU 0\n",
    "elif gpu == 1:\n",
    "    os.environ['MASTER_PORT'] = '9995'\n",
    "os.environ['RANK'] = \"0\"\n",
    "os.environ['LOCAL_RANK'] = str(gpu)\n",
    "os.environ['WORLD_SIZE'] = \"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DeepSpeed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "if ds:\n",
    "    if zero2:\n",
    "        ds_config = 'ds_config_zero2.json'\n",
    "    elif zero3:\n",
    "        ds_config = 'ds_config_zero3.json'       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's setup the `DeepSpeed` configuration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use a LR Linear Decay after warmup as scheduler (equivalent of the one by default in the HF Trainer), we changed `WarmupLR` to `WarmupDecayLR` in the DeepSpeed configuration file, and kept a copy of the scheduler initial code here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "````\n",
    "# the lr stays constant after the warmup (this is not equivalent to teh default scheduler of HF which is Linear)\n",
    "    \"scheduler\": {\n",
    "        \"type\": \"WarmupLR\",\n",
    "        \"params\": {\n",
    "            \"warmup_min_lr\": \"auto\",\n",
    "            \"warmup_max_lr\": \"auto\",\n",
    "            \"warmup_num_steps\": \"auto\"\n",
    "        }\n",
    "    },\n",
    "````"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ZeRO-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "source: https://huggingface.co/transformers/master/main_classes/deepspeed.html#zero-2-example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "cat <<'EOT' > ds_config_zero2.json\n",
    "{\n",
    "    \"fp16\": {\n",
    "        \"enabled\": \"auto\",\n",
    "        \"loss_scale\": 0,\n",
    "        \"loss_scale_window\": 1000,\n",
    "        \"initial_scale_power\": 16,\n",
    "        \"hysteresis\": 2,\n",
    "        \"min_loss_scale\": 1\n",
    "    },\n",
    "\n",
    "    \"optimizer\": {\n",
    "        \"type\": \"AdamW\",\n",
    "        \"params\": {\n",
    "            \"lr\": \"auto\",\n",
    "            \"betas\": \"auto\",\n",
    "            \"eps\": \"auto\",\n",
    "            \"weight_decay\": \"auto\"\n",
    "        }\n",
    "    },\n",
    "\n",
    "    \"scheduler\": {\n",
    "         \"type\": \"WarmupDecayLR\",\n",
    "         \"params\": {\n",
    "             \"last_batch_iteration\": -1,\n",
    "             \"total_num_steps\": \"auto\",\n",
    "             \"warmup_min_lr\": \"auto\",\n",
    "             \"warmup_max_lr\": \"auto\",\n",
    "             \"warmup_num_steps\": \"auto\"\n",
    "         }\n",
    "     },\n",
    "\n",
    "    \"zero_optimization\": {\n",
    "        \"stage\": 2,\n",
    "        \"offload_optimizer\": {\n",
    "            \"device\": \"cpu\",\n",
    "            \"pin_memory\": true\n",
    "        },\n",
    "        \"allgather_partitions\": true,\n",
    "        \"allgather_bucket_size\": 2e8,\n",
    "        \"overlap_comm\": true,\n",
    "        \"reduce_scatter\": true,\n",
    "        \"reduce_bucket_size\": 2e8,\n",
    "        \"contiguous_gradients\": true\n",
    "    },\n",
    "\n",
    "    \"gradient_accumulation_steps\": \"auto\",\n",
    "    \"gradient_clipping\": \"auto\",\n",
    "    \"steps_per_print\": 2000,\n",
    "    \"train_batch_size\": \"auto\",\n",
    "    \"train_micro_batch_size_per_gpu\": \"auto\",\n",
    "    \"wall_clock_breakdown\": false\n",
    "}\n",
    "EOT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ZeRO-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "cat <<'EOT' > ds_config_zero3.json\n",
    "{\n",
    "    \"fp16\": {\n",
    "        \"enabled\": \"auto\",\n",
    "        \"loss_scale\": 0,\n",
    "        \"loss_scale_window\": 1000,\n",
    "        \"initial_scale_power\": 16,\n",
    "        \"hysteresis\": 2,\n",
    "        \"min_loss_scale\": 1\n",
    "    },\n",
    "\n",
    "    \"optimizer\": {\n",
    "        \"type\": \"AdamW\",\n",
    "        \"params\": {\n",
    "            \"lr\": \"auto\",\n",
    "            \"betas\": \"auto\",\n",
    "            \"eps\": \"auto\",\n",
    "            \"weight_decay\": \"auto\"\n",
    "        }\n",
    "    },\n",
    "\n",
    "    \"scheduler\": {\n",
    "         \"type\": \"WarmupDecayLR\",\n",
    "         \"params\": {\n",
    "             \"last_batch_iteration\": -1,\n",
    "             \"total_num_steps\": \"auto\",\n",
    "             \"warmup_min_lr\": \"auto\",\n",
    "             \"warmup_max_lr\": \"auto\",\n",
    "             \"warmup_num_steps\": \"auto\"\n",
    "         }\n",
    "     },\n",
    "\n",
    "    \"zero_optimization\": {\n",
    "        \"stage\": 3,\n",
    "        \"offload_optimizer\": {\n",
    "            \"device\": \"cpu\",\n",
    "            \"pin_memory\": true\n",
    "        },\n",
    "        \"offload_param\": {\n",
    "            \"device\": \"cpu\",\n",
    "            \"pin_memory\": true\n",
    "        },\n",
    "        \"overlap_comm\": true,\n",
    "        \"contiguous_gradients\": true,\n",
    "        \"sub_group_size\": 1e9,\n",
    "        \"reduce_bucket_size\": \"auto\",\n",
    "        \"stage3_prefetch_bucket_size\": \"auto\",\n",
    "        \"stage3_param_persistence_threshold\": \"auto\",\n",
    "        \"stage3_max_live_parameters\": 1e9,\n",
    "        \"stage3_max_reuse_distance\": 1e9,\n",
    "        \"stage3_gather_fp16_weights_on_model_save\": true\n",
    "    },\n",
    "\n",
    "    \"gradient_accumulation_steps\": \"auto\",\n",
    "    \"gradient_clipping\": \"auto\",\n",
    "    \"steps_per_print\": 2000,\n",
    "    \"train_batch_size\": \"auto\",\n",
    "    \"train_micro_batch_size_per_gpu\": \"auto\",\n",
    "    \"wall_clock_breakdown\": false\n",
    "}\n",
    "EOT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adapters config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task adapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task adapter config\n",
    "adapter_config_name = \"pfeiffer\" # houlsby is possible, too\n",
    "if adapter_config_name == \"pfeiffer\":\n",
    "    adapter_non_linearity = 'gelu' # relu is possible, too\n",
    "elif adapter_config_name == \"houlsby\":\n",
    "    adapter_non_linearity = 'swish'\n",
    "adapter_reduction_factor = 16\n",
    "language = 'pt' # pt = Portuguese"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lang adapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "if with_adapters_mlm:\n",
    "    \n",
    "    # hyperparameters used for fine-tuning the MLM with lang adapter\n",
    "    learning_rate_mlm = 1e-4\n",
    "    batch_size_mlm = 16\n",
    "    gradient_accumulation_steps_mlm = 1\n",
    "    adam_epsilon_mlm = 1e-4\n",
    "    num_train_epoch_mlm = 100.\n",
    "    early_stopping_patience_mlm = 10\n",
    "    madx2_mlm = madx2\n",
    "    houlsby_MHA_lastlayer_mlm = houlsby_MHA_lastlayer\n",
    "    ds_mlm = False\n",
    "    fp16_mlm = True\n",
    "    load_best_model_at_end_mlm = True\n",
    "    metric_for_best_model_mlm = \"loss\"\n",
    "    adapter_config_mlm = adapter_config_name + '+inv'\n",
    "\n",
    "    # path to lang adapter\n",
    "    outputs_mlm = model_checkpoint.replace('/','-') + '_' + dataset_name + '/' + 'mlm' + '/' \\\n",
    "    + 'lr' + str(learning_rate_mlm) \\\n",
    "    + '_bs' + str(batch_size_mlm) \\\n",
    "    + '_GAS' + str(gradient_accumulation_steps_mlm) \\\n",
    "    + '_eps' + str(adam_epsilon_mlm) \\\n",
    "    + '_epochs' + str(num_train_epoch_mlm) \\\n",
    "    + '_patience' + str(early_stopping_patience_mlm) \\\n",
    "    + '_madx2' + str(madx2_mlm) \\\n",
    "    + '_houlsby_MHA_lastlayer' + str(houlsby_MHA_lastlayer_mlm) \\\n",
    "    + '_ds' + str(ds_mlm) \\\n",
    "    + '_fp16' + str(fp16_mlm) \\\n",
    "    + '_best' + str(load_best_model_at_end_mlm) \\\n",
    "    + '_metric' + str(metric_for_best_model_mlm) \\\n",
    "    + '_adapterconfig' + str(adapter_config_mlm)\n",
    "\n",
    "    path_to_outputs_mlm = root/'outputs'/outputs_mlm\n",
    "    \n",
    "    # Config of the lang adapter\n",
    "    lang_adapter_path = path_to_outputs_mlm/'adapters-mlm/'\n",
    "    \n",
    "    load_lang_adapter = str(lang_adapter_path)\n",
    "    lang_adapter_config = str(lang_adapter_path) + \"/adapter_config.json\"\n",
    "    if adapter_config_mlm == \"pfeiffer+inv\":\n",
    "        lang_adapter_non_linearity = 'gelu' # relu is possible, too\n",
    "    elif adapter_config_mlm == \"houlsby+inv\":\n",
    "        lang_adapter_non_linearity = 'swish'\n",
    "    lang_adapter_reduction_factor = 2\n",
    "    language_mlm = language\n",
    "    \n",
    "else:\n",
    "    type_lang_adapter = None\n",
    "\n",
    "    load_lang_adapter = None\n",
    "    lang_adapter_config = None\n",
    "    lang_adapter_non_linearity = None\n",
    "    lang_adapter_reduction_factor = None\n",
    "    \n",
    "    language_mlm = None "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### QA Adapter Arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# folder for training outputs\n",
    "\n",
    "outputs_ad1 = model_checkpoint.replace('/','-') + '_' + dataset_name_ad1\n",
    "\n",
    "if with_adapters_mlm:\n",
    "    outputs_ad1 = outputs_ad1 + '/' + 'mlm_' + str(task_ad1) + '_AdCompo' + str(adapter_composition) + '/'\n",
    "else:\n",
    "    outputs_ad1 = outputs_ad1 + '/' + str(task_ad1) + '/'\n",
    "\n",
    "# with LANG adapter\n",
    "if with_adapters_mlm:  \n",
    "    learning_rate_ad1 = 1e-4\n",
    "    batch_size_ad1 = 8\n",
    "    gradient_accumulation_steps_ad1 = 2\n",
    "    adam_epsilon_ad1 = 1e-7\n",
    "    num_train_epochs_ad1 = 15.0\n",
    "    early_stopping_patience_ad1 = 5\n",
    "    with_adapters_mlm_ad1 = with_adapters_mlm\n",
    "    madx2_ad1 = madx2\n",
    "    houlsby_MHA_lastlayer_ad1 = houlsby_MHA_lastlayer\n",
    "    ds_ad1 = False\n",
    "    fp16_ad1 = True\n",
    "    load_best_model_at_end_ad1 = load_best_model_at_end\n",
    "    metric_for_best_model_ad1 = metric_for_best_model\n",
    "    adapter_config_name_ad1 = adapter_config_name.replace('+inv','')\n",
    "# without LANG adapter\n",
    "else:\n",
    "    learning_rate_ad1 = 1e-4\n",
    "    batch_size_ad1 = 16\n",
    "    gradient_accumulation_steps_ad1 = 1\n",
    "    adam_epsilon_ad1 = 1e-8\n",
    "    num_train_epochs_ad1 = 15.0\n",
    "    early_stopping_patience_ad1 = 5\n",
    "    with_adapters_mlm_ad1 = with_adapters_mlm\n",
    "    madx2_ad1 = madx2\n",
    "    houlsby_MHA_lastlayer_ad1 = houlsby_MHA_lastlayer\n",
    "    ds_ad1 = False\n",
    "    fp16_ad1 = True\n",
    "    load_best_model_at_end_ad1 = load_best_model_at_end\n",
    "    metric_for_best_model_ad1 = 'eval_loss'\n",
    "    adapter_config_name_ad1 = adapter_config_name.replace('+inv','')\n",
    "\n",
    "outputs_ad1 = outputs_ad1 \\\n",
    "+ 'lr' + str(learning_rate_ad1) \\\n",
    "+ '_bs' + str(batch_size_ad1) \\\n",
    "+ '_GAS' + str(gradient_accumulation_steps_ad1) \\\n",
    "+ '_eps' + str(adam_epsilon_ad1) \\\n",
    "+ '_epochs' + str(num_train_epochs_ad1) \\\n",
    "+ '_patience' + str(early_stopping_patience_ad1) \\\n",
    "+ '_wamlm' + str(with_adapters_mlm_ad1) \\\n",
    "+ '_madx2' + str(madx2_ad1) \\\n",
    "+ '_houlsby_MHA_lastlayer' + str(houlsby_MHA_lastlayer_ad1) \\\n",
    "+ '_ds' + str(ds_ad1) \\\n",
    "+ '_fp16' + str(fp16_ad1) \\\n",
    "+ '_best' + str(load_best_model_at_end_ad1) \\\n",
    "+ '_metric' + str(metric_for_best_model_ad1) \\\n",
    "+ '_adapterconfig' + str(adapter_config_name_ad1)\n",
    "\n",
    "# path to outputs\n",
    "path_to_outputs_ad1 = root/'outputs'/outputs_ad1\n",
    "\n",
    "path_to_save_adapter_ad1 = path_to_outputs_ad1/'adapters-qa'\n",
    "\n",
    "load_qa_adapter = str(path_to_save_adapter_ad1)\n",
    "qa_adapter_config = str(path_to_save_adapter_ad1) + \"/adapter_config.json\"\n",
    "qa_adapter = \"pfeiffer\"\n",
    "qa_adapter_non_linearity = 'gelu'\n",
    "qa_adapter_reduction_factor = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NER Adapter Arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# folder for training outputs\n",
    "\n",
    "outputs_ad2 = model_checkpoint.replace('/','-') + '_' + dataset_name_ad2 + '/' + str(task_ad2) + '/'\n",
    "    \n",
    "learning_rate_ad2 = 1e-4\n",
    "batch_size_ad2 = 4\n",
    "gradient_accumulation_steps_ad2 = 2\n",
    "adam_epsilon_ad2 = 1e-4\n",
    "num_train_epochs_ad2 = 30.0\n",
    "early_stopping_patience_ad2 = 10\n",
    "with_adapters_mlm_ad2 = False\n",
    "madx2_ad2 = madx2\n",
    "houlsby_MHA_lastlayer_ad2 = houlsby_MHA_lastlayer\n",
    "ds_ad2 = False\n",
    "fp16_ad2 = True\n",
    "load_best_model_at_end_ad2 = load_best_model_at_end\n",
    "metric_for_best_model_ad2 = 'f1'\n",
    "weighted_loss_ad2 = True\n",
    "if not weighted_loss_ad2:\n",
    "    c_ad2 = 0.0\n",
    "else:\n",
    "    c_ad2 = 0.5\n",
    "next_sentence_ad2 = True\n",
    "adapter_config_name_ad2 = adapter_config_name.replace('+inv','')\n",
    "\n",
    "outputs_ad2 = outputs_ad2 \\\n",
    "+ 'lr' + str(learning_rate_ad2) \\\n",
    "+ '_bs' + str(batch_size_ad2) \\\n",
    "+ '_GAS' + str(gradient_accumulation_steps_ad2) \\\n",
    "+ '_eps' + str(adam_epsilon_ad2) \\\n",
    "+ '_epochs' + str(num_train_epochs_ad2) \\\n",
    "+ '_patience' + str(early_stopping_patience_ad2) \\\n",
    "+ '_wamlm' + str(with_adapters_mlm_ad2) \\\n",
    "+ '_madx2' + str(madx2_ad2) \\\n",
    "+ '_houlsby_MHA_lastlayer' + str(houlsby_MHA_lastlayer_ad2) \\\n",
    "+ '_ds' + str(ds_ad2) \\\n",
    "+ '_fp16' + str(fp16_ad2) \\\n",
    "+ '_best' + str(load_best_model_at_end_ad2) \\\n",
    "+ '_metric' + str(metric_for_best_model_ad2) \\\n",
    "+ '_weightedloss' + str(weighted_loss_ad2) \\\n",
    "+ '_c' + str(c_ad2) \\\n",
    "+ '_ns' + str(next_sentence_ad2) \\\n",
    "+ '_adapterconfig' + str(adapter_config_name_ad2)\n",
    "\n",
    "# path to outputs\n",
    "path_to_outputs_ad2 = root/'outputs'/outputs_ad2\n",
    "\n",
    "path_to_save_adapter_ad2 = path_to_outputs_ad2/'adapters-ner'\n",
    "\n",
    "load_ner_adapter = str(path_to_save_adapter_ad2)\n",
    "ner_adapter_config = str(path_to_save_adapter_ad2) + \"/adapter_config.json\"\n",
    "ner_adapter = \"pfeiffer\"\n",
    "ner_adapter_non_linearity = 'gelu'\n",
    "ner_adapter_reduction_factor = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training arguments of the HF trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup the training argument\n",
    "do_train = True # False\n",
    "do_eval = True \n",
    "\n",
    "# if you want to test the trainer, set up the following variables\n",
    "max_train_samples = 200 # None\n",
    "max_eval_samples = 50 # None\n",
    "\n",
    "# epochs, bs, GA\n",
    "evaluation_strategy = \"epoch\" # no\n",
    "\n",
    "# fp16\n",
    "fp16_opt_level = 'O1'\n",
    "fp16_backend = \"auto\"\n",
    "fp16_full_eval = False\n",
    "\n",
    "# optimizer (AdamW)\n",
    "weight_decay = 0.01 # 0.0\n",
    "adam_beta1 = 0.9\n",
    "adam_beta2 = 0.999\n",
    "\n",
    "# scheduler\n",
    "lr_scheduler_type = 'linear'\n",
    "warmup_ratio = 0.0\n",
    "warmup_steps = 0\n",
    "\n",
    "# logs\n",
    "logging_strategy = \"steps\"\n",
    "logging_first_step = True # False\n",
    "logging_steps = 500     # if strategy = \"steps\"\n",
    "eval_steps = logging_steps # logging_steps\n",
    "\n",
    "# checkpoints\n",
    "save_strategy = \"epoch\" # steps\n",
    "save_steps = 500 # if save_strategy = \"steps\"\n",
    "save_total_limit = 1 # None\n",
    "\n",
    "# no cuda, seed\n",
    "no_cuda = False\n",
    "seed = 42\n",
    "\n",
    "# bar\n",
    "disable_tqdm = False # True\n",
    "remove_unused_columns = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# folder for training outputs\n",
    "\n",
    "outputs = model_checkpoint.replace('/','-') + '_' + dataset_name\n",
    "\n",
    "if with_adapters_mlm:\n",
    "    outputs = outputs + '/' + 'mlm_' + str(task) + '_AdCompo' + str(adapter_composition) + '/'\n",
    "else:\n",
    "    outputs = outputs + '/' + str(task) + '/'\n",
    "\n",
    "outputs = outputs \\\n",
    "+ 'lr' + str(learning_rate) \\\n",
    "+ '_bs' + str(batch_size) \\\n",
    "+ '_GAS' + str(gradient_accumulation_steps) \\\n",
    "+ '_eps' + str(adam_epsilon) \\\n",
    "+ '_epochs' + str(num_train_epochs) \\\n",
    "+ '_patience' + str(early_stopping_patience) \\\n",
    "+ '_wamlm' + str(with_adapters_mlm) \\\n",
    "+ '_madx2' + str(madx2) \\\n",
    "+ '_houlsby_MHA_lastlayer' + str(houlsby_MHA_lastlayer) \\\n",
    "+ '_ds' + str(ds) \\\n",
    "+ '_fp16' + str(fp16) \\\n",
    "+ '_best' + str(load_best_model_at_end) \\\n",
    "+ '_metric' + str(metric_for_best_model) \\\n",
    "+ '_adapterconfig' + str(adapter_config_name) \\\n",
    "+ '_ad1' + str(task_ad1) + dataset_name_ad1 \\\n",
    "+ '_ad2' + str(task_ad2) + dataset_name_ad2\n",
    "\n",
    "# path to outputs\n",
    "path_to_outputs = root/'outputs'/outputs\n",
    "\n",
    "# subfolder for model outputs\n",
    "output_dir = path_to_outputs/'output_dir' \n",
    "overwrite_output_dir = True # False\n",
    "\n",
    "# logs\n",
    "logging_dir = path_to_outputs/'logging_dir'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1r_n9OWV3l-Q"
   },
   "source": [
    "## 6. Preparing the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kswRMhPc3l-Q"
   },
   "source": [
    "For each of those tasks, we will use the [Wikitext 2]() dataset as an example. You can load it very easily with the 🤗 Datasets library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%time\n",
    "# if dataset_name == \"squad11pt\":\n",
    "    \n",
    "#     # create dataset folder \n",
    "#     path_to_dataset = root.parent/'data'/dataset_name\n",
    "#     path_to_dataset.mkdir(parents=True, exist_ok=True) \n",
    "\n",
    "#     # Get dataset SQUAD in Portuguese\n",
    "#     %cd {path_to_dataset}\n",
    "#     !wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn\" -O squad-pt.tar.gz && rm -rf /tmp/cookies.txt\n",
    "\n",
    "#     # unzip \n",
    "#     !tar -xvf squad-pt.tar.gz\n",
    "\n",
    "#     # Get the train and validation json file in the HF script format \n",
    "#     # inspiration: file squad.py at https://github.com/huggingface/datasets/tree/master/datasets/squad\n",
    "\n",
    "#     import json \n",
    "#     files = ['squad-train-v1.1.json','squad-dev-v1.1.json']\n",
    "\n",
    "#     for file in files:\n",
    "\n",
    "#         # Opening JSON file & returns JSON object as a dictionary \n",
    "#         f = open(file, encoding=\"utf-8\") \n",
    "#         data = json.load(f) \n",
    "\n",
    "#         # Iterating through the json list \n",
    "#         entry_list = list()\n",
    "#         id_list = list()\n",
    "\n",
    "#         for row in data['data']: \n",
    "#             title = row['title']\n",
    "\n",
    "#             for paragraph in row['paragraphs']:\n",
    "#                 context = paragraph['context']\n",
    "\n",
    "#                 for qa in paragraph['qas']:\n",
    "#                     entry = {}\n",
    "\n",
    "#                     qa_id = qa['id']\n",
    "#                     question = qa['question']\n",
    "#                     answers = qa['answers']\n",
    "\n",
    "#                     entry['id'] = qa_id\n",
    "#                     entry['title'] = title.strip()\n",
    "#                     entry['context'] = context.strip()\n",
    "#                     entry['question'] = question.strip()\n",
    "\n",
    "#                     answer_starts = [answer[\"answer_start\"] for answer in answers]\n",
    "#                     answer_texts = [answer[\"text\"].strip() for answer in answers]\n",
    "#                     entry['answers'] = {}\n",
    "#                     entry['answers']['answer_start'] = answer_starts\n",
    "#                     entry['answers']['text'] = answer_texts\n",
    "\n",
    "#                     entry_list.append(entry)\n",
    "\n",
    "#         reverse_entry_list = entry_list[::-1]\n",
    "\n",
    "#         # for entries with same id, keep only last one (corrected texts by the group Deep Learning Brasil)\n",
    "#         unique_ids_list = list()\n",
    "#         unique_entry_list = list()\n",
    "#         for entry in reverse_entry_list:\n",
    "#             qa_id = entry['id']\n",
    "#             if qa_id not in unique_ids_list:\n",
    "#                 unique_ids_list.append(qa_id)\n",
    "#                 unique_entry_list.append(entry)\n",
    "\n",
    "#         # Closing file \n",
    "#         f.close() \n",
    "\n",
    "#         new_dict = {}\n",
    "#         new_dict['data'] = unique_entry_list\n",
    "\n",
    "#         file_name = 'pt_' + str(file)\n",
    "#         with open(file_name, 'w') as json_file:\n",
    "#             json.dump(new_dict, json_file)\n",
    "            \n",
    "# %cd {root}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "n2ZRs1cL3l-R",
    "outputId": "11151c56-be90-4d11-e7df-db85e745ca5c"
   },
   "outputs": [],
   "source": [
    "# from datasets import load_dataset\n",
    "# datasets = load_dataset('wikitext', 'wikitext-2-raw-v1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f1-9jepM3l-W"
   },
   "source": [
    "You can replace the dataset above with any dataset hosted on [the hub](https://huggingface.co/datasets) or use your own files. Just uncomment the following cell and replace the paths with values that will lead to your files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "uxSaGa_l3l-W"
   },
   "outputs": [],
   "source": [
    "# datasets = load_dataset(\"text\", data_files={\"train\": path_to_train.txt, \"validation\": path_to_validation.txt}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jY1SwIrY3l-a"
   },
   "source": [
    "You can also load datasets from a csv or a JSON file, see the [full documentation](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u3EtYfeHIrIz"
   },
   "source": [
    "To access an actual element, you need to select a split first, then give an index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset, load_metric\n",
    "\n",
    "if dataset_name == \"squad11pt\":\n",
    "    \n",
    "    # dataset folder \n",
    "    root = Path.cwd()\n",
    "    path_to_dataset = root.parent/'data'/dataset_name\n",
    "    \n",
    "    # paths to files\n",
    "    train_file = str(path_to_dataset/'pt_squad-train-v1.1.json')\n",
    "    validation_file = str(path_to_dataset/'pt_squad-dev-v1.1.json')\n",
    "    \n",
    "    datasets = load_dataset('json', \n",
    "                            data_files={'train': train_file, \\\n",
    "                                        'validation': validation_file, \\\n",
    "                                       }, \n",
    "                            field='data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "X6HrpprwIrIz",
    "outputId": "d7670bc0-42e4-4c09-8a6a-5c018ded7d95"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '5735c47ae853931400426b64',\n",
       " 'title': 'Kathmandu',\n",
       " 'context': 'A maioria das cozinhas encontradas em Katmandu não é vegetariana. No entanto, a prática do vegetarianismo não é incomum, e a culinária vegetariana pode ser encontrada em toda a cidade. O consumo de carne bovina é muito incomum e considerado tabu em muitos lugares. Buff (carne de búfalo Marinho) é muito comum. Há uma forte tradição de consumo de buffs em Katmandu, especialmente entre Newars, que não é encontrado em outras partes do Nepal. O consumo de carne de porco era considerado tabu até algumas décadas atrás. Devido à mistura com a cozinha Kirat do leste do Nepal, a carne de porco encontrou um lugar nos pratos de Katmandu. Uma população marginal de hindus e muçulmanos devotos o considera tabu. Os muçulmanos proíbem comer buff a partir do Alcorão, enquanto os hindus comem todas as variedades, exceto a carne de vaca, pois consideram a vaca uma deusa e símbolo da pureza. O café da manhã principal para moradores e visitantes é principalmente Momo ou Chowmein.',\n",
       " 'question': 'De que animal vem o lustre?',\n",
       " 'answers': {'answer_start': [280], 'text': ['búfalo Marinho']}}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets[\"train\"][10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WHUmphG3IrI3"
   },
   "source": [
    "To get a sense of what the data looks like, the following function will show some examples picked randomly in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "ur5sNUcZ3l-g"
   },
   "outputs": [],
   "source": [
    "from datasets import ClassLabel, Sequence\n",
    "import random\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "def show_random_elements(dataset, num_examples=10):\n",
    "    assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
    "    picks = []\n",
    "    for _ in range(num_examples):\n",
    "        pick = random.randint(0, len(dataset)-1)\n",
    "        while pick in picks:\n",
    "            pick = random.randint(0, len(dataset)-1)\n",
    "        picks.append(pick)\n",
    "    \n",
    "    df = pd.DataFrame(dataset[picks])\n",
    "    for column, typ in dataset.features.items():\n",
    "        if isinstance(typ, ClassLabel):\n",
    "            df[column] = df[column].transform(lambda i: typ.names[i])\n",
    "        elif isinstance(typ, Sequence) and isinstance(typ.feature, ClassLabel):\n",
    "            df[column] = df[column].transform(lambda x: [typ.feature.names[i] for i in x])\n",
    "    display(HTML(df.to_html()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "1Uk8NROQ3l-k",
    "outputId": "a822dcec-51e3-4dba-c73c-dba9e0301726"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>answers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>57274fac5951b619008f8815</td>\n",
       "      <td>Late_Middle_Ages</td>\n",
       "      <td>A escola de pensamento predominante no século XIII foi a reconciliação tomística dos ensinamentos de Aristóteles com a teologia cristã. A condenação de 1277, promulgada na Universidade de Paris, impôs restrições a idéias que poderiam ser interpretadas como heréticas; restrições que tiveram implicações no pensamento aristotélico. Uma alternativa foi apresentada por Guilherme de Ockham, que insistia que o mundo da razão e o mundo da fé deviam ser mantidos separados. Ockham introduziu o princípio da parcimônia - ou a navalha de Occam -, segundo a qual uma teoria simples é preferida a uma mais complexa, e a especulação sobre fenômenos não observáveis é evitada.</td>\n",
       "      <td>Qual filósofo acreditava que razão e fé deveriam ser mantidas separadas?</td>\n",
       "      <td>{'answer_start': [367], 'text': ['Guilherme de Ockham']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>572a85fc111d821400f38bb9</td>\n",
       "      <td>Miami</td>\n",
       "      <td>O sistema rodoviário de Miami é baseado no \"Miami Grid\" numérico, onde a Flagler Street forma a linha de base leste-oeste e a Miami Avenue forma o meridiano norte-sul. A esquina da Flagler Street e Miami Avenue fica no meio do centro da cidade, em frente ao Downtown Macy's (anteriormente a sede do Burdine). A grade de Miami é principalmente numérica, de modo que, por exemplo, todos os endereços ao norte da Flagler Street e a oeste da Miami Avenue têm \"NW\" em seu endereço. Como seu ponto de origem é no Downtown, que fica próximo à costa, os quadrantes \"NW\" e \"SW\" são muito maiores que os quadrantes \"SE\" e \"NE\". Muitas estradas, especialmente as principais, também são nomeadas (por exemplo, Tamiami Trail / SW 8th St), embora, com exceções, o número seja mais comum entre os habitantes locais.</td>\n",
       "      <td>Como é chamado o plano de rua de Miami?</td>\n",
       "      <td>{'answer_start': [44], 'text': ['Miami Grid']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>572879b54b864d1900164a22</td>\n",
       "      <td>History_of_India</td>\n",
       "      <td>Ronald Inden escreve que, no século VIII dC, os símbolos dos deuses hindus \"substituíram o Buda no centro imperial e no auge do sistema político-cosmo, a imagem ou o símbolo do deus hindu passa a ser alojado em um templo monumental e é cada vez mais elaborado. adoração ao puja em estilo imperial \". Embora o budismo não tenha desaparecido da Índia por vários séculos após o oitavo, as tendências reais pelos cultos de Vishnu e Shiva enfraqueceram a posição do budismo dentro do contexto sociopolítico e ajudaram a tornar possível seu declínio.</td>\n",
       "      <td>Com que dentadura o budismo desapareceu da maior parte da Índia?</td>\n",
       "      <td>{'answer_start': [375], 'text': ['oitavo']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5725dbef271a42140099d2af</td>\n",
       "      <td>Montevideo</td>\n",
       "      <td>O Museu Nacional de História de Montevidéu está localizado na residência histórica do general Fructuoso Rivera. Ele exibe artefatos relacionados à história do Uruguai. Em um processo iniciado em 1998, o Museu Nacional de História Natural (1837) e o Museu Nacional de Antropologia (1981), fundiram-se em 2001, tornando-se o Museu Nacional de História Natural e Antropologia. Em julho de 2009, as duas instituições tornaram-se novamente independentes. O Museu Histórico anexou oito casas históricas na cidade, cinco das quais estão localizadas na Ciudad Vieja. Um deles, no mesmo quarteirão do edifício principal, é a residência histórica de Antonio Montero, que abriga o Museo Romantico.</td>\n",
       "      <td>Quantas casas o Museu Histórico anexou?</td>\n",
       "      <td>{'answer_start': [475], 'text': ['oito']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>570e68670dc6ce1900205033</td>\n",
       "      <td>Order_of_the_British_Empire</td>\n",
       "      <td>A Ordem Mais Excelente do Império Britânico é a \"ordem da cavalaria da monarquia constitucional britânica\", recompensando contribuições para as artes e as ciências, trabalhando com organizações de caridade e bem-estar e serviço público fora do Serviço Civil. Foi criada em 4 de junho de 1917 pelo rei George V e compreende cinco classes, nas divisões civil e militar, das quais as duas mais antigas tornam o beneficiário um cavaleiro, se for homem, ou dame, se for mulher. Há também a Medalha do Império Britânico relacionada, cujos destinatários são afiliados, mas não são membros do pedido.</td>\n",
       "      <td>Quando foi estabelecida a ordem de cavalaria da monarquia constitucional britânica?</td>\n",
       "      <td>{'answer_start': [273], 'text': ['4 de junho de 1917']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>56f8b5ef9e9bad19000a037f</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>O maior teatro da cidade é o Teatro Mayflower, com capacidade para 2.300 pessoas (anteriormente conhecido como Gaumont), que, como o maior teatro do sul da Inglaterra nos arredores de Londres, já recebeu espetáculos no West End, como Les Misérables, The Rocky Horror Show e Chitty Chitty Bang Bang, bem como visitas regulares da Ópera Nacional de Gales e do Ballet Nacional Inglês. Há também o Nuffield Theatre, localizado no campus Highfield da Universidade de Southampton, que é o principal teatro de produção da cidade. Foi agraciado com o Prêmio Stage de Melhor Teatro Regional em 2015. Ele também abriga empresas de turismo e sociedades locais (como Southampton Operatic Society, Maskers e University Players).</td>\n",
       "      <td>Qual é o maior teatro de Southampton?</td>\n",
       "      <td>{'answer_start': [29], 'text': ['Teatro Mayflower']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>57307a82069b53140083210e</td>\n",
       "      <td>Translation</td>\n",
       "      <td>Enquanto isso, na Itália renascentista, um novo período na história da tradução se abriu em Florença com a chegada, na corte de Cosimo de 'Medici, do estudioso bizantino Georgius Gemistus Pletho, pouco antes da queda de Constantinopla nos turcos (1453) . Uma tradução para o latim das obras de Platão foi realizada por Marsilio Ficino. Esta e a edição latina do Novo Testamento em Erasmus levaram a uma nova atitude em relação à tradução. Pela primeira vez, os leitores exigiram rigor na renderização, pois as crenças filosóficas e religiosas dependiam das palavras exatas de Platão, Aristóteles e Jesus.</td>\n",
       "      <td>Quem se comprometeu a traduzir as obras de Platão para o latim?</td>\n",
       "      <td>{'answer_start': [319], 'text': ['Marsilio Ficino']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>5730c9a4f6cb411900e244a9</td>\n",
       "      <td>United_States_Air_Force</td>\n",
       "      <td>Assegurar / Dissuadir / Deter é um conjunto de missões derivado da prontidão da Força Aérea em realizar a missão de operações de ataque nuclear, bem como de ações específicas tomadas para garantir aliados como parte de uma dissuasão prolongada. Dissuadir outras pessoas de adquirir ou proliferar armas de destruição em massa, e os meios para entregá-las, contribui para promover a segurança e também é parte integrante desta missão. Além disso, diferentes estratégias de dissuasão são necessárias para dissuadir vários adversários, sejam eles um estado nacional ou um ator não estatal / transnacional. A Força Aérea mantém e apresenta capacidades dissuasivas credíveis por meio de demonstrações e exercícios visíveis bem-sucedidos que garantem aliados, dissuadem a proliferação, impedem adversários em potencial de ações que ameaçam a segurança nacional dos EUA ou as populações e mobilizam forças militares dos EUA, seus aliados e amigos.</td>\n",
       "      <td>O que a Força Aérea dos EUA está preparada para fazer para apoiar seus aliados?</td>\n",
       "      <td>{'answer_start': [95], 'text': ['realizar a missão de operações de ataque nuclear']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>5727f6d83acd2414000df0fa</td>\n",
       "      <td>History_of_science</td>\n",
       "      <td>A disposição de questionar verdades anteriormente contidas e procurar novas respostas resultou em um período de grandes avanços científicos, agora conhecido como Revolução Científica. A Revolução Científica é tradicionalmente considerada pela maioria dos historiadores como tendo começado em 1543, quando os livros De humani corporis fabrica (Sobre o funcionamento do corpo humano), de Andreas Vesalius, e também De Revolutionibus, do astrônomo Nicolaus Copernicus, foram impressos pela primeira vez. A tese do livro de Copérnico era que a Terra se movia ao redor do Sol. O período culminou com a publicação do Philosophiæ Naturalis Principia Mathematica em 1687 por Isaac Newton, representante do crescimento sem precedentes de publicações científicas em toda a Europa.</td>\n",
       "      <td>Que ano é considerado o começo da Revolução Científica?</td>\n",
       "      <td>{'answer_start': [292], 'text': ['1543']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>57282cb83acd2414000df647</td>\n",
       "      <td>London</td>\n",
       "      <td>Outros mamíferos encontrados na Grande Londres são ouriços, ratos, camundongos, coelhos, musaranhos, ratazana e esquilos. Em áreas mais selvagens do Outer London, como Epping Forest, uma grande variedade de mamíferos é encontrada, incluindo lebre, texugo, campo, banco e ratazana de água, rato de madeira, rato de pescoço amarelo, toupeira, musaranho e doninha, além de raposa, esquilo e ouriço. Uma lontra morta foi encontrada na rodovia, em Wapping, a cerca de 1,6 km da Tower Bridge, o que sugere que eles começaram a voltar atrás depois de estarem ausentes cem anos da cidade. Dez das dezoito espécies de morcegos da Inglaterra foram registradas na floresta de Epping: soprano, nathusius e pipistrelles comuns, noctule, serotine, barbastelle, daubenton, orelhudo, natterer e lazer.</td>\n",
       "      <td>Epping Forest é o lar de quantas espécies de morcegos?</td>\n",
       "      <td>{'answer_start': [581], 'text': ['Dez']}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_random_elements(datasets[\"train\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CKerdF353l-o"
   },
   "source": [
    "As we can see, some of the texts are a full paragraph of a Wikipedia article while others are just titles or empty lines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n9qywopnIrJH"
   },
   "source": [
    "## 7. Preprocessing the training data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YVx71GdAIrJH"
   },
   "source": [
    "Before we can feed those texts to our model, we need to preprocess them. This is done by a 🤗 Transformers `Tokenizer` which will (as the name indicates) tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary) and put it in a format the model expects, as well as generate the other inputs that model requires.\n",
    "\n",
    "To do all of this, we instantiate our tokenizer with the `AutoTokenizer.from_pretrained` method, which will ensure:\n",
    "\n",
    "- we get a tokenizer that corresponds to the model architecture we want to use,\n",
    "- we download the vocabulary used when pretraining this specific checkpoint.\n",
    "\n",
    "That vocabulary will be cached, so it's not downloaded again the next time we run the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "eXNLu_-nIrJI"
   },
   "outputs": [],
   "source": [
    "# from transformers import AutoTokenizer\n",
    "    \n",
    "# tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "if model_checkpoint_original == \"neuralmind/bert-large-portuguese-cased\":\n",
    "    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint_original_local)\n",
    "else:\n",
    "    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vl6IidfdIrJK"
   },
   "source": [
    "The following assertion ensures that our tokenizer is a fast tokenizers (backed by Rust) from the 🤗 Tokenizers library. Those fast tokenizers are available for almost all models, and we will need some of the special features they have for our preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import transformers\n",
    "assert isinstance(tokenizer, transformers.PreTrainedTokenizerFast)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check which type of models have a fast tokenizer available and which don't on the [big table of models](https://huggingface.co/transformers/index.html#bigtable)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rowT4iCLIrJK"
   },
   "source": [
    "You can directly call this tokenizer on two sentences (one for the answer, one for the context):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "a5hBlsrHIrJL",
    "outputId": "acdaa98a-a8cd-4a20-89b8-cc26437bbe90"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 4629, 352, 847, 17008, 22282, 12204, 136, 102, 5494, 12204, 847, 16775, 256, 147, 119, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(\"What is your name?\", \"My name is Sylvain.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on the model you selected, you will see different keys in the dictionary returned by the cell above. They don't matter much for what we're doing here (just know they are required by the model we will instantiate later), you can learn more about them in [this tutorial](https://huggingface.co/transformers/preprocessing.html) if you're interested.\n",
    "\n",
    "Now one specific thing for the preprocessing in question answering is how to deal with very long documents. We usually truncate them in other tasks, when they are longer than the model maximum sentence length, but here, removing part of the the context might result in losing the answer we are looking for. To deal with this, we will allow one (long) example in our dataset to give several input features, each of length shorter than the maximum length of the model (or the one we set as a hyper-parameter). Also, just in case the answer lies at the point we split a long context, we allow some overlap between the features we generate controlled by the hyper-parameter `doc_stride`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_length = 384 # The maximum length of a feature (question and context)\n",
    "doc_stride = 128 # The authorized overlap between two part of the context when splitting it is needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's find one long example in our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, example in enumerate(datasets[\"train\"]):\n",
    "    if len(tokenizer(example[\"question\"], example[\"context\"])[\"input_ids\"]) > 384:\n",
    "        break\n",
    "example = datasets[\"train\"][i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without any truncation, we get the following length for the input IDs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "407"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokenizer(example[\"question\"], example[\"context\"])[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, if we just truncate, we will lose information (and possibly the answer to our question):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "384"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokenizer(example[\"question\"], example[\"context\"], max_length=max_length, truncation=\"only_second\")[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we never want to truncate the question, only the context, else the `only_second` truncation picked. Now, our tokenizer can automatically return us a list of features capped by a certain maximum length, with the overlap we talked above, we just have to tell it with `return_overflowing_tokens=True` and by passing the stride:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_example = tokenizer(\n",
    "    example[\"question\"],\n",
    "    example[\"context\"],\n",
    "    max_length=max_length,\n",
    "    truncation=\"only_second\",\n",
    "    return_overflowing_tokens=True,\n",
    "    stride=doc_stride\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we don't have one list of `input_ids`, but several: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[384, 165]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[len(x) for x in tokenized_example[\"input_ids\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if we decode them, we can see the overlap:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CLS] Em que meses a monção ocorre no Nepal? [SEP] A cidade geralmente tem um clima com dias quentes, seguido de noites e manhãs frias. Prevê - se um clima imprevisível, uma vez que as temperaturas podem cair para 1 ° C ( 34 ° F ) ou menos durante o inverno. Durante uma frente fria de 2013, as temperaturas de inverno de Katmandu caíram para - 4 ° C ( 25 ° F ), e a temperatura mais baixa foi registrada em 10 de janeiro de 2013, a - 9, 2 ° C ( 15, 4 ° F ). As chuvas são principalmente baseadas em monções ( cerca de 65 % do total concentrado durante os meses das monções de Junho a agosto ) e diminuem substancialmente ( 100 a 200 cm ) do leste do Nepal ao oeste do Nepal. As chuvas foram registradas em cerca de 1. 400 milímetros ( 55, 1 polegadas ) no vale de Katmandu e em média 1. 407 milímetros ( 55, 4 polegadas ) na cidade de Katmandu. Em média, a umidade é de 75 %. O gráfico abaixo é baseado em dados do Bureau de Padrões e Meteorologia do Nepal, \" Meteorologia Meteorológica \" de 2005. O gráfico fornece temperaturas mínimas e máximas durante cada mês. A quantidade anual de precipitação foi de 1. 124 milímetros ( 44, 3 polegadas ) em 2005, conforme dados mensais incluídos na tabela acima. A década de 2000 - 2010 viu anomalias de precipitação altamente variáveis e sem precedentes em Katmandu. Isso ocorreu principalmente devido à variação anual das monções do sudoeste. [ Citação necessário ] Por exemplo, 2003 foi o ano mais chuvoso de todos os tempos em Katmandu, totalizando mais de 2. 900 mm ( 114 in ) de precipitação devido a uma estação de monções excepcionalmente forte. Em contraste, 2001 registrou [SEP]\n",
      "[CLS] Em que meses a monção ocorre no Nepal? [SEP] fornece temperaturas mínimas e máximas durante cada mês. A quantidade anual de precipitação foi de 1. 124 milímetros ( 44, 3 polegadas ) em 2005, conforme dados mensais incluídos na tabela acima. A década de 2000 - 2010 viu anomalias de precipitação altamente variáveis e sem precedentes em Katmandu. Isso ocorreu principalmente devido à variação anual das monções do sudoeste. [ Citação necessário ] Por exemplo, 2003 foi o ano mais chuvoso de todos os tempos em Katmandu, totalizando mais de 2. 900 mm ( 114 in ) de precipitação devido a uma estação de monções excepcionalmente forte. Em contraste, 2001 registrou apenas 356 mm ( 14 pol ) de precipitação devido a uma estação de monções extraordinariamente fraca. [SEP]\n"
     ]
    }
   ],
   "source": [
    "for x in tokenized_example[\"input_ids\"][:2]:\n",
    "    print(tokenizer.decode(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now this will give us some work to properly treat the answers: we need to find in which of those features the answer actually is, and where exactly in that feature. The models we will use require the start and end positions of these answers in the tokens, so we will also need to to map parts of the original context to some tokens. Thankfully, the tokenizer we're using can help us with that by returning an `offset_mapping`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0), (0, 2), (3, 6), (7, 12), (13, 14), (15, 18), (18, 21), (22, 28), (29, 31), (32, 34), (34, 37), (37, 38), (0, 0), (0, 1), (2, 8), (9, 19), (20, 23), (24, 26), (27, 32), (33, 36), (37, 41), (42, 49), (49, 50), (51, 58), (59, 61), (62, 68), (69, 70), (71, 76), (76, 77), (78, 82), (82, 83), (83, 84), (85, 88), (88, 90), (90, 91), (91, 93), (94, 96), (97, 102), (103, 108), (108, 115), (115, 116), (117, 120), (121, 124), (125, 128), (129, 131), (132, 144), (145, 150), (151, 155), (156, 160), (161, 162), (163, 164), (165, 166), (167, 168), (168, 170), (171, 172), (173, 174), (174, 175), (176, 178), (179, 184), (185, 192), (193, 194), (195, 202), (202, 203), (204, 211), (212, 215), (216, 222), (223, 227), (228, 230), (231, 235), (235, 236), (237, 239), (240, 252), (253, 255), (256, 263), (264, 266), (267, 270), (270, 274), (274, 275), (276, 282), (283, 287), (288, 289), (289, 290), (291, 292), (293, 294), (295, 296), (296, 298), (299, 300), (301, 302), (302, 303), (303, 304), (305, 306), (307, 308), (309, 320), (321, 325), (326, 331), (332, 335), (336, 346), (347, 349), (350, 352), (353, 355)]\n"
     ]
    }
   ],
   "source": [
    "tokenized_example = tokenizer(\n",
    "    example[\"question\"],\n",
    "    example[\"context\"],\n",
    "    max_length=max_length,\n",
    "    truncation=\"only_second\",\n",
    "    return_overflowing_tokens=True,\n",
    "    return_offsets_mapping=True,\n",
    "    stride=doc_stride\n",
    ")\n",
    "print(tokenized_example[\"offset_mapping\"][0][:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives, for each index of our input IDS, the corresponding start and end character in the original text that gave our token. The very first token (`[CLS]`) has (0, 0) because it doesn't correspond to any part of the question/answer, then the second token is the same as the characters 0 to 3 of the question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Em Em\n"
     ]
    }
   ],
   "source": [
    "first_token_id = tokenized_example[\"input_ids\"][0][1]\n",
    "offsets = tokenized_example[\"offset_mapping\"][0][1]\n",
    "print(tokenizer.convert_ids_to_tokens([first_token_id])[0], example[\"question\"][offsets[0]:offsets[1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can use this mapping to find the position of the start and end tokens of our answer in a given feature. We just have to distinguish which parts of the offsets correspond to the question and which part correspond to the context, this is where the `sequence_ids` method of our `tokenized_example` can be useful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[None, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, None, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, None]\n"
     ]
    }
   ],
   "source": [
    "sequence_ids = tokenized_example.sequence_ids()\n",
    "print(sequence_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It returns `None` for the special tokens, then 0 or 1 depending on whether the corresponding token comes from the first sentence past (the question) or the second (the context). Now with all of this, we can find the first and last token of the answer in one of our input feature (or if the answer is not in this feature):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "143 145\n"
     ]
    }
   ],
   "source": [
    "answers = example[\"answers\"]\n",
    "start_char = answers[\"answer_start\"][0]\n",
    "end_char = start_char + len(answers[\"text\"][0])\n",
    "\n",
    "# Start token index of the current span in the text.\n",
    "token_start_index = 0\n",
    "while sequence_ids[token_start_index] != 1:\n",
    "    token_start_index += 1\n",
    "\n",
    "# End token index of the current span in the text.\n",
    "token_end_index = len(tokenized_example[\"input_ids\"][0]) - 1\n",
    "while sequence_ids[token_end_index] != 1:\n",
    "    token_end_index -= 1\n",
    "\n",
    "# Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).\n",
    "offsets = tokenized_example[\"offset_mapping\"][0]\n",
    "if (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
    "    # Move the token_start_index and token_end_index to the two ends of the answer.\n",
    "    # Note: we could go after the last offset if the answer is the last word (edge case).\n",
    "    while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
    "        token_start_index += 1\n",
    "    start_position = token_start_index - 1\n",
    "    while offsets[token_end_index][1] >= end_char:\n",
    "        token_end_index -= 1\n",
    "    end_position = token_end_index + 1\n",
    "    print(start_position, end_position)\n",
    "else:\n",
    "    print(\"The answer is not in this feature.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can double check that it is indeed the theoretical answer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Junho a agosto\n",
      "Junho a agosto\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.decode(tokenized_example[\"input_ids\"][0][start_position: end_position+1]))\n",
    "print(answers[\"text\"][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this notebook to work with any kind of models, we need to account for the special case where the model expects padding on the left (in which case we switch the order of the question and the context):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "pad_on_right = tokenizer.padding_side == \"right\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's put everything together in one function we will apply to our training set. In the case of impossible answers (the answer is in another feature given by an example with a long context), we set the cls index for both the start and end position. We could also simply discard those examples from the training set if the flag `allow_impossible_answers` is `False`. Since the preprocessing is already complex enough as it is, we've kept is simple for this part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_train_features(examples):\n",
    "    # Tokenize our examples with truncation and padding, but keep the overflows using a stride. This results\n",
    "    # in one example possible giving several features when a context is long, each of those features having a\n",
    "    # context that overlaps a bit the context of the previous feature.\n",
    "    tokenized_examples = tokenizer(\n",
    "        examples[\"question\" if pad_on_right else \"context\"],\n",
    "        examples[\"context\" if pad_on_right else \"question\"],\n",
    "        truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
    "        max_length=max_length,\n",
    "        stride=doc_stride,\n",
    "        return_overflowing_tokens=True,\n",
    "        return_offsets_mapping=True,\n",
    "        padding=\"max_length\",\n",
    "    )\n",
    "\n",
    "    # Since one example might give us several features if it has a long context, we need a map from a feature to\n",
    "    # its corresponding example. This key gives us just that.\n",
    "    sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
    "    # The offset mappings will give us a map from token to character position in the original context. This will\n",
    "    # help us compute the start_positions and end_positions.\n",
    "    offset_mapping = tokenized_examples.pop(\"offset_mapping\")\n",
    "\n",
    "    # Let's label those examples!\n",
    "    tokenized_examples[\"start_positions\"] = []\n",
    "    tokenized_examples[\"end_positions\"] = []\n",
    "\n",
    "    for i, offsets in enumerate(offset_mapping):\n",
    "        # We will label impossible answers with the index of the CLS token.\n",
    "        input_ids = tokenized_examples[\"input_ids\"][i]\n",
    "        cls_index = input_ids.index(tokenizer.cls_token_id)\n",
    "\n",
    "        # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n",
    "        sequence_ids = tokenized_examples.sequence_ids(i)\n",
    "\n",
    "        # One example can give several spans, this is the index of the example containing this span of text.\n",
    "        sample_index = sample_mapping[i]\n",
    "        answers = examples[\"answers\"][sample_index]\n",
    "        # If no answers are given, set the cls_index as answer.\n",
    "        if len(answers[\"answer_start\"]) == 0:\n",
    "            tokenized_examples[\"start_positions\"].append(cls_index)\n",
    "            tokenized_examples[\"end_positions\"].append(cls_index)\n",
    "        else:\n",
    "            # Start/end character index of the answer in the text.\n",
    "            start_char = answers[\"answer_start\"][0]\n",
    "            end_char = start_char + len(answers[\"text\"][0])\n",
    "\n",
    "            # Start token index of the current span in the text.\n",
    "            token_start_index = 0\n",
    "            while sequence_ids[token_start_index] != (1 if pad_on_right else 0):\n",
    "                token_start_index += 1\n",
    "\n",
    "            # End token index of the current span in the text.\n",
    "            token_end_index = len(input_ids) - 1\n",
    "            while sequence_ids[token_end_index] != (1 if pad_on_right else 0):\n",
    "                token_end_index -= 1\n",
    "\n",
    "            # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).\n",
    "            if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
    "                tokenized_examples[\"start_positions\"].append(cls_index)\n",
    "                tokenized_examples[\"end_positions\"].append(cls_index)\n",
    "            else:\n",
    "                # Otherwise move the token_start_index and token_end_index to the two ends of the answer.\n",
    "                # Note: we could go after the last offset if the answer is the last word (edge case).\n",
    "                while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
    "                    token_start_index += 1\n",
    "                tokenized_examples[\"start_positions\"].append(token_start_index - 1)\n",
    "                while offsets[token_end_index][1] >= end_char:\n",
    "                    token_end_index -= 1\n",
    "                tokenized_examples[\"end_positions\"].append(token_end_index + 1)\n",
    "\n",
    "    return tokenized_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0lm8ozrJIrJR"
   },
   "source": [
    "This function works with one or several examples. In the case of several examples, the tokenizer will return a list of lists for each key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "id": "-b70jh26IrJS",
    "outputId": "acd3a42d-985b-44ee-9daa-af5d944ce1d9"
   },
   "outputs": [],
   "source": [
    "features = prepare_train_features(datasets['train'][:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zS-6iXTkIrJT"
   },
   "source": [
    "To apply this function on all the sentences (or pairs of sentences) in our dataset, we just use the `map` method of our `dataset` object we created earlier. This will apply the function on all the elements of all the splits in `dataset`, so our training, validation and testing data will be preprocessed in one single command. Since our preprocessing changes the number of samples, we need to remove the old columns when applying it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_datasets = datasets.map(prepare_train_features, batched=True, remove_columns=datasets[\"train\"].column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "voWiw8C7IrJV"
   },
   "source": [
    "Even better, the results are automatically cached by the 🤗 Datasets library to avoid spending time on this step the next time you run your notebook. The 🤗 Datasets library is normally smart enough to detect when the function you pass to map has changed (and thus requires to not use the cache data). For instance, it will properly detect if you change the task in the first cell and rerun the notebook. 🤗 Datasets warns you when it uses cached files, you can pass `load_from_cache_file=False` in the call to `map` to not use the cached files and force the preprocessing to be applied again.\n",
    "\n",
    "Note that we passed `batched=True` to encode the texts by batches together. This is to leverage the full benefit of the fast tokenizer we loaded earlier, which will use multi-threading to treat the texts in a batch concurrently."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "545PP3o8IrJV"
   },
   "source": [
    "## 8. Fine-tuning the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FBiW8UpKIrJW"
   },
   "source": [
    "Now that our data is ready for training, we can download the pretrained model and fine-tune it. Since our task is question answering, we use the `AutoModelForQuestionAnswering` class. Like with the tokenizer, the `from_pretrained` method will download and cache the model for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "id": "TlqNaB8jIrJW",
    "outputId": "84916cf3-6e6c-47f3-d081-032ec30a4132"
   },
   "outputs": [],
   "source": [
    "# from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer\n",
    "\n",
    "# model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer\n",
    "\n",
    "if model_checkpoint_original == \"neuralmind/bert-large-portuguese-cased\":\n",
    "    model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint_original_local)\n",
    "else:\n",
    "    model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "333348866"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_num_param=0\n",
    "for p in model.parameters():\n",
    "    model_num_param+=p.numel()\n",
    "model_num_param"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CczA5lJlIrJX"
   },
   "source": [
    "The warning is telling us we are throwing away some weights (the `vocab_transform` and `vocab_layer_norm` layers) and randomly initializing some other (the `pre_classifier` and `classifier` layers). This is absolutely normal in this case, because we are removing the head used to pretrain the model on a masked language modeling objective and replacing it with a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Adapters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "if madx2:\n",
    "    leave_out = [len(model.bert.encoder.layer)-1]\n",
    "else:\n",
    "    leave_out = list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have everything set up to load our AdapterFusion setup. First, we load the (best) lang adapter pre-trained on SQuAD 1.1 pt and the 2 adapters QA and NER respectively pre-trained on SQuAd 1.1 pt and on LeNER-Br. As we don't need their prediction heads, we pass `with_head=False` to the loading method. Next, we add a new fusion layer that combines all the adapters we've just loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers.adapters.composition import Fuse\n",
    "\n",
    "task_mlm_load_as = 'mlm'\n",
    "load_as_ad1 = str(task_ad1) + '_' + dataset_name_ad1\n",
    "load_as_ad2 = str(task_ad2) + '_' + dataset_name_ad2\n",
    "\n",
    "# Load the pre-trained adapters we want to fuse\n",
    "if with_adapters_mlm:\n",
    "    model.load_adapter(load_lang_adapter, config=lang_adapter_config, load_as=task_mlm_load_as, with_head=False)\n",
    "model.load_adapter(load_qa_adapter, load_as=load_as_ad1, with_head=False)\n",
    "model.load_adapter(load_ner_adapter, load_as=load_as_ad2, with_head=False)\n",
    "\n",
    "# Add a fusion layer for QA and NER adapters\n",
    "model.add_adapter_fusion(Fuse(load_as_ad1, load_as_ad2))\n",
    "\n",
    "# Put the adapters on the forward ppath\n",
    "if with_adapters_mlm:\n",
    "    model.set_active_adapters([task_mlm_load_as, Fuse(load_as_ad1, load_as_ad2)])\n",
    "else:\n",
    "    model.set_active_adapters(Fuse(load_as_ad1, load_as_ad2))\n",
    "\n",
    "# Unfreeze and activate fusion setup\n",
    "adapter_setup = Fuse(load_as_ad1, load_as_ad2)\n",
    "model.train_adapter_fusion(adapter_setup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "if madx2:\n",
    "    model.bert.encoder.layer[-1].output.adapter_fusion_layer = nn.ModuleDict(dict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertForQuestionAnswering(\n",
       "  (bert): BertModel(\n",
       "    (invertible_adapters): ModuleDict()\n",
       "    (embeddings): BertEmbeddings(\n",
       "      (word_embeddings): Embedding(29794, 1024, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 1024)\n",
       "      (token_type_embeddings): Embedding(2, 1024)\n",
       "      (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "    (encoder): BertEncoder(\n",
       "      (layer): ModuleList(\n",
       "        (0): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (1): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (2): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (3): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (4): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (5): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (6): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (7): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (8): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (9): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (10): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (11): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (12): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (13): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (14): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (15): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (16): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (17): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (18): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (19): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (20): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (21): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (22): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (23): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict()\n",
       "            (adapter_fusion_layer): ModuleDict()\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (qa_outputs): Linear(in_features=1024, out_features=2, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- model_adapter_num_param=0\n",
    "for p in model.parameters():\n",
    "    model_adapter_num_param+=p.numel()\n",
    "model_adapter_num_param\n",
    "\n",
    "wam = False\n",
    "411827074 -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "411827074"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_adapter_num_param=0\n",
    "for p in model.parameters():\n",
    "    model_adapter_num_param+=p.numel()\n",
    "model_adapter_num_param"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of parameters of the model with adapter: 411827074\n",
      "Number of parameters of the model without adapter: 333348866\n",
      "Number of parameters of the adapter: 78478208\n",
      "Pourcentage of additional parameters through adapter: 23.54 %\n"
     ]
    }
   ],
   "source": [
    "print(f\"Number of parameters of the model with adapter: {model_adapter_num_param:.0f}\")\n",
    "print(f\"Number of parameters of the model without adapter: {model_num_param:.0f}\")\n",
    "print(f\"Number of parameters of the adapter: {model_adapter_num_param - model_num_param:.0f}\")\n",
    "print(f\"Pourcentage of additional parameters through adapter:\",round(((model_adapter_num_param - model_num_param)/model_num_param)*100,2),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jElf8LJ33l_K"
   },
   "source": [
    "## 10. Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_N8urzhyIrJY"
   },
   "source": [
    "To instantiate a `Trainer`, we will need to define three more things. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to customize the training. It requires one folder name, which will be used to save the checkpoints of the model, and all other arguments are optional:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "id": "YbSwEhQ63l_L"
   },
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments\n",
    "\n",
    "if ds:\n",
    "    training_args = TrainingArguments(\n",
    "        output_dir=output_dir,\n",
    "        overwrite_output_dir=overwrite_output_dir,\n",
    "        do_train=do_train,\n",
    "        do_eval=do_eval,\n",
    "        evaluation_strategy=evaluation_strategy,\n",
    "        per_device_train_batch_size=batch_size,\n",
    "        per_device_eval_batch_size=batch_size,\n",
    "        gradient_accumulation_steps=gradient_accumulation_steps,\n",
    "        learning_rate=learning_rate,\n",
    "        weight_decay=weight_decay,\n",
    "        adam_beta1=adam_beta1,\n",
    "        adam_beta2=adam_beta2,\n",
    "        adam_epsilon=adam_epsilon,\n",
    "        num_train_epochs=num_train_epochs,\n",
    "        lr_scheduler_type=lr_scheduler_type,\n",
    "        warmup_ratio=warmup_ratio,\n",
    "        warmup_steps=warmup_steps,\n",
    "        logging_dir=logging_dir,         # directory for storing logs\n",
    "        logging_strategy=evaluation_strategy,\n",
    "        logging_steps=logging_steps,     # if strategy = \"steps\"\n",
    "        save_strategy=evaluation_strategy,          # model checkpoint saving strategy\n",
    "        save_steps=logging_steps,        # if strategy = \"steps\"\n",
    "        save_total_limit=save_total_limit,\n",
    "        fp16=fp16,\n",
    "        eval_steps=logging_steps,        # if strategy = \"steps\"\n",
    "        load_best_model_at_end=load_best_model_at_end,\n",
    "        metric_for_best_model=metric_for_best_model,\n",
    "        greater_is_better=greater_is_better,\n",
    "        local_rank=gpu,\n",
    "        deepspeed = ds_config\n",
    "        )\n",
    "else:\n",
    "    training_args = TrainingArguments(\n",
    "        output_dir=output_dir,\n",
    "        overwrite_output_dir=overwrite_output_dir,\n",
    "        do_train=do_train,\n",
    "        do_eval=do_eval,\n",
    "        evaluation_strategy=evaluation_strategy,\n",
    "        per_device_train_batch_size=batch_size,\n",
    "        per_device_eval_batch_size=batch_size,\n",
    "        gradient_accumulation_steps=gradient_accumulation_steps,\n",
    "        learning_rate=learning_rate,\n",
    "        weight_decay=weight_decay,\n",
    "        adam_beta1=adam_beta1,\n",
    "        adam_beta2=adam_beta2,\n",
    "        adam_epsilon=adam_epsilon,\n",
    "        num_train_epochs=num_train_epochs,\n",
    "        lr_scheduler_type=lr_scheduler_type,\n",
    "        warmup_ratio=warmup_ratio,\n",
    "        warmup_steps=warmup_steps,\n",
    "        logging_dir=logging_dir,         # directory for storing logs\n",
    "        logging_strategy=evaluation_strategy,\n",
    "        logging_steps=logging_steps,     # if strategy = \"steps\"\n",
    "        save_strategy=evaluation_strategy,          # model checkpoint saving strategy\n",
    "        save_steps=logging_steps,        # if strategy = \"steps\"\n",
    "        save_total_limit=save_total_limit,\n",
    "        fp16=fp16,\n",
    "        eval_steps=logging_steps,        # if strategy = \"steps\"\n",
    "        load_best_model_at_end=load_best_model_at_end,\n",
    "        metric_for_best_model=metric_for_best_model,\n",
    "        greater_is_better=greater_is_better,\n",
    "        local_rank=gpu,\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z6uuUnvz3l_b"
   },
   "source": [
    "And second, we use a special `data_collator`. The `data_collator` is a function that is responsible of taking the samples and batching them in tensors. In the previous example, we had nothing special to do, so we just used the default for this argument. Here we want to do the random-masking. We could do it as a pre-processing step (like the tokenization) but then the tokens would always be masked the same way at each epoch. By doing this step inside the `data_collator`, we ensure this random masking is done in a new way each time we go over the data.\n",
    "\n",
    "To do this masking for us, the library provides a `DataCollatorForLanguageModeling`. We can adjust the probability of the masking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "id": "nRZ-5v_P3l_b"
   },
   "outputs": [],
   "source": [
    "from transformers import default_data_collator\n",
    "\n",
    "data_collator = default_data_collator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bqHnWcYC3l_d"
   },
   "source": [
    "Then we just have to pass everything to `Trainer` and begin training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "id": "V-Y3gNqV3l_d"
   },
   "outputs": [],
   "source": [
    "from transformers.trainer_callback import EarlyStoppingCallback\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_datasets[\"train\"],\n",
    "    eval_dataset=tokenized_datasets[\"validation\"],\n",
    "    tokenizer=tokenizer,\n",
    "    data_collator=data_collator,\n",
    "#         do_save_full_model=train_adapter, # save full model as we finetuned head + embeddings\n",
    "    do_save_full_model=not train_adapter, # save full model as we finetuned head + embeddings\n",
    "    do_save_adapters=train_adapter,\n",
    "    callbacks=[EarlyStoppingCallback(early_stopping_patience=early_stopping_patience)],\n",
    "    )    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check weights dtype\n",
    "trainer.model.bert.embeddings.word_embeddings.weight.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 277M\r\n",
      "-rw-rw-r-- 1 pierre pierre  406 Aug  1 05:44 adapter_fusion_config.json\r\n",
      "-rw-rw-r-- 1 pierre pierre  225 Aug  1 05:44 head_config.json\r\n",
      "drwxrwxr-x 2 pierre pierre 4.0K Aug  1 05:44 ner_lenerbr\r\n",
      "-rw-rw-r-- 1 pierre pierre 277M Aug  1 05:44 pytorch_model_adapter_fusion.bin\r\n",
      "-rw-rw-r-- 1 pierre pierre 9.0K Aug  1 05:44 pytorch_model_head.bin\r\n",
      "drwxrwxr-x 2 pierre pierre 4.0K Aug  1 05:44 qa_squad11pt\r\n"
     ]
    }
   ],
   "source": [
    "# save adapter + head\n",
    "adapters_folder = 'adapters-' + task_ad1\n",
    "path_to_save_adapter = path_to_outputs/adapters_folder\n",
    "load_as_fusion = load_as_ad1 + ',' + load_as_ad2\n",
    "\n",
    "trainer.model.save_adapter_fusion(path_to_save_adapter, load_as_fusion)\n",
    "trainer.model.save_all_adapters(path_to_save_adapter)\n",
    "trainer.model.save_head(path_to_save_adapter)\n",
    "\n",
    "!ls -lh {path_to_save_adapter}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KDBi0reX3l_g"
   },
   "source": [
    "Like before, we can evaluate our model on the validation set. The perplexity is much lower than for the CLM objective because for the MLM objective, we only have to make predictions for the masked tokens (which represent 15% of the total here) while having access to the rest of the tokens. It's thus an easier task for the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "eval_results = trainer.evaluate()\n",
    "eval_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. TensorBoard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "PATH = os.getenv('PATH')\n",
    "# repalce xxxx by username\n",
    "# replace yyyy by virtual ambiente name\n",
    "%env PATH=/mnt/home/xxxx/anaconda3/envs/yyyy/bin:$PATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext tensorboard\n",
    "# %reload_ext tensorboard\n",
    "%tensorboard --logdir {logging_dir} --bind_all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluating our model will require a bit more work, as we will need to map the predictions of our model back to parts of the context. The model itself predicts logits for the start and en position of our answers: if we take a batch from our validation datalaoder, here is the output our model gives us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "odict_keys(['loss', 'start_logits', 'end_logits'])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "for batch in trainer.get_eval_dataloader():\n",
    "    break\n",
    "batch = {k: v.to(trainer.args.device) for k, v in batch.items()}\n",
    "with torch.no_grad():\n",
    "    output = trainer.model(**batch)\n",
    "output.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the model is a dict-like object that contains the loss (since we provided labels), the start and end logits. We won't need the loss for our predictions, let's have a look a the logits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([16, 384]), torch.Size([16, 384]))"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.start_logits.shape, output.end_logits.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have one logit for each feature and each token. The most obvious thing to predict an answer for each featyre is to take the index for the maximum of the start logits as a start position and the index of the maximum of the end logits as an end position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 35,  80, 105,  77, 183,   0,  22,   0,  40, 129,  47,  53,  53,  39,\n",
       "          39,  76], device='cuda:1'),\n",
       " tensor([ 38,  81, 107,  80, 184,   0,  22,   0,  42, 149,  52,  60,  60,  51,\n",
       "          51, 105], device='cuda:1'))"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.start_logits.argmax(dim=-1), output.end_logits.argmax(dim=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will work great in a lot of cases, but what if this prediction gives us something impossible: the start position could be greater than the end position, or point to a span of text in the question instead of the answer. In that case, we might want to look at the second best prediction to see if it gives a possible answer and select that instead.\n",
    "\n",
    "However, picking the second best answer is not as easy as picking the best one: is it the second best index in the start logits with the best index in the end logits? Or the best index in the start logits with the second best index in the end logits? And if that second best answer is not possible either, it gets even trickier for the third best answer.\n",
    "\n",
    "\n",
    "To classify our answers, we will use the score obtained by adding the start and end logits. We won't try to order all the possible answers and limit ourselves to with a hyper-parameter we call `n_best_size`. We'll pick the best indices in the start and end logits and gather all the answers this predicts. After checking if each one is valid, we will sort them by their score and keep the best one. Here is how we would do this on the first feature in the batch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_best_size = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "start_logits = output.start_logits[0].cpu().numpy()\n",
    "end_logits = output.end_logits[0].cpu().numpy()\n",
    "# Gather the indices the best start/end logits:\n",
    "start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "valid_answers = []\n",
    "for start_index in start_indexes:\n",
    "    for end_index in end_indexes:\n",
    "        if start_index <= end_index: # We need to refine that test to check the answer is inside the context\n",
    "            valid_answers.append(\n",
    "                {\n",
    "                    \"score\": start_logits[start_index] + end_logits[end_index],\n",
    "                    \"text\": \"\" # We need to find a way to get back the original substring corresponding to the answer in the context\n",
    "                }\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we can sort the `valid_answers` according to their `score` and only keep the best one. The only point left is how to check a given span is inside the context (and not the question) and how to get back the text inside. To do this, we need to add two things to our validation features:\n",
    "- the ID of the example that generated the feature (since each example can generate several features, as seen before);\n",
    "- the offset mapping that will give us a map from token indices to character positions in the context.\n",
    "\n",
    "That's why we will re-process the validation set with the following function, slightly different from `prepare_train_features`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_validation_features(examples):\n",
    "    # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results\n",
    "    # in one example possible giving several features when a context is long, each of those features having a\n",
    "    # context that overlaps a bit the context of the previous feature.\n",
    "    tokenized_examples = tokenizer(\n",
    "        examples[\"question\" if pad_on_right else \"context\"],\n",
    "        examples[\"context\" if pad_on_right else \"question\"],\n",
    "        truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
    "        max_length=max_length,\n",
    "        stride=doc_stride,\n",
    "        return_overflowing_tokens=True,\n",
    "        return_offsets_mapping=True,\n",
    "        padding=\"max_length\",\n",
    "    )\n",
    "\n",
    "    # Since one example might give us several features if it has a long context, we need a map from a feature to\n",
    "    # its corresponding example. This key gives us just that.\n",
    "    sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
    "\n",
    "    # We keep the example_id that gave us this feature and we will store the offset mappings.\n",
    "    tokenized_examples[\"example_id\"] = []\n",
    "\n",
    "    for i in range(len(tokenized_examples[\"input_ids\"])):\n",
    "        # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n",
    "        sequence_ids = tokenized_examples.sequence_ids(i)\n",
    "        context_index = 1 if pad_on_right else 0\n",
    "\n",
    "        # One example can give several spans, this is the index of the example containing this span of text.\n",
    "        sample_index = sample_mapping[i]\n",
    "        tokenized_examples[\"example_id\"].append(examples[\"id\"][sample_index])\n",
    "\n",
    "        # Set to None the offset_mapping that are not part of the context so it's easy to determine if a token\n",
    "        # position is part of the context or not.\n",
    "        tokenized_examples[\"offset_mapping\"][i] = [\n",
    "            (o if sequence_ids[k] == context_index else None)\n",
    "            for k, o in enumerate(tokenized_examples[\"offset_mapping\"][i])\n",
    "        ]\n",
    "\n",
    "    return tokenized_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And like before, we can apply that function to our validation set easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_features = datasets[\"validation\"].map(\n",
    "    prepare_validation_features,\n",
    "    batched=True,\n",
    "    remove_columns=datasets[\"validation\"].column_names\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can grab the predictions for all features by using the `Trainer.predict` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_predictions = trainer.predict(validation_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Trainer` *hides* the columns that are not used by the model (here `example_id` and `offset_mapping` which we will need for our post-processing), so we set them back:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_features.set_format(type=validation_features.format[\"type\"], columns=list(validation_features.features.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now refine the test we had before: since we set `None` in the offset mappings when it corresponds to a part of the question, it's easy to check if an answer is fully inside the context. We also eliminate very long answers from our considerations (with an hyper-parameter we can tune)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_answer_length = 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_logits = output.start_logits[0].cpu().numpy()\n",
    "end_logits = output.end_logits[0].cpu().numpy()\n",
    "offset_mapping = validation_features[0][\"offset_mapping\"]\n",
    "# The first feature comes from the first example. For the more general case, we will need to be match the example_id to\n",
    "# an example index\n",
    "context = datasets[\"validation\"][0][\"context\"]\n",
    "\n",
    "# Gather the indices the best start/end logits:\n",
    "start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "valid_answers = []\n",
    "for start_index in start_indexes:\n",
    "    for end_index in end_indexes:\n",
    "        # Don't consider out-of-scope answers, either because the indices are out of bounds or correspond\n",
    "        # to part of the input_ids that are not in the context.\n",
    "        if (\n",
    "            start_index >= len(offset_mapping)\n",
    "            or end_index >= len(offset_mapping)\n",
    "            or offset_mapping[start_index] is None\n",
    "            or offset_mapping[end_index] is None\n",
    "        ):\n",
    "            continue\n",
    "        # Don't consider answers with a length that is either < 0 or > max_answer_length.\n",
    "        if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n",
    "            continue\n",
    "        if start_index <= end_index: # We need to refine that test to check the answer is inside the context\n",
    "            start_char = offset_mapping[start_index][0]\n",
    "            end_char = offset_mapping[end_index][1]\n",
    "            valid_answers.append(\n",
    "                {\n",
    "                    \"score\": start_logits[start_index] + end_logits[end_index],\n",
    "                    \"text\": context[start_char: end_char]\n",
    "                }\n",
    "            )\n",
    "\n",
    "valid_answers = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[:n_best_size]\n",
    "valid_answers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compare to the actual ground-truth answer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets[\"validation\"][0][\"answers\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model picked the right as the most likely answer!\n",
    "\n",
    "As we mentioned in the code above, this was easy on the first feature because we knew it comes from the first example. For the other features, we will need a map between examples and their corresponding features. Also, since one example can give several features, we will need to gather together all the answers in all the features generated by a given example, then pick the best one. The following code builds a map from example index to its corresponding features indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "\n",
    "examples = datasets[\"validation\"]\n",
    "features = validation_features\n",
    "\n",
    "example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n",
    "features_per_example = collections.defaultdict(list)\n",
    "for i, feature in enumerate(features):\n",
    "    features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're almost ready for our post-processing function. The last bit to deal with is the impossible answer (when `squad_v2 = True`). The code above only keeps answers that are inside the context, we need to also grab the score for the impossible answer (which has start and end indices corresponding to the index of the CLS token). When one example gives several features, we have to predict the impossible answer when all the features give a high score to the impossible answer (since one feature could predict the impossible answer just because the answer isn't in the part of the context it has access too), which is why the score of the impossible answer for one example is the *minimum* of the scores for the impossible answer in each feature generated by the example.\n",
    "\n",
    "We then predict the impossible answer when that score is greater than the score of the best non-impossible answer. All combined together, this gives us this post-processing function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm\n",
    "\n",
    "def postprocess_qa_predictions(examples, features, raw_predictions, n_best_size = 20, max_answer_length = 30):\n",
    "    all_start_logits, all_end_logits = raw_predictions\n",
    "    # Build a map example to its corresponding features.\n",
    "    example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n",
    "    features_per_example = collections.defaultdict(list)\n",
    "    for i, feature in enumerate(features):\n",
    "        features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n",
    "\n",
    "    # The dictionaries we have to fill.\n",
    "    predictions = collections.OrderedDict()\n",
    "\n",
    "    # Logging.\n",
    "    print(f\"Post-processing {len(examples)} example predictions split into {len(features)} features.\")\n",
    "\n",
    "    # Let's loop over all the examples!\n",
    "    for example_index, example in enumerate(tqdm(examples)):\n",
    "        # Those are the indices of the features associated to the current example.\n",
    "        feature_indices = features_per_example[example_index]\n",
    "\n",
    "        min_null_score = None # Only used if squad_v2 is True.\n",
    "        valid_answers = []\n",
    "        \n",
    "        context = example[\"context\"]\n",
    "        # Looping through all the features associated to the current example.\n",
    "        for feature_index in feature_indices:\n",
    "            # We grab the predictions of the model for this feature.\n",
    "            start_logits = all_start_logits[feature_index]\n",
    "            end_logits = all_end_logits[feature_index]\n",
    "            # This is what will allow us to map some the positions in our logits to span of texts in the original\n",
    "            # context.\n",
    "            offset_mapping = features[feature_index][\"offset_mapping\"]\n",
    "\n",
    "            # Update minimum null prediction.\n",
    "            cls_index = features[feature_index][\"input_ids\"].index(tokenizer.cls_token_id)\n",
    "            feature_null_score = start_logits[cls_index] + end_logits[cls_index]\n",
    "            if min_null_score is None or min_null_score < feature_null_score:\n",
    "                min_null_score = feature_null_score\n",
    "\n",
    "            # Go through all possibilities for the `n_best_size` greater start and end logits.\n",
    "            start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "            end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "            for start_index in start_indexes:\n",
    "                for end_index in end_indexes:\n",
    "                    # Don't consider out-of-scope answers, either because the indices are out of bounds or correspond\n",
    "                    # to part of the input_ids that are not in the context.\n",
    "                    if (\n",
    "                        start_index >= len(offset_mapping)\n",
    "                        or end_index >= len(offset_mapping)\n",
    "                        or offset_mapping[start_index] is None\n",
    "                        or offset_mapping[end_index] is None\n",
    "                    ):\n",
    "                        continue\n",
    "                    # Don't consider answers with a length that is either < 0 or > max_answer_length.\n",
    "                    if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n",
    "                        continue\n",
    "\n",
    "                    start_char = offset_mapping[start_index][0]\n",
    "                    end_char = offset_mapping[end_index][1]\n",
    "                    valid_answers.append(\n",
    "                        {\n",
    "                            \"score\": start_logits[start_index] + end_logits[end_index],\n",
    "                            \"text\": context[start_char: end_char]\n",
    "                        }\n",
    "                    )\n",
    "        \n",
    "        if len(valid_answers) > 0:\n",
    "            best_answer = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[0]\n",
    "        else:\n",
    "            # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid\n",
    "            # failure.\n",
    "            best_answer = {\"text\": \"\", \"score\": 0.0}\n",
    "        \n",
    "        # Let's pick our final answer: the best one or the null answer (only for squad_v2)\n",
    "        if not squad_v2:\n",
    "            predictions[example[\"id\"]] = best_answer[\"text\"]\n",
    "        else:\n",
    "            answer = best_answer[\"text\"] if best_answer[\"score\"] > min_null_score else \"\"\n",
    "            predictions[example[\"id\"]] = answer\n",
    "\n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can apply our post-processing function to our raw predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Post-processing 10570 example predictions split into 10917 features.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "faca00704d58456bb90d755443632338",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=10570.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "final_predictions = postprocess_qa_predictions(datasets[\"validation\"], validation_features, raw_predictions.predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can load the metric from the datasets library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric = load_metric(\"squad_v2\" if squad_v2 else \"squad\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can call compute on it. We just need to format predictions and labels a bit as it expects a list of dictionaries and not one big dictionary. In the case of squad_v2, we also have to set a `no_answer_probability` argument (which we set to 0.0 here as we have already set the answer to empty if we picked it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if squad_v2:\n",
    "    formatted_predictions = [{\"id\": k, \"prediction_text\": v, \"no_answer_probability\": 0.0} for k, v in final_predictions.items()]\n",
    "else:\n",
    "    formatted_predictions = [{\"id\": k, \"prediction_text\": v} for k, v in final_predictions.items()]\n",
    "references = [{\"id\": ex[\"id\"], \"answers\": ex[\"answers\"]} for ex in datasets[\"validation\"]]\n",
    "metric.compute(predictions=formatted_predictions, references=references)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13. Application QA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "hidden": true,
    "id": "Ckl2Fzn3is0F"
   },
   "outputs": [],
   "source": [
    "### import transformers\n",
    "import pathlib\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForQuestionAnswering, AutoTokenizer\n",
    "\n",
    "if model_checkpoint_original == \"neuralmind/bert-large-portuguese-cased\":\n",
    "    tokenizer_qa = AutoTokenizer.from_pretrained(model_checkpoint_original_local)\n",
    "    model_qa = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint_original_local)\n",
    "else:\n",
    "    tokenizer_qa = AutoTokenizer.from_pretrained(model_checkpoint)\n",
    "    model_qa = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lang adapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "if with_adapters_mlm:\n",
    "    \n",
    "    # hyperparameters used for fine-tuning the MLM with lang adapter\n",
    "    learning_rate_mlm = 1e-4\n",
    "    batch_size_mlm = 16\n",
    "    gradient_accumulation_steps_mlm = 1\n",
    "    adam_epsilon_mlm = 1e-4\n",
    "    num_train_epoch_mlm = 100.\n",
    "    early_stopping_patience_mlm = 10\n",
    "    madx2_mlm = madx2\n",
    "    houlsby_MHA_lastlayer_mlm = houlsby_MHA_lastlayer\n",
    "    ds_mlm = False\n",
    "    fp16_mlm = True\n",
    "    load_best_model_at_end_mlm = True\n",
    "    metric_for_best_model_mlm = \"loss\"\n",
    "    adapter_config_mlm = adapter_config_name + '+inv'\n",
    "\n",
    "    # path to lang adapter\n",
    "    outputs_mlm = model_checkpoint.replace('/','-') + '_' + dataset_name + '/' + 'mlm' + '/' \\\n",
    "    + 'lr' + str(learning_rate_mlm) \\\n",
    "    + '_bs' + str(batch_size_mlm) \\\n",
    "    + '_GAS' + str(gradient_accumulation_steps_mlm) \\\n",
    "    + '_eps' + str(adam_epsilon_mlm) \\\n",
    "    + '_epochs' + str(num_train_epoch_mlm) \\\n",
    "    + '_patience' + str(early_stopping_patience_mlm) \\\n",
    "    + '_madx2' + str(madx2_mlm) \\\n",
    "    + '_houlsby_MHA_lastlayer' + str(houlsby_MHA_lastlayer_mlm) \\\n",
    "    + '_ds' + str(ds_mlm) \\\n",
    "    + '_fp16' + str(fp16_mlm) \\\n",
    "    + '_best' + str(load_best_model_at_end_mlm) \\\n",
    "    + '_metric' + str(metric_for_best_model_mlm) \\\n",
    "    + '_adapterconfig' + str(adapter_config_mlm)\n",
    "\n",
    "    path_to_outputs_mlm = root/'outputs'/outputs_mlm\n",
    "\n",
    "    # Config of the lang adapter\n",
    "    lang_adapter_path = path_to_outputs_mlm/'adapters-mlm/'\n",
    "    \n",
    "    load_lang_adapter = str(lang_adapter_path)\n",
    "    lang_adapter_config = str(lang_adapter_path) + \"/adapter_config.json\"\n",
    "    \n",
    "else:\n",
    "    type_lang_adapter = None\n",
    "\n",
    "    load_lang_adapter = None\n",
    "    lang_adapter_config = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "if with_adapters_mlm:\n",
    "    # load the language adapter without head\n",
    "    task_mlm_load_as = 'mlm'\n",
    "    lang_adapter_name = model_qa.load_adapter(\n",
    "        load_lang_adapter,\n",
    "        config=lang_adapter_config,\n",
    "        load_as=task_mlm_load_as,\n",
    "        with_head=False\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fusion adapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "hidden": true,
    "id": "NiuPQTxuzRrm"
   },
   "outputs": [],
   "source": [
    "# load adapter\n",
    "adapter_name_ad1 = model_qa.load_adapter(\n",
    "    load_qa_adapter,\n",
    "    config=qa_adapter_config,\n",
    "    load_as=load_as_ad1, \n",
    "    with_head=False\n",
    ")\n",
    "    \n",
    "adapter_name_ad2 = model_qa.load_adapter(\n",
    "    load_ner_adapter,\n",
    "    config=ner_adapter_config,\n",
    "    load_as=load_as_ad2, \n",
    "    with_head=False\n",
    ")\n",
    "    \n",
    "model_qa.load_adapter_fusion(path_to_save_adapter)\n",
    "    \n",
    "model_qa.load_head(path_to_save_adapter)\n",
    "\n",
    "adapter_setup = [\n",
    "    [\n",
    "        load_as_ad1,\n",
    "        load_as_ad2\n",
    "    ]\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Active adapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the adapters to be used in every forward pass\n",
    "if with_adapters_mlm:\n",
    "    model_qa.set_active_adapters([lang_adapter_name,adapter_setup[0]]) \n",
    "else:\n",
    "    model_qa.set_active_adapters(adapter_setup[0])    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertForQuestionAnswering(\n",
       "  (bert): BertModel(\n",
       "    (invertible_adapters): ModuleDict(\n",
       "      (mlm): NICECouplingBlock(\n",
       "        (F): Sequential(\n",
       "          (0): Linear(in_features=512, out_features=256, bias=True)\n",
       "          (1): Activation_Function_Class()\n",
       "          (2): Linear(in_features=256, out_features=512, bias=True)\n",
       "        )\n",
       "        (G): Sequential(\n",
       "          (0): Linear(in_features=512, out_features=256, bias=True)\n",
       "          (1): Activation_Function_Class()\n",
       "          (2): Linear(in_features=256, out_features=512, bias=True)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (embeddings): BertEmbeddings(\n",
       "      (word_embeddings): Embedding(29794, 1024, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 1024)\n",
       "      (token_type_embeddings): Embedding(2, 1024)\n",
       "      (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "    (encoder): BertEncoder(\n",
       "      (layer): ModuleList(\n",
       "        (0): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (1): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (2): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (3): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (4): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (5): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (6): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (7): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (8): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (9): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (10): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (11): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (12): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (13): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (14): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (15): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (16): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (17): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (18): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (19): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (20): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (21): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (22): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict(\n",
       "              (mlm): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=512, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=512, out_features=1024, bias=True)\n",
       "              )\n",
       "              (qa_squad11pt): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "              (ner_lenerbr): Adapter(\n",
       "                (non_linearity): Activation_Function_Class()\n",
       "                (adapter_down): Sequential(\n",
       "                  (0): Linear(in_features=1024, out_features=64, bias=True)\n",
       "                  (1): Activation_Function_Class()\n",
       "                )\n",
       "                (adapter_up): Linear(in_features=64, out_features=1024, bias=True)\n",
       "              )\n",
       "            )\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "        (23): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (value): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "              (adapters): ModuleDict()\n",
       "              (adapter_fusion_layer): ModuleDict()\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
       "            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (adapters): ModuleDict()\n",
       "            (adapter_fusion_layer): ModuleDict(\n",
       "              (qa_squad11pt,ner_lenerbr): BertFusion(\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "                (query): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (key): Linear(in_features=1024, out_features=1024, bias=True)\n",
       "                (value): Linear(in_features=1024, out_features=1024, bias=False)\n",
       "              )\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (qa_outputs): Linear(in_features=1024, out_features=2, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_qa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "nlp = pipeline(\"question-answering\", model=model_qa, tokenizer=tokenizer_qa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "# source: https://pt.wikipedia.org/wiki/Pandemia_de_COVID-19\n",
    "context = r\"\"\"A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, \n",
    "uma doença respiratória causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). \n",
    "O vírus tem origem zoonótica e o primeiro caso conhecido da doença remonta a dezembro de 2019 em Wuhan, na China. \n",
    "Em 20 de janeiro de 2020, a Organização Mundial da Saúde (OMS) classificou o surto \n",
    "como Emergência de Saúde Pública de Âmbito Internacional e, em 11 de março de 2020, como pandemia. \n",
    "Em 18 de junho de 2021, 177 349 274 casos foram confirmados em 192 países e territórios, \n",
    "com 3 840 181 mortes atribuídas à doença, tornando-se uma das pandemias mais mortais da história.\n",
    "Os sintomas de COVID-19 são altamente variáveis, variando de nenhum a doenças com risco de morte. \n",
    "O vírus se espalha principalmente pelo ar quando as pessoas estão perto umas das outras. \n",
    "Ele deixa uma pessoa infectada quando ela respira, tosse, espirra ou fala e entra em outra pessoa pela boca, nariz ou olhos.\n",
    "Ele também pode se espalhar através de superfícies contaminadas. \n",
    "As pessoas permanecem contagiosas por até duas semanas e podem espalhar o vírus mesmo se forem assintomáticas.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Quando começou a pandemia de Covid-19 no mundo?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Qual é a data de início da pandemia Covid-19 em todo o mundo?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"A Covid-19 tem algo a ver com animais?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Onde foi descoberta a Covid-19?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Quantos casos houve?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Quantos mortes?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Quantos paises tiveram casos?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Quais são sintomas de COVID-19\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Como se espalha o vírus?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# END"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "Language Modeling",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "adapter-transformers",
   "language": "python",
   "name": "adapter-transformers"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}