translation-playground.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimize M2M100 model with ONNX\n",
    "\n",
    "\n",
    "In this notebook we will describes steps to laod models with m2m100 translation models from HuggingFace and optimize them with ONNX Runtime. We will also show how to use the optimized model to perform translation.\n",
    "\n",
    "Once the model are optimize we will deploy them as an Api so that they can be used in a web application.\n",
    "\n",
    "At the first step we will load the vanilla model from Hugginface and use it for inference, then we will convert it to ONNX and Finally we will optimize it with ONNX Runtime."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First Step\n",
    "\n",
    "Loading the vanilla model from hugginface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/es.py/Projects/Personal/masakhane-web/.venv/lib/python3.10/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoTokenizer, M2M100ForConditionalGeneration, pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME = \"masakhane/m2m100_418M_en_swa_rel_news\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = M2M100ForConditionalGeneration.from_pretrained(MODEL_NAME)\n",
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_to_translate = \"Hello, my name is Espoir Murhabazi,  I am a Software Engineer from Congo DRC but living in UK\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_input = tokenizer(text_to_translate, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/es.py/Projects/Personal/masakhane-web/.venv/lib/python3.10/site-packages/transformers/generation/utils.py:1273: UserWarning: Neither `max_length` nor `max_new_tokens` has been set, `max_length` will default to 200 (`generation_config.max_length`). Controlling `max_length` via the config is deprecated and `max_length` will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "generated_tokens = model.generate(**model_input, forced_bos_token_id=tokenizer.lang_code_to_id[\"sw\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "translated_text = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Jina langu ni Espoir Murhabazi, Mimi ni mhandisi wa programu za kompyuta kutoka Kongo DRC lakini ninaishi Uingereza']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translated_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trying to export the model manually and see if we can load the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_SUFFIX = MODEL_NAME.replace('masakhane/', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "onnx_inputs, onnx_outputs = export_onnx(\n",
    "    preprocessor=tokenizer,\n",
    "    model=model,\n",
    "    config=onnx_config,\n",
    "    opset=13,\n",
    "    output=output_path,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'m2m100_418M_en_swa_rel_news'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MODEL_SUFFIX"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This command is not working properly, It is saving the model as one file instead of two file one for the encoder another one for the decoder.\n",
    "\n",
    "The best approach is to use CLI as suggested in the documentation.\n",
    "\n",
    "` optimum-cli export onnx --model masakhane/m2m100_418M_en_swa_rel_news --task seq2seq-lm-with-past --for-ort onnx`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Framework not specified. Using pt to export to ONNX.\n",
      "Using framework PyTorch: 1.13.1\n",
      "Overriding 1 configuration item(s)\n",
      "\t- use_cache -> False\n",
      "/Users/es.py/Projects/Personal/masakhane-web/.venv/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py:172: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if max_pos > self.weights.size(0):\n",
      "/Users/es.py/Projects/Personal/masakhane-web/.venv/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py:293: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):\n",
      "/Users/es.py/Projects/Personal/masakhane-web/.venv/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py:300: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if attention_mask.size() != (bsz, 1, tgt_len, src_len):\n",
      "/Users/es.py/Projects/Personal/masakhane-web/.venv/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py:332: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):\n",
      "Using framework PyTorch: 1.13.1\n",
      "Overriding 1 configuration item(s)\n",
      "\t- use_cache -> True\n",
      "/Users/es.py/Projects/Personal/masakhane-web/.venv/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py:1003: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if input_shape[-1] > 1:\n",
      "/Users/es.py/Projects/Personal/masakhane-web/.venv/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py:81: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.\n",
      "  mask = torch.full((tgt_len, tgt_len), torch.tensor(torch.finfo(dtype).min))\n",
      "Using framework PyTorch: 1.13.1\n",
      "Overriding 1 configuration item(s)\n",
      "\t- use_cache -> True\n",
      "Asked a sequence length of 16, but a sequence length of 1 will be used with use_past ==True for `decoder_input_ids`.\n",
      "/Users/es.py/Projects/Personal/masakhane-web/.venv/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py:252: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if (\n",
      "Validating ONNX model...\n",
      "\t-[✓] ONNX model output names match reference model (last_hidden_state)\n",
      "\t- Validating ONNX Model output \"last_hidden_state\":\n",
      "\t\t-[✓] (2, 16, 1024) matches (2, 16, 1024)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "Validating ONNX model...\n",
      "\t-[✓] ONNX model output names match reference model (present.3.decoder.key, present.7.encoder.key, present.9.decoder.key, present.1.decoder.value, present.10.decoder.key, present.11.encoder.value, present.4.encoder.value, present.10.encoder.value, present.8.encoder.value, present.0.encoder.key, present.2.encoder.value, present.8.decoder.value, encoder_last_hidden_state, present.5.encoder.value, present.4.encoder.key, present.1.encoder.key, present.1.encoder.value, present.9.encoder.key, present.9.encoder.value, present.5.encoder.key, present.6.encoder.value, present.6.decoder.value, present.6.decoder.key, present.10.decoder.value, present.2.decoder.value, present.7.decoder.value, present.0.decoder.key, present.7.decoder.key, present.6.encoder.key, present.11.decoder.key, present.9.decoder.value, present.5.decoder.key, present.8.decoder.key, present.4.decoder.value, present.3.encoder.key, present.2.encoder.key, present.7.encoder.value, present.11.encoder.key, present.11.decoder.value, present.1.decoder.key, present.2.decoder.key, present.10.encoder.key, present.8.encoder.key, present.0.decoder.value, present.4.decoder.key, present.3.decoder.value, present.5.decoder.value, present.0.encoder.value, logits, present.3.encoder.value)\n",
      "\t- Validating ONNX Model output \"logits\":\n",
      "\t\t-[✓] (2, 16, 128112) matches (2, 16, 128112)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"encoder_last_hidden_state\":\n",
      "\t\t-[✓] (2, 16, 1024) matches (2, 16, 1024)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "Validating ONNX model...\n",
      "\t-[✓] ONNX model output names match reference model (present.3.decoder.key, present.7.encoder.key, present.9.decoder.key, present.1.decoder.value, present.10.decoder.key, present.11.encoder.value, present.4.encoder.value, present.10.encoder.value, present.8.encoder.value, present.0.encoder.key, present.2.encoder.value, present.8.decoder.value, encoder_last_hidden_state, present.5.encoder.value, present.4.encoder.key, present.1.encoder.key, present.1.encoder.value, present.9.encoder.key, present.9.encoder.value, present.5.encoder.key, present.6.encoder.value, present.6.decoder.value, present.6.decoder.key, present.10.decoder.value, present.2.decoder.value, present.7.decoder.value, present.0.decoder.key, present.7.decoder.key, present.6.encoder.key, present.11.decoder.key, present.9.decoder.value, present.5.decoder.key, present.8.decoder.key, present.4.decoder.value, present.3.encoder.key, present.2.encoder.key, present.7.encoder.value, present.11.encoder.key, present.11.decoder.value, present.1.decoder.key, present.2.decoder.key, present.10.encoder.key, present.8.encoder.key, present.0.decoder.value, present.4.decoder.key, present.3.decoder.value, present.5.decoder.value, present.0.encoder.value, logits, present.3.encoder.value)\n",
      "\t- Validating ONNX Model output \"logits\":\n",
      "\t\t-[✓] (2, 1, 128112) matches (2, 1, 128112)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.decoder.key\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.decoder.value\":\n",
      "\t\t-[✓] (2, 16, 17, 64) matches (2, 16, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.encoder.key\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.encoder.value\":\n",
      "\t\t-[✓] (2, 16, 16, 64) matches (2, 16, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"encoder_last_hidden_state\":\n",
      "\t\t-[✓] (2, 16, 1024) matches (2, 16, 1024)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "The ONNX export succeeded and the exported model was saved at: onnx/m2m100_418M_en_swa_rel_news\n"
     ]
    }
   ],
   "source": [
    "! optimum-cli export onnx --model masakhane/m2m100_418M_en_swa_rel_news --task seq2seq-lm-with-past --for-ort onnx/m2m100_418M_en_swa_rel_news"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "check if the model is correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_model_onnx_dir = Path(\"onnx\").joinpath(MODEL_SUFFIX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('onnx/m2m100_418M_en_swa_rel_news')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base_model_onnx_dir"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use the optimization to Opimze the model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will apply the first optimization to the model we saved in the previous step.\n",
    "\n",
    "We will start by testing the basica optimization to see "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.onnxruntime import ORTOptimizer\n",
    "from optimum.onnxruntime.configuration import OptimizationConfig\n",
    "from transformers import AutoConfig\n",
    "from optimum.onnxruntime import ORTModelForSeq2SeqLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/es.py/Projects/Personal/masakhane-web/.venv/lib/python3.10/site-packages/optimum/onnxruntime/configuration.py:726: FutureWarning: disable_embed_layer_norm will be deprecated soon, use disable_embed_layer_norm_fusion instead, disable_embed_layer_norm_fusion is set to True.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "optimization_config = OptimizationConfig(optimization_level=99)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "onnx_model =  ORTModelForSeq2SeqLM.from_pretrained(base_model_onnx_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = ORTOptimizer.from_pretrained(onnx_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimized_model_path = Path(\"onnx\").joinpath(f\"{MODEL_SUFFIX}_optimized/\")\n",
    "optimized_model_path.mkdir(parents=True, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-03-01 15:37:47.100773 [W:onnxruntime:, inference_session.cc:1546 Initialize] Serializing optimized model with Graph Optimization level greater than ORT_ENABLE_EXTENDED and the NchwcTransformer enabled. The generated model may contain hardware specific optimizations, and should only be used in the same environment the model was optimized in.\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "2023-03-01 15:38:03.759503 [W:onnxruntime:, inference_session.cc:1546 Initialize] Serializing optimized model with Graph Optimization level greater than ORT_ENABLE_EXTENDED and the NchwcTransformer enabled. The generated model may contain hardware specific optimizations, and should only be used in the same environment the model was optimized in.\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "2023-03-01 15:38:34.212186 [W:onnxruntime:, inference_session.cc:1546 Initialize] Serializing optimized model with Graph Optimization level greater than ORT_ENABLE_EXTENDED and the NchwcTransformer enabled. The generated model may contain hardware specific optimizations, and should only be used in the same environment the model was optimized in.\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n",
      "symbolic shape infer failed. it's safe to ignore this message if there is no issue with optimized model\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('onnx/m2m100_418M_en_swa_rel_news_optimized')"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "optimizer.optimize(save_dir=optimized_model_path, optimization_config=optimization_config)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the optimize model and check if the model is working."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use the optimized model\n",
    "\n",
    "Once we have developed the model, let us now use the optimized model to run the inference and check if the model is working."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.onnxruntime import ORTModelForSeq2SeqLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimized_model = ORTModelForSeq2SeqLM.from_pretrained(optimized_model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.pipelines import pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "onnx_optimize = pipeline(\"translation_en_to_sw\", model=optimized_model, tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "translated_text = onnx_optimize(text_to_translate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'translation_text': 'Jina langu ni Espoir Murhabazi, Mimi ni mhandisi wa programu za kompyuta kutoka Kongo DRC lakini ninaishi Uingereza'}]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translated_text"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have managed to apply optimization and run the inference on the model, the last issue will be to run the test to check if the performance of the predicted model is good but at least the model is now working. I need to now move to the next step which is deploying the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying Quantization"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learn more about quantization here.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.onnxruntime import ORTQuantizer, ORTModelForSeq2SeqLM\n",
    "from optimum.onnxruntime.configuration import AutoQuantizationConfig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_quantizer = ORTQuantizer.from_pretrained(base_model_onnx_dir, file_name=\"encoder_model.onnx\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_quantizer = ORTQuantizer.from_pretrained(base_model_onnx_dir, file_name=\"decoder_model.onnx\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_with_past_quantizer = ORTQuantizer.from_pretrained(base_model_onnx_dir, file_name=\"decoder_with_past_model.onnx\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "quantizers = [encoder_quantizer, decoder_quantizer, decoder_with_past_quantizer]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "dynamic_quantization_config = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "quantized_model_path = Path(\"onnx\").joinpath(f\"{MODEL_SUFFIX}_quantized/\")\n",
    "quantized_model_path.mkdir(parents=True, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "for quantizer in quantizers:\n",
    "    quantizer.quantize(quantization_config=dynamic_quantization_config, save_dir=quantized_model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "quantized_model = ORTModelForSeq2SeqLM.from_pretrained(quantized_model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "quantized_pipeline = pipeline(\"translation_en_to_sw\", model=quantized_model, tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "translated_text_quantized = quantized_pipeline(text_to_translate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'translation_text': 'Jina langu ni Espoir Murhabazi, Mimi ni mhandisi wa programu za kompyuta kutoka Kongo DRC lakini ninaishi Uingereza'}]\n"
     ]
    }
   ],
   "source": [
    "print(translated_text_quantized)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The quantization seems to reduce the size of the model but keeping the same performance, as per the documentaiton and experience performed on other models, we need to perform the quantization on other model to check for the performance."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "e7d420a2576d2f2cf4aee17bb1c719cb2b545f2d9fd7bdced2270e528bc643b9"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}