Sentence_Similarity.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Sentence Similarity.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyM/CF8F1oT1CgSqFKiHjbbF",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/elvinagam/Hands-MediumON-ML/blob/master/Sentence_Similarity.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Semantic Text Similarity and Paraphrase [Mining](https://github.com/UKPLab/sentence-transformers)"
      ],
      "metadata": {
        "id": "qt43RVh_bUJK"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "NvbagU8EUbIY"
      },
      "outputs": [],
      "source": [
        "# 1 - Import Libraries & Install Files\n",
        "import torch\n",
        "from sentence_transformers import SentenceTransformer\n",
        "from sentence_transformers import util "
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#We will find similarity for these two sentences\n",
        "sentence1 = \"This is a sentence\"\n",
        "sentence2 = \"This is also a sentence\""
      ],
      "metadata": {
        "id": "fkfsbXSWUefS"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Download model\n",
        "model = SentenceTransformer('all-MiniLM-L6-v2')"
      ],
      "metadata": {
        "id": "makC0vx8Zjnm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Find sentence embeddings for sentences using SBERT model\n",
        "embedding_sen1 = model.encode(sentence1)\n",
        "embedding_sen2 = model.encode(sentence2)\n",
        "\n",
        "#Find cosine distance between the sentences\n",
        "cos_sim = util.cos_sim(embedding_sen1, embedding_sen2)\n",
        "print(\"Cosine-Similarity:\", cos_sim.numpy()[0][0])\n",
        "print(\"Percentage Similarity: \", cos_sim.numpy()[0][0]*100, \"%\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Sh4QOq6rZkqB",
        "outputId": "045267b9-b594-4f17-a75a-8fa7f7bb27d1"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cosine-Similarity: 0.87709093\n",
            "Percentage Similarity:  87.70909309387207 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Single list of sentences\n",
        "sentences = ['The cat sits outside',\n",
        "             'A man is playing guitar',\n",
        "             'I love pasta',\n",
        "             'The new movie is awesome',\n",
        "             'The cat plays in the garden',\n",
        "             'A woman watches TV',\n",
        "             'The new movie is so great',\n",
        "             'Do you like pizza?']"
      ],
      "metadata": {
        "id": "QtyaZc8-ZoY7"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Compute embeddings\n",
        "embeddings = model.encode(sentences, convert_to_tensor=True)\n",
        "\n",
        "#Compute cosine-similarities for each sentence with each other sentence\n",
        "cosine_scores = util.cos_sim(embeddings, embeddings)\n",
        "\n",
        "#Find the pairs with the highest cosine similarity scores\n",
        "pairs = []\n",
        "for i in range(len(cosine_scores)-1):\n",
        "    for j in range(i+1, len(cosine_scores)):\n",
        "        pairs.append({'index': [i, j], 'score': cosine_scores[i][j]})\n",
        "\n",
        "#Sort scores in decreasing order\n",
        "pairs = sorted(pairs, key=lambda x: x['score'], reverse=True)"
      ],
      "metadata": {
        "id": "YHEuF36YahTI"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for pair in pairs[0:10]:\n",
        "    i, j = pair['index']\n",
        "    print(\"{} \\t\\t {} \\t\\t Score: {:.4f}\".format(sentences[i], sentences[j], pair['score']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xY7GHrL8akY-",
        "outputId": "fa5f4ffd-0505-4c31-cb79-f7f1a0db7a5f"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The new movie is awesome \t\t The new movie is so great \t\t Score: 0.8939\n",
            "The cat sits outside \t\t The cat plays in the garden \t\t Score: 0.6788\n",
            "I love pasta \t\t Do you like pizza? \t\t Score: 0.5096\n",
            "I love pasta \t\t The new movie is so great \t\t Score: 0.2560\n",
            "I love pasta \t\t The new movie is awesome \t\t Score: 0.2440\n",
            "A man is playing guitar \t\t The cat plays in the garden \t\t Score: 0.2105\n",
            "The new movie is awesome \t\t Do you like pizza? \t\t Score: 0.1969\n",
            "The new movie is so great \t\t Do you like pizza? \t\t Score: 0.1692\n",
            "The cat sits outside \t\t A woman watches TV \t\t Score: 0.1310\n",
            "The cat plays in the garden \t\t Do you like pizza? \t\t Score: 0.0900\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The approach presented above used a brute-force approach to score and rank all pairs.However, as this has a quadratic runtime, it fails to scale to large (10,000 and more) collections of sentences.\n",
        "\n",
        "For larger collections, util offers the paraphrase_mining function that can be used like this:"
      ],
      "metadata": {
        "id": "cY1eMLMDbAIY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "mZbBrIMBbAFX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = ['The cat sits outside',\n",
        "             'A man is playing guitar',\n",
        "             'I love pasta',\n",
        "             'The new movie is awesome',\n",
        "             'The cat plays in the garden',\n",
        "             'A woman watches TV',\n",
        "             'The new movie is so great',\n",
        "             'Do you like pizza?']\n",
        "\n",
        "paraphrases = util.paraphrase_mining(model, sentences)\n",
        "\n",
        "for paraphrase in paraphrases[0:10]:\n",
        "    score, i, j = paraphrase\n",
        "    print(\"{} \\t\\t {} \\t\\t Score: {:.4f}\".format(sentences[i], sentences[j], score))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "71oszKg0alVW",
        "outputId": "cf2eaa4e-3107-4d70-df9d-876dbddd1184"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The new movie is so great \t\t The new movie is awesome \t\t Score: 0.8939\n",
            "The cat sits outside \t\t The cat plays in the garden \t\t Score: 0.6788\n",
            "Do you like pizza? \t\t I love pasta \t\t Score: 0.5096\n",
            "I love pasta \t\t The new movie is so great \t\t Score: 0.2560\n",
            "I love pasta \t\t The new movie is awesome \t\t Score: 0.2440\n",
            "A man is playing guitar \t\t The cat plays in the garden \t\t Score: 0.2105\n",
            "The new movie is awesome \t\t Do you like pizza? \t\t Score: 0.1969\n",
            "The new movie is so great \t\t Do you like pizza? \t\t Score: 0.1692\n",
            "The cat sits outside \t\t A woman watches TV \t\t Score: 0.1310\n",
            "The cat plays in the garden \t\t Do you like pizza? \t\t Score: 0.0900\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Instead of computing all pairwise cosine scores and ranking all possible, combintations, the approach is a bit more complex (and hence efficient). We chunk our corpus into smaller pieces, which is defined by query_chunk_size and corpus_chunk_size. For example, if we set query_chunk_size=1000, we search paraphrases for 1,000 sentences at a time in the remaining corpus (all other sentences). However, the remaining corpus is also chunked, for example, if we set corpus_chunk_size=10000, we look for paraphrases in 10k sentences at a time.\n",
        "\n",
        "If we pass a list of 20k sentences, we will chunk it to 20x1000 sentences, and each of the query is compared first against sentences 0-10k and then 10k-20k.\n",
        "\n",
        "This is done to reduce the memory requirement. Increasing both values improves the speed, but increases also the memory requirement.\n",
        "\n",
        "The next critical thing is finding the pairs with the highest similarities. Instead of getting and sorting all n^2 pairwise scores, we take for each query only the top_k scores. So with top_k=100, we find at most 100 paraphrases per sentence per chunk. You can play around with top_k to the ensure a certain behaviour.\n",
        "\n",
        "So for example, with\n",
        "\n",
        "paraphrases = util.paraphrase_mining(model, sentences, corpus_chunk_size=len(sentences), top_k=1)\n",
        "You will get for each sentence only the one most other relevant sentence. Note, if B is the most similar sentence for A, A must not be the most similar sentence for B. So it can happen that the returned list contains entries like (A, B) and (B, C).\n",
        "\n",
        "The final relevant parameter is max_pairs, which determines the maximum number of paraphrase pairs you like to get returned. If you set it to e.g. max_pairs=100, you will not get more than 100 paraphrase pairs returned. Usually, you get fewer pairs returned as the list is cleaned of duplicates, e.g., if it contains (A, B) and (B, A), then only one is returned."
      ],
      "metadata": {
        "id": "xG1-s3QGbRhd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "S8g4stCVa1-A"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}