nlp-llm-vector-embeddings-db-search.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":7296128,"sourceType":"datasetVersion","datasetId":4232024},{"sourceId":1428159,"sourceType":"datasetVersion","datasetId":836401}],"dockerImageVersionId":30627,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# | NLP | LLM | Vector | Embeddings DB Search |\n\n## Natural Language Processing (NLP), Large Language Models (LLM), and the Power of Vector Embeddings and Databases\n\n![Learning](https://t3.ftcdn.net/jpg/06/14/01/52/360_F_614015247_EWZHvC6AAOsaIOepakhyJvMqUu5tpLfY.jpg)\n\n# <b>1 <span style='color:#78D118'>|</span> Overview</b>\n\n### Embeddings, Vector Databases, and Advanced Search\n\nConverting text into embedding vectors is the first step to any text processing pipeline. As the amount of text gets larger, there is often a need to save these embedding vectors into a dedicated vector index or library, so that developers won't have to recompute the embeddings and the retrieval process is faster. We can then search for documents based on our intended query and pass these relevant documents into a language model (LM) as additional context. We also refer to this context as supplying the LM with \"state\" or \"memory\". The LM then generates a response based on the additional context it receives! \n\nIn this notebook, we will implement the full workflow of text vectorization, vector search, and question answering workflow. While we use [FAISS](https://faiss.ai/) (vector library) and [ChromaDB](https://docs.trychroma.com/) (vector database), and a Hugging Face model, know that you can easily swap these tools out for your preferred tools or models!\n\n<img src=\"https://storage.googleapis.com/kagglesdsdata/datasets/4232024/7296128/vector.png?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=databundle-worker-v2%40kaggle-161607.iam.gserviceaccount.com%2F20231228%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20231228T094528Z&X-Goog-Expires=345600&X-Goog-SignedHeaders=host&X-Goog-Signature=2358155e44397b47c5cd178ae645e0ad4a656e3f30b773ce9ce29dc596a7c59319465e6fa9f2c8019257a0be627bbff2e3452aa2cb9b8e117813fdf62b82d6864f2b6a31f00e4bdab75eafff8b634426b6d0781331179283408fa357eaf43ce17dfcafea30461d305fb647abdfbe4309ca0921b5bd816451e084e12df550ebad38f020e39b1ebc2df4821c02cc861d5b44a6eb581bfada85c074e6913d64554777cab3c035f2e946146fc743dadbeee62019fa5649219573616c341566ffc3a122b04070cb72152f4bc4810c4be4bd45f28442a7d1e1b8e20c279c511b506274a31208f7aa8b61b91af0dbaab42b3aa6c52dc5024dd98d1c2538d08d47f9d6ce\" width=1000 target=\"_blank\" > \n\n### Learning Objectives\n1. Implement the workflow of reading text, converting text to embeddings, saving them to FAISS and ChromaDB \n2. Query for similar documents using FAISS and ChromaDB \n3. Apply a Hugging Face language model for question answering.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"## Setup\n\n### Libraries","metadata":{}},{"cell_type":"code","source":"pip install faiss-cpu==1.7.4 chromadb==0.3.21","metadata":{"execution":{"iopub.status.busy":"2023-12-28T09:54:52.573388Z","iopub.execute_input":"2023-12-28T09:54:52.573660Z","iopub.status.idle":"2023-12-28T09:55:45.600639Z","shell.execute_reply.started":"2023-12-28T09:54:52.573635Z","shell.execute_reply":"2023-12-28T09:55:45.599466Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":1,"outputs":[{"name":"stdout","text":"Collecting faiss-cpu==1.7.4\n  Downloading faiss_cpu-1.7.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.6 MB)\n\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.6/17.6 MB\u001b[0m \u001b[31m59.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n\u001b[?25hCollecting chromadb==0.3.21\n  Downloading chromadb-0.3.21-py3-none-any.whl (46 kB)\n\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.4/46.4 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hRequirement already satisfied: pandas>=1.3 in /opt/conda/lib/python3.10/site-packages (from chromadb==0.3.21) (2.0.3)\nRequirement already satisfied: requests>=2.28 in /opt/conda/lib/python3.10/site-packages (from chromadb==0.3.21) (2.31.0)\nRequirement already satisfied: pydantic>=1.9 in /opt/conda/lib/python3.10/site-packages (from chromadb==0.3.21) (1.10.12)\nCollecting hnswlib>=0.7 (from chromadb==0.3.21)\n  Downloading hnswlib-0.8.0.tar.gz (36 kB)\n  Installing build dependencies ... \u001b[?25ldone\n\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n\u001b[?25hCollecting clickhouse-connect>=0.5.7 (from chromadb==0.3.21)\n  Obtaining dependency information for clickhouse-connect>=0.5.7 from https://files.pythonhosted.org/packages/4c/7c/a86a5fd861f584623465b8960713a5cfbf7ff1ad6f79caba55608d0d38d1/clickhouse_connect-0.6.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata\n  Downloading clickhouse_connect-0.6.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.8 kB)\nCollecting sentence-transformers>=2.2.2 (from chromadb==0.3.21)\n  Downloading sentence-transformers-2.2.2.tar.gz (85 kB)\n\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.0/86.0 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n\u001b[?25hCollecting duckdb>=0.7.1 (from chromadb==0.3.21)\n  Obtaining dependency information for duckdb>=0.7.1 from https://files.pythonhosted.org/packages/7c/d5/7688cdc1b57c9ada17b6f6143a923606294b462e4b15e3ac249c04649bde/duckdb-0.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata\n  Downloading duckdb-0.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (760 bytes)\nRequirement already satisfied: fastapi>=0.85.1 in /opt/conda/lib/python3.10/site-packages (from chromadb==0.3.21) (0.101.1)\nRequirement already satisfied: uvicorn[standard]>=0.18.3 in /opt/conda/lib/python3.10/site-packages (from chromadb==0.3.21) (0.23.2)\nRequirement already satisfied: numpy>=1.21.6 in /opt/conda/lib/python3.10/site-packages (from chromadb==0.3.21) (1.24.3)\nCollecting posthog>=2.4.0 (from chromadb==0.3.21)\n  Obtaining dependency information for posthog>=2.4.0 from https://files.pythonhosted.org/packages/3b/82/441cb77a43499661228048dcd0d21e0ae3235b442d0f1b9b606e29c2a5ed/posthog-3.1.0-py2.py3-none-any.whl.metadata\n  Downloading posthog-3.1.0-py2.py3-none-any.whl.metadata (2.0 kB)\nRequirement already satisfied: certifi in /opt/conda/lib/python3.10/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.21) (2023.11.17)\nRequirement already satisfied: urllib3>=1.26 in /opt/conda/lib/python3.10/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.21) (1.26.15)\nRequirement already satisfied: pytz in /opt/conda/lib/python3.10/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.21) (2023.3)\nRequirement already satisfied: zstandard in /opt/conda/lib/python3.10/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.21) (0.22.0)\nRequirement already satisfied: lz4 in /opt/conda/lib/python3.10/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.21) (4.3.2)\nRequirement already satisfied: starlette<0.28.0,>=0.27.0 in /opt/conda/lib/python3.10/site-packages (from fastapi>=0.85.1->chromadb==0.3.21) (0.27.0)\nRequirement already satisfied: typing-extensions>=4.5.0 in /opt/conda/lib/python3.10/site-packages (from fastapi>=0.85.1->chromadb==0.3.21) (4.5.0)\nRequirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.3->chromadb==0.3.21) (2.8.2)\nRequirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.3->chromadb==0.3.21) (2023.3)\nRequirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb==0.3.21) (1.16.0)\nCollecting monotonic>=1.5 (from posthog>=2.4.0->chromadb==0.3.21)\n  Downloading monotonic-1.6-py2.py3-none-any.whl (8.2 kB)\nRequirement already satisfied: backoff>=1.10.0 in /opt/conda/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb==0.3.21) (2.2.1)\nRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests>=2.28->chromadb==0.3.21) (3.2.0)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests>=2.28->chromadb==0.3.21) (3.4)\nRequirement already satisfied: transformers<5.0.0,>=4.6.0 in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.2->chromadb==0.3.21) (4.36.0)\nRequirement already satisfied: tqdm in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.2->chromadb==0.3.21) (4.66.1)\nRequirement already satisfied: torch>=1.6.0 in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.2->chromadb==0.3.21) (2.0.0)\nRequirement already satisfied: torchvision in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.2->chromadb==0.3.21) (0.15.1)\nRequirement already satisfied: scikit-learn in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.2->chromadb==0.3.21) (1.2.2)\nRequirement already satisfied: scipy in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.2->chromadb==0.3.21) (1.11.4)\nRequirement already satisfied: nltk in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.2->chromadb==0.3.21) (3.2.4)\nRequirement already satisfied: sentencepiece in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.2->chromadb==0.3.21) (0.1.99)\nRequirement already satisfied: huggingface-hub>=0.4.0 in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.2->chromadb==0.3.21) (0.19.4)\nRequirement already satisfied: click>=7.0 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.21) (8.1.7)\nRequirement already satisfied: h11>=0.8 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.21) (0.14.0)\nRequirement already satisfied: httptools>=0.5.0 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.21) (0.6.1)\nRequirement already satisfied: python-dotenv>=0.13 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.21) (1.0.0)\nRequirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.21) (6.0.1)\nRequirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.21) (0.19.0)\nRequirement already satisfied: watchfiles>=0.13 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.21) (0.21.0)\nRequirement already satisfied: websockets>=10.4 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.21) (12.0)\nRequirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (3.12.2)\nRequirement already satisfied: fsspec>=2023.5.0 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (2023.12.2)\nRequirement already satisfied: packaging>=20.9 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (21.3)\nRequirement already satisfied: anyio<5,>=3.4.0 in /opt/conda/lib/python3.10/site-packages (from starlette<0.28.0,>=0.27.0->fastapi>=0.85.1->chromadb==0.3.21) (3.7.1)\nRequirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (1.12)\nRequirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (3.1)\nRequirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (3.1.2)\nRequirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (2023.8.8)\nRequirement already satisfied: tokenizers<0.19,>=0.14 in /opt/conda/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (0.15.0)\nRequirement already satisfied: safetensors>=0.3.1 in /opt/conda/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (0.4.1)\nRequirement already satisfied: joblib>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from scikit-learn->sentence-transformers>=2.2.2->chromadb==0.3.21) (1.3.2)\nRequirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from scikit-learn->sentence-transformers>=2.2.2->chromadb==0.3.21) (3.2.0)\nRequirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/lib/python3.10/site-packages (from torchvision->sentence-transformers>=2.2.2->chromadb==0.3.21) (10.1.0)\nRequirement already satisfied: sniffio>=1.1 in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi>=0.85.1->chromadb==0.3.21) (1.3.0)\nRequirement already satisfied: exceptiongroup in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi>=0.85.1->chromadb==0.3.21) (1.1.3)\nRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.10/site-packages (from packaging>=20.9->huggingface-hub>=0.4.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (3.0.9)\nRequirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2->torch>=1.6.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (2.1.3)\nRequirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch>=1.6.0->sentence-transformers>=2.2.2->chromadb==0.3.21) (1.3.0)\nDownloading clickhouse_connect-0.6.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (964 kB)\n\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m964.5/964.5 kB\u001b[0m \u001b[31m53.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hDownloading duckdb-0.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.5 MB)\n\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.5/16.5 MB\u001b[0m \u001b[31m62.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n\u001b[?25hDownloading posthog-3.1.0-py2.py3-none-any.whl (37 kB)\nBuilding wheels for collected packages: hnswlib, sentence-transformers\n  Building wheel for hnswlib (pyproject.toml) ... \u001b[?25ldone\n\u001b[?25h  Created wheel for hnswlib: filename=hnswlib-0.8.0-cp310-cp310-linux_x86_64.whl size=199239 sha256=5d960a8f07d4c8a3c72e41305d7c2620a2541c04050e37289469f85ca17408aa\n  Stored in directory: /root/.cache/pip/wheels/af/a9/3e/3e5d59ee41664eb31a4e6de67d1846f86d16d93c45f277c4e7\n  Building wheel for sentence-transformers (setup.py) ... \u001b[?25ldone\n\u001b[?25h  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125923 sha256=19fed80857355c66bfcab7138e314add4b5f0f19231ba87cad20586d11ee957f\n  Stored in directory: /root/.cache/pip/wheels/62/f2/10/1e606fd5f02395388f74e7462910fe851042f97238cbbd902f\nSuccessfully built hnswlib sentence-transformers\nInstalling collected packages: monotonic, faiss-cpu, hnswlib, duckdb, clickhouse-connect, posthog, sentence-transformers, chromadb\nSuccessfully installed chromadb-0.3.21 clickhouse-connect-0.6.23 duckdb-0.9.2 faiss-cpu-1.7.4 hnswlib-0.8.0 monotonic-1.6 posthog-3.1.0 sentence-transformers-2.2.2\nNote: you may need to restart the kernel to use updated packages.\n","output_type":"stream"}]},{"cell_type":"code","source":"cache_dir = \"./cache\"","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:11:22.332198Z","iopub.execute_input":"2023-12-28T10:11:22.332842Z","iopub.status.idle":"2023-12-28T10:11:22.337011Z","shell.execute_reply.started":"2023-12-28T10:11:22.332812Z","shell.execute_reply":"2023-12-28T10:11:22.336204Z"},"trusted":true},"execution_count":9,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_column', None)\npd.set_option('display.max_rows', None)\npd.set_option('display.max_seq_items', None)\npd.set_option('display.max_colwidth', 500)\npd.set_option('expand_frame_repr', True)","metadata":{"execution":{"iopub.status.busy":"2023-12-28T09:58:18.178288Z","iopub.execute_input":"2023-12-28T09:58:18.178643Z","iopub.status.idle":"2023-12-28T09:58:18.184422Z","shell.execute_reply.started":"2023-12-28T09:58:18.178613Z","shell.execute_reply":"2023-12-28T09:58:18.183513Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"# <b>2 <span style='color:#78D118'>|</span> Reading data</b>\n\nIn this section, we are going to use the data on <a href=\"https://newscatcherapi.com/\" target=\"_blank\">news topics collected by the NewsCatcher team</a>, who collect and index news articles and release them to the open-source community. The dataset can be downloaded from <a href=\"https://www.kaggle.com/kotartemiy/topic-labeled-news-dataset\" target=\"_blank\">Kaggle</a>.\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\npdf = pd.read_csv(f\"/kaggle/input/topic-labeled-news-dataset/labelled_newscatcher_dataset.csv\", sep=\";\")\npdf[\"id\"] = pdf.index\ndisplay(pdf.head(10))\n","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:01:59.805653Z","iopub.execute_input":"2023-12-28T10:01:59.806038Z","iopub.status.idle":"2023-12-28T10:02:00.349819Z","shell.execute_reply.started":"2023-12-28T10:01:59.806006Z","shell.execute_reply":"2023-12-28T10:02:00.349020Z"},"trusted":true},"execution_count":5,"outputs":[{"output_type":"display_data","data":{"text/plain":"     topic  \\\n0  SCIENCE   \n1  SCIENCE   \n2  SCIENCE   \n3  SCIENCE   \n4  SCIENCE   \n5  SCIENCE   \n6  SCIENCE   \n7  SCIENCE   \n8  SCIENCE   \n9  SCIENCE   \n\n                                                                                                                                         link  \\\n0                                                                          https://www.eurekalert.org/pub_releases/2020-08/dbnl-acl080620.php   \n1                                               https://www.pulse.ng/news/world/an-irresistible-scent-makes-locusts-swarm-study-finds/jy784jw   \n2                 https://www.express.co.uk/news/science/1322607/artificial-intelligence-warning-machine-learning-algorithm-social-media-data   \n3                                                     https://www.ndtv.com/world-news/glaciers-could-have-sculpted-mars-valleys-study-2273648   \n4                                                               https://www.thesun.ie/tech/5742187/perseid-meteor-shower-tonight-time-uk-see/   \n5                                                    https://interestingengineering.com/nasa-releases-in-depth-map-of-beirut-explosion-damage   \n6                                       https://www.thequint.com/tech-and-auto/spacex-nasa-demo-2-rocket-launch-set-for-saturday-how-to-watch   \n7                                                                                               https://www.thespacereview.com/article/4003/1   \n8                                                       https://www.businessinsider.com/greenland-melting-ice-sheet-past-tipping-point-2020-8   \n9  https://www.thehindubusinessline.com/news/science/nasa-invites-engineering-students-to-help-harvest-water-on-mars-moon/article32352915.ece   \n\n                       domain       published_date  \\\n0              eurekalert.org  2020-08-06 13:59:45   \n1                    pulse.ng  2020-08-12 15:14:19   \n2               express.co.uk  2020-08-13 21:01:00   \n3                    ndtv.com  2020-08-03 22:18:26   \n4                   thesun.ie  2020-08-12 19:54:36   \n5  interestingengineering.com  2020-08-08 11:05:45   \n6                thequint.com  2020-05-28 09:09:46   \n7          thespacereview.com  2020-08-10 22:48:23   \n8         businessinsider.com  2020-08-16 00:28:54   \n9    thehindubusinessline.com  2020-08-14 07:43:25   \n\n                                                                                                  title  \\\n0                                               A closer look at water-splitting's solar fuel potential   \n1                                                An irresistible scent makes locusts swarm, study finds   \n2                        Artificial intelligence warning: AI will know us better than we know ourselves   \n3                                                      Glaciers Could Have Sculpted Mars Valleys: Study   \n4  Perseid meteor shower 2020: What time and how to see the huge bright FIREBALLS over UK again tonight   \n5                                                 NASA Releases In-Depth Map of Beirut Explosion Damage   \n6                                      SpaceX, NASA Demo-2 Rocket Launch Set for Saturday: How to Watch   \n7                                                         Orbital space tourism set for rebirth in 2021   \n8                                     Greenland's melting ice sheet has 'passed the point of no return'   \n9                                 NASA invites engineering students to help harvest water on Mars, Moon   \n\n  lang  id  \n0   en   0  \n1   en   1  \n2   en   2  \n3   en   3  \n4   en   4  \n5   en   5  \n6   en   6  \n7   en   7  \n8   en   8  \n9   en   9  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>topic</th>\n      <th>link</th>\n      <th>domain</th>\n      <th>published_date</th>\n      <th>title</th>\n      <th>lang</th>\n      <th>id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>SCIENCE</td>\n      <td>https://www.eurekalert.org/pub_releases/2020-08/dbnl-acl080620.php</td>\n      <td>eurekalert.org</td>\n      <td>2020-08-06 13:59:45</td>\n      <td>A closer look at water-splitting's solar fuel potential</td>\n      <td>en</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>SCIENCE</td>\n      <td>https://www.pulse.ng/news/world/an-irresistible-scent-makes-locusts-swarm-study-finds/jy784jw</td>\n      <td>pulse.ng</td>\n      <td>2020-08-12 15:14:19</td>\n      <td>An irresistible scent makes locusts swarm, study finds</td>\n      <td>en</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>SCIENCE</td>\n      <td>https://www.express.co.uk/news/science/1322607/artificial-intelligence-warning-machine-learning-algorithm-social-media-data</td>\n      <td>express.co.uk</td>\n      <td>2020-08-13 21:01:00</td>\n      <td>Artificial intelligence warning: AI will know us better than we know ourselves</td>\n      <td>en</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>SCIENCE</td>\n      <td>https://www.ndtv.com/world-news/glaciers-could-have-sculpted-mars-valleys-study-2273648</td>\n      <td>ndtv.com</td>\n      <td>2020-08-03 22:18:26</td>\n      <td>Glaciers Could Have Sculpted Mars Valleys: Study</td>\n      <td>en</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>SCIENCE</td>\n      <td>https://www.thesun.ie/tech/5742187/perseid-meteor-shower-tonight-time-uk-see/</td>\n      <td>thesun.ie</td>\n      <td>2020-08-12 19:54:36</td>\n      <td>Perseid meteor shower 2020: What time and how to see the huge bright FIREBALLS over UK again tonight</td>\n      <td>en</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>SCIENCE</td>\n      <td>https://interestingengineering.com/nasa-releases-in-depth-map-of-beirut-explosion-damage</td>\n      <td>interestingengineering.com</td>\n      <td>2020-08-08 11:05:45</td>\n      <td>NASA Releases In-Depth Map of Beirut Explosion Damage</td>\n      <td>en</td>\n      <td>5</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>SCIENCE</td>\n      <td>https://www.thequint.com/tech-and-auto/spacex-nasa-demo-2-rocket-launch-set-for-saturday-how-to-watch</td>\n      <td>thequint.com</td>\n      <td>2020-05-28 09:09:46</td>\n      <td>SpaceX, NASA Demo-2 Rocket Launch Set for Saturday: How to Watch</td>\n      <td>en</td>\n      <td>6</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>SCIENCE</td>\n      <td>https://www.thespacereview.com/article/4003/1</td>\n      <td>thespacereview.com</td>\n      <td>2020-08-10 22:48:23</td>\n      <td>Orbital space tourism set for rebirth in 2021</td>\n      <td>en</td>\n      <td>7</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>SCIENCE</td>\n      <td>https://www.businessinsider.com/greenland-melting-ice-sheet-past-tipping-point-2020-8</td>\n      <td>businessinsider.com</td>\n      <td>2020-08-16 00:28:54</td>\n      <td>Greenland's melting ice sheet has 'passed the point of no return'</td>\n      <td>en</td>\n      <td>8</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>SCIENCE</td>\n      <td>https://www.thehindubusinessline.com/news/science/nasa-invites-engineering-students-to-help-harvest-water-on-mars-moon/article32352915.ece</td>\n      <td>thehindubusinessline.com</td>\n      <td>2020-08-14 07:43:25</td>\n      <td>NASA invites engineering students to help harvest water on Mars, Moon</td>\n      <td>en</td>\n      <td>9</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":"# <b>3 <span style='color:#78D118'>|</span> Vector Library: FAISS</b>","metadata":{}},{"cell_type":"markdown","source":"Vector libraries are often sufficient for small, static data. Since it's not a full-fledged database solution, it doesn't have the CRUD (Create, Read, Update, Delete) support. Once the index has been built, if there are more vectors that need to be added/removed/edited, the index has to be rebuilt from scratch. \n\nThat said, vector libraries are easy, lightweight, and fast to use. Examples of vector libraries are [FAISS](https://faiss.ai/), [ScaNN](https://github.com/google-research/google-research/tree/master/scann), [ANNOY](https://github.com/spotify/annoy), and [HNSM](https://arxiv.org/abs/1603.09320).\n\nFAISS has several ways for similarity search: L2 (Euclidean distance), cosine similarity. You can read more about their implementation on their [GitHub](https://github.com/facebookresearch/faiss/wiki/Getting-started#searching) page or [blog post](https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/). They also published their own [best practice guide here](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index).\n\nIf you'd like to read up more on the comparisons between vector libraries and databases, [here is a good blog post](https://weaviate.io/blog/vector-library-vs-vector-database#feature-comparison---library-versus-database).\n\n#### Feature Comparison - Library versus Database\nThe table below summarizes the differences between vector libraries and databases. This is by no means an exhaustive list of features, and not every library or database has the same features.\n\n<table><thead><tr><th><strong>Feature</strong></th><th><strong>Vector Library</strong></th><th><strong>Vector Database</strong> (Weaviate as an example)</th></tr></thead><tbody><tr><td>Filtering (in combination with Vector Search)</td><td>No</td><td>Yes</td></tr><tr><td>Updatability (CRUD)</td><td>No (some do, e.g. hnswlib)</td><td>Yes</td></tr><tr><td>Incremental importing, concurrent reading while importing</td><td>No (some do, e.g. hnswlib)</td><td>Yes</td></tr><tr><td>Stores objects and vectors</td><td>No</td><td>Yes</td></tr><tr><td>Speed</td><td>Typically faster than full-blown database</td><td>Typically slower than pure library</td></tr><tr><td>Performance optimized for</td><td>In-memory similarity search</td><td>End2end-callstack, including: vector search, object retrieval from persistent storage, optional inverted index filtering, network requests, etc.</td></tr><tr><td>Durability, crash recovery</td><td>No</td><td>Yes</td></tr><tr><td>Persistence</td><td>Only at explicit snapshot</td><td>Immediate (after each insert, update or delete)</td></tr><tr><td>Sharding (i.e. distribute dataset among more than one machine, automatically merge/aggregate results at query time across machines)</td><td>No (there are some 3rd-party projects that wrap sharding around existing libraries)</td><td>Yes</td></tr><tr><td>Replication</td><td>No</td><td>Yes (from v1.17)</td></tr><tr><td>Automated backups (i.e. to and from cloud storage)</td><td>No</td><td>Yes</td></tr><tr><td>Deployment ecosystem (Docker, K8s, Helm, SaaS)</td><td>No (you have to build it yourself)</td><td>Yes</td></tr><tr><td>SDKs / language clients</td><td>No (although most have python bindings)</td><td>Yes (Python, Java, JS, Golang)</td></tr><tr><td>Execution</td><td>Embedded (can be turned into standalone service if you build a simple wrapper app around it)</td><td>Standalone service</td></tr><tr><td>Communication with app</td><td>Foreign Function Interface (FFI) / Language Bindings</td><td>Network calls (HTTP, gRPC, etc.)</td></tr><tr><td>Multi-tenancy</td><td>No</td><td>Yes</td></tr><tr><td>Arbitrary aggregations of stored objects</td><td>No</td><td>Yes</td></tr><tr><td>Module ecosystem (e.g. vectorization, QnA)</td><td>No (have to build it yourself)</td><td>Yes</td></tr><tr><td>Hybrid BM25+dense vector search</td><td>No</td><td>Yes (from v1.17)</td></tr></tbody></table>","metadata":{}},{"cell_type":"markdown","source":"The overall workflow of FAISS is captured in the diagram below. \n\n<img src=\"https://miro.medium.com/v2/resize:fit:1400/0*ouf0eyQskPeGWIGm\" width=1000>\n\nSource: [How to use FAISS to build your first similarity search by Asna Shafiq](https://medium.com/loopio-tech/how-to-use-faiss-to-build-your-first-similarity-search-bf0f708aa772).\n","metadata":{}},{"cell_type":"code","source":"from sentence_transformers import InputExample\n\npdf_subset = pdf.head(1000)\n\ndef example_create_fn(doc1: pd.Series) -> InputExample:\n    \"\"\"\n    Helper function that outputs a sentence_transformer guid, label, and text\n    \"\"\"\n    return InputExample(texts=[doc1])\n\nfaiss_train_examples = pdf_subset.apply(\n    lambda x: example_create_fn(x[\"title\"]), axis=1\n).tolist()","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:07:48.064283Z","iopub.execute_input":"2023-12-28T10:07:48.064634Z","iopub.status.idle":"2023-12-28T10:07:54.464898Z","shell.execute_reply.started":"2023-12-28T10:07:48.064606Z","shell.execute_reply":"2023-12-28T10:07:54.463906Z"},"trusted":true},"execution_count":6,"outputs":[{"name":"stderr","text":"/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3\n  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n","output_type":"stream"}]},{"cell_type":"code","source":"faiss_train_examples[0]","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:12:42.578697Z","iopub.execute_input":"2023-12-28T10:12:42.579354Z","iopub.status.idle":"2023-12-28T10:12:42.585000Z","shell.execute_reply.started":"2023-12-28T10:12:42.579321Z","shell.execute_reply":"2023-12-28T10:12:42.584153Z"},"trusted":true},"execution_count":12,"outputs":[{"execution_count":12,"output_type":"execute_result","data":{"text/plain":"<sentence_transformers.readers.InputExample.InputExample at 0x79bca3b54a30>"},"metadata":{}}]},{"cell_type":"markdown","source":"## Step 1: Vectorize text into embedding vectors</b>\n\nWe will be using `Sentence-Transformers` [library](https://www.sbert.net/) to load a language model to vectorize our text into embeddings. The library hosts some of the most popular transformers on [Hugging Face Model Hub](https://huggingface.co/sentence-transformers).\n\nHere, we are using the `model = SentenceTransformer(\"all-MiniLM-L6-v2\")` to generate embeddings.\n","metadata":{}},{"cell_type":"code","source":"from sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\n    \"all-MiniLM-L6-v2\", \n    cache_folder=cache_dir\n)  # Use a pre-cached model\nfaiss_title_embedding = model.encode(pdf_subset.title.values.tolist())\nlen(faiss_title_embedding), len(faiss_title_embedding[0])","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:11:24.242085Z","iopub.execute_input":"2023-12-28T10:11:24.242439Z","iopub.status.idle":"2023-12-28T10:11:33.553808Z","shell.execute_reply.started":"2023-12-28T10:11:24.242412Z","shell.execute_reply":"2023-12-28T10:11:33.552828Z"},"trusted":true},"execution_count":10,"outputs":[{"output_type":"display_data","data":{"text/plain":".gitattributes:   0%|          | 0.00/1.18k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"b11e55e878344dcf9f2e9803fd92fe37"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"1_Pooling/config.json:   0%|          | 0.00/190 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"4b95c1117383489ebe96f970e6bfacf4"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"README.md:   0%|          | 0.00/10.6k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"84ff92d22cb3415e88b04f3660fe4a6f"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"config.json:   0%|          | 0.00/612 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"a020432b8cb94e6f91f0fbce6461f7cf"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"config_sentence_transformers.json:   0%|          | 0.00/116 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"866c4d821c1e403c88f8749d5e498c44"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"data_config.json:   0%|          | 0.00/39.3k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"a20bc78317914bd598a360c9deed7460"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"pytorch_model.bin:   0%|          | 0.00/90.9M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"0f6bf5ced138438ea594fb25cda5f8d7"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"sentence_bert_config.json:   0%|          | 0.00/53.0 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"71988a5ad91c4ee4beb5b11c65720d10"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"special_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"2083b2ea7de24a248794d8ec4bc72321"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"a00248ee07c84c5683246ea74695af42"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"tokenizer_config.json:   0%|          | 0.00/350 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"6ebfb5b66c3d45418a462a129b3d6622"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"train_script.py:   0%|          | 0.00/13.2k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"192c27851e394bb9b3d84be1d910d27a"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"1dd13f44a23a43ec8fa3dc8b454fcc66"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"modules.json:   0%|          | 0.00/349 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"e68cb5da0e054a4bac6da60ac5af2527"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Batches:   0%|          | 0/32 [00:00<?, ?it/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"bb968f2a60b44afb95010f1d763bee10"}},"metadata":{}},{"execution_count":10,"output_type":"execute_result","data":{"text/plain":"(1000, 384)"},"metadata":{}}]},{"cell_type":"markdown","source":"## Step 2: Saving embedding vectors to FAISS index</b>\n\nBelow, we create the FAISS index object based on our embedding vectors, normalize vectors, and add these vectors to the FAISS index. \n","metadata":{}},{"cell_type":"code","source":"# Import necessary libraries\nimport numpy as np\nimport faiss\n\n# Set 'id' column as the index for the DataFrame 'pdf_subset'\npdf_to_index = pdf_subset.set_index([\"id\"], drop=False)\n\n# Extract the 'id' values from the DataFrame and convert to a flattened numpy array\nid_index = np.array(pdf_to_index.id.values).flatten().astype(\"int\")\n\n# Create a normalized copy of the title embeddings using the L2 normalization\ncontent_encoded_normalized = faiss_title_embedding.copy()\nfaiss.normalize_L2(content_encoded_normalized)\n\n# Initialize an IndexIDMap, which translates search results to IDs\n# Reference: https://faiss.ai/cpp_api/file/IndexIDMap_8h.html#_CPPv4I0EN5faiss18IndexIDMapTemplateE\n# The IndexFlatIP below builds an index with Inner Product (IP) metric\n\n# Explanation: We are creating a FAISS index that is capable of performing similarity searches.\n# The 'content_encoded_normalized' array represents the normalized title embeddings of our data.\n# Each row in this array corresponds to the embedding of a document, and 'id_index' holds the IDs of these documents.\n\n# The normalization step is crucial for ensuring that vector magnitudes do not affect similarity calculations.\n\n# The index is constructed using the Inner Product (IP) metric, which measures the similarity between vectors.\n\n# The IndexIDMap is used to associate each document's embedding with its corresponding 'id' for easy retrieval.\n\nindex_content = faiss.IndexIDMap(faiss.IndexFlatIP(len(faiss_title_embedding[0])))\n\n# Add the normalized embeddings to the index, associating each embedding with its corresponding 'id'\nindex_content.add_with_ids(content_encoded_normalized, id_index)\n","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:14:53.091158Z","iopub.execute_input":"2023-12-28T10:14:53.091912Z","iopub.status.idle":"2023-12-28T10:14:53.132123Z","shell.execute_reply.started":"2023-12-28T10:14:53.091879Z","shell.execute_reply":"2023-12-28T10:14:53.131335Z"},"trusted":true},"execution_count":15,"outputs":[]},{"cell_type":"markdown","source":"## Step 3: Search for relevant documents</b>\n\nWe define a search function below to first vectorize our query text, and then search for the vectors with the closest distance. \n","metadata":{}},{"cell_type":"code","source":"def search_content(query, pdf_to_index, k=3):\n    query_vector = model.encode([query])\n    faiss.normalize_L2(query_vector)\n\n    # We set k to limit the number of vectors we want to return\n    top_k = index_content.search(query_vector, k)\n    ids = top_k[1][0].tolist()\n    similarities = top_k[0][0].tolist()\n    results = pdf_to_index.loc[ids]\n    results[\"similarities\"] = similarities\n    return results","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:50:21.814872Z","iopub.execute_input":"2023-12-28T10:50:21.815953Z","iopub.status.idle":"2023-12-28T10:50:21.823394Z","shell.execute_reply.started":"2023-12-28T10:50:21.815911Z","shell.execute_reply":"2023-12-28T10:50:21.822292Z"},"trusted":true},"execution_count":16,"outputs":[]},{"cell_type":"markdown","source":"Now you can query for similar content, Notice that you did not have to configure any database networks beforehand nor pass in any credentials. FAISS works locally with your code.\n","metadata":{}},{"cell_type":"code","source":"display(search_content(\"animal\", pdf_to_index))","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:51:05.123087Z","iopub.execute_input":"2023-12-28T10:51:05.123454Z","iopub.status.idle":"2023-12-28T10:51:05.171491Z","shell.execute_reply.started":"2023-12-28T10:51:05.123423Z","shell.execute_reply":"2023-12-28T10:51:05.170648Z"},"trusted":true},"execution_count":17,"outputs":[{"output_type":"display_data","data":{"text/plain":"Batches:   0%|          | 0/1 [00:00<?, ?it/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"f89c7ed73ce94eb0b51c07290bd75f7a"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"          topic  \\\nid                \n176  TECHNOLOGY   \n975      HEALTH   \n99   TECHNOLOGY   \n\n                                                                                                                                        link  \\\nid                                                                                                                                             \n176                                  https://www.pushsquare.com/news/2020/08/random_you_can_pick_up_and_pet_cats_in_assassins_creed_valhalla   \n975  https://www.news-medical.net/news/20200813/Researchers-explore-the-social-behavior-of-animals-towards-emerging-infectious-diseases.aspx   \n99                                                                      https://www.gematsu.com/2020/08/ghostwire-tokyo-confirms-dog-petting   \n\n               domain       published_date  \\\nid                                           \n176    pushsquare.com  2020-08-03 16:30:00   \n975  news-medical.net  2020-08-13 05:18:00   \n99        gematsu.com  2020-08-07 16:43:13   \n\n                                                                                  title  \\\nid                                                                                        \n176                   Random: You Can Pick Up and Pet Cats in Assassin's Creed Valhalla   \n975  Researchers explore social behavior of animals toward emerging infectious diseases   \n99                                                Ghostwire: Tokyo confirms dog petting   \n\n    lang   id  similarities  \nid                           \n176   en  176      0.391902  \n975   en  975      0.376784  \n99    en   99      0.344058  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>topic</th>\n      <th>link</th>\n      <th>domain</th>\n      <th>published_date</th>\n      <th>title</th>\n      <th>lang</th>\n      <th>id</th>\n      <th>similarities</th>\n    </tr>\n    <tr>\n      <th>id</th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>176</th>\n      <td>TECHNOLOGY</td>\n      <td>https://www.pushsquare.com/news/2020/08/random_you_can_pick_up_and_pet_cats_in_assassins_creed_valhalla</td>\n      <td>pushsquare.com</td>\n      <td>2020-08-03 16:30:00</td>\n      <td>Random: You Can Pick Up and Pet Cats in Assassin's Creed Valhalla</td>\n      <td>en</td>\n      <td>176</td>\n      <td>0.391902</td>\n    </tr>\n    <tr>\n      <th>975</th>\n      <td>HEALTH</td>\n      <td>https://www.news-medical.net/news/20200813/Researchers-explore-the-social-behavior-of-animals-towards-emerging-infectious-diseases.aspx</td>\n      <td>news-medical.net</td>\n      <td>2020-08-13 05:18:00</td>\n      <td>Researchers explore social behavior of animals toward emerging infectious diseases</td>\n      <td>en</td>\n      <td>975</td>\n      <td>0.376784</td>\n    </tr>\n    <tr>\n      <th>99</th>\n      <td>TECHNOLOGY</td>\n      <td>https://www.gematsu.com/2020/08/ghostwire-tokyo-confirms-dog-petting</td>\n      <td>gematsu.com</td>\n      <td>2020-08-07 16:43:13</td>\n      <td>Ghostwire: Tokyo confirms dog petting</td>\n      <td>en</td>\n      <td>99</td>\n      <td>0.344058</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":"Up until now, we haven't done the last step of conducting Q/A with a language model yet. We are going to demonstrate this with Chroma, a vector database.","metadata":{}},{"cell_type":"markdown","source":"# <b>4 <span style='color:#78D118'>|</span> Vector Database: Chroma</b>","metadata":{}},{"cell_type":"markdown","source":"Chroma is an open-source embedding database. The company just raised its [seed funding in April 2023](https://www.trychroma.com/blog/seed) and is quickly becoming popular to support LLM-based applications. \n","metadata":{}},{"cell_type":"code","source":"import chromadb\nfrom chromadb.config import Settings\n\nchroma_client = chromadb.Client(\n    Settings(\n        chroma_db_impl=\"duckdb+parquet\",\n        persist_directory=cache_dir,  # this is an optional argument. If you don't supply this, the data will be ephemeral\n    )\n)","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:54:21.934169Z","iopub.execute_input":"2023-12-28T10:54:21.934536Z","iopub.status.idle":"2023-12-28T10:54:22.031667Z","shell.execute_reply.started":"2023-12-28T10:54:21.934506Z","shell.execute_reply":"2023-12-28T10:54:22.030890Z"},"trusted":true},"execution_count":19,"outputs":[]},{"cell_type":"markdown","source":"### Chroma Concept: Collection\n\nChroma `collection` is akin to an index that stores one set of your documents. \n\nAccording to the [docs](https://docs.trychroma.com/getting-started): \n> Collections are where you will store your embeddings, documents, and additional metadata\n\nThe nice thing about ChromaDB is that if you don't supply a model to vectorize text into embeddings, it will automatically load a default embedding function, i.e. `SentenceTransformerEmbeddingFunction`. It can handle tokenization, embedding, and indexing automatically for you. If you would like to change the embedding model, read [here on how to do that](https://docs.trychroma.com/embeddings). TLDR: you can add an optional `model_name` argument. \n\nYou can read [the documentation here](https://docs.trychroma.com/usage-guide#using-collections) on rules for collection names.\n","metadata":{}},{"cell_type":"code","source":"collection_name = \"my_news\"\n\n# If you have created the collection before, you need to delete the collection first\nif len(chroma_client.list_collections()) > 0 and collection_name in [chroma_client.list_collections()[0].name]:\n    chroma_client.delete_collection(name=collection_name)\n\nprint(f\"Creating collection: '{collection_name}'\")\ncollection = chroma_client.create_collection(name=collection_name)","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:55:57.069307Z","iopub.execute_input":"2023-12-28T10:55:57.070374Z","iopub.status.idle":"2023-12-28T10:56:00.646235Z","shell.execute_reply.started":"2023-12-28T10:55:57.070338Z","shell.execute_reply":"2023-12-28T10:56:00.645440Z"},"trusted":true},"execution_count":20,"outputs":[{"name":"stdout","text":"Creating collection: 'my_news'\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":".gitattributes:   0%|          | 0.00/1.18k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"dd9fff2ff9e3473eba3a06bf91e4a26b"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"1_Pooling/config.json:   0%|          | 0.00/190 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"123c7c3bdf4f45d7b38cf30e088b70be"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"README.md:   0%|          | 0.00/10.6k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"ac9c5e0f91b74b51a74921e808cebb9e"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"config.json:   0%|          | 0.00/612 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"5021c72c874242a1884ea061289e81f1"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"config_sentence_transformers.json:   0%|          | 0.00/116 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"23688be0a5414969b89f97e4aa391abd"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"data_config.json:   0%|          | 0.00/39.3k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"cd16c7b3a9a149e690ad01d37d9699e0"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"pytorch_model.bin:   0%|          | 0.00/90.9M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"0c72e2ccda77454194fda70fa8d605c4"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"sentence_bert_config.json:   0%|          | 0.00/53.0 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"3945903412b144d78ac1e426d7345027"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"special_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"9931e9665a144f6cac84c7629d38e5e5"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"8a5a5dd8a7af40fc8ee53b4537e01903"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"tokenizer_config.json:   0%|          | 0.00/350 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"b0de4862614640fc890b1ba53573e82f"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"train_script.py:   0%|          | 0.00/13.2k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"19c2b583bf9a45119868b71cd2430f33"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"b5c3073681514adab7d3ae460cf34530"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"modules.json:   0%|          | 0.00/349 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"2bb1930200a0424ea7dbb275e3ff7d2c"}},"metadata":{}}]},{"cell_type":"markdown","source":"## Step 1: Add data to collection\n\nSince we are re-using the same data, we can skip the step of reading data. As mentioned in the text above, Chroma can take care of text vectorization for us, so we can directly add text to the collection and Chroma will convert the text into embeddings behind the scene. \n","metadata":{}},{"cell_type":"code","source":"display(pdf_subset.head(10))","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:56:50.290539Z","iopub.execute_input":"2023-12-28T10:56:50.290908Z","iopub.status.idle":"2023-12-28T10:56:50.304236Z","shell.execute_reply.started":"2023-12-28T10:56:50.290876Z","shell.execute_reply":"2023-12-28T10:56:50.303229Z"},"trusted":true},"execution_count":22,"outputs":[{"output_type":"display_data","data":{"text/plain":"     topic  \\\n0  SCIENCE   \n1  SCIENCE   \n2  SCIENCE   \n3  SCIENCE   \n4  SCIENCE   \n5  SCIENCE   \n6  SCIENCE   \n7  SCIENCE   \n8  SCIENCE   \n9  SCIENCE   \n\n                                                                                                                                         link  \\\n0                                                                          https://www.eurekalert.org/pub_releases/2020-08/dbnl-acl080620.php   \n1                                               https://www.pulse.ng/news/world/an-irresistible-scent-makes-locusts-swarm-study-finds/jy784jw   \n2                 https://www.express.co.uk/news/science/1322607/artificial-intelligence-warning-machine-learning-algorithm-social-media-data   \n3                                                     https://www.ndtv.com/world-news/glaciers-could-have-sculpted-mars-valleys-study-2273648   \n4                                                               https://www.thesun.ie/tech/5742187/perseid-meteor-shower-tonight-time-uk-see/   \n5                                                    https://interestingengineering.com/nasa-releases-in-depth-map-of-beirut-explosion-damage   \n6                                       https://www.thequint.com/tech-and-auto/spacex-nasa-demo-2-rocket-launch-set-for-saturday-how-to-watch   \n7                                                                                               https://www.thespacereview.com/article/4003/1   \n8                                                       https://www.businessinsider.com/greenland-melting-ice-sheet-past-tipping-point-2020-8   \n9  https://www.thehindubusinessline.com/news/science/nasa-invites-engineering-students-to-help-harvest-water-on-mars-moon/article32352915.ece   \n\n                       domain       published_date  \\\n0              eurekalert.org  2020-08-06 13:59:45   \n1                    pulse.ng  2020-08-12 15:14:19   \n2               express.co.uk  2020-08-13 21:01:00   \n3                    ndtv.com  2020-08-03 22:18:26   \n4                   thesun.ie  2020-08-12 19:54:36   \n5  interestingengineering.com  2020-08-08 11:05:45   \n6                thequint.com  2020-05-28 09:09:46   \n7          thespacereview.com  2020-08-10 22:48:23   \n8         businessinsider.com  2020-08-16 00:28:54   \n9    thehindubusinessline.com  2020-08-14 07:43:25   \n\n                                                                                                  title  \\\n0                                               A closer look at water-splitting's solar fuel potential   \n1                                                An irresistible scent makes locusts swarm, study finds   \n2                        Artificial intelligence warning: AI will know us better than we know ourselves   \n3                                                      Glaciers Could Have Sculpted Mars Valleys: Study   \n4  Perseid meteor shower 2020: What time and how to see the huge bright FIREBALLS over UK again tonight   \n5                                                 NASA Releases In-Depth Map of Beirut Explosion Damage   \n6                                      SpaceX, NASA Demo-2 Rocket Launch Set for Saturday: How to Watch   \n7                                                         Orbital space tourism set for rebirth in 2021   \n8                                     Greenland's melting ice sheet has 'passed the point of no return'   \n9                                 NASA invites engineering students to help harvest water on Mars, Moon   \n\n  lang  id  \n0   en   0  \n1   en   1  \n2   en   2  \n3   en   3  \n4   en   4  \n5   en   5  \n6   en   6  \n7   en   7  \n8   en   8  \n9   en   9  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>topic</th>\n      <th>link</th>\n      <th>domain</th>\n      <th>published_date</th>\n      <th>title</th>\n      <th>lang</th>\n      <th>id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>SCIENCE</td>\n      <td>https://www.eurekalert.org/pub_releases/2020-08/dbnl-acl080620.php</td>\n      <td>eurekalert.org</td>\n      <td>2020-08-06 13:59:45</td>\n      <td>A closer look at water-splitting's solar fuel potential</td>\n      <td>en</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>SCIENCE</td>\n      <td>https://www.pulse.ng/news/world/an-irresistible-scent-makes-locusts-swarm-study-finds/jy784jw</td>\n      <td>pulse.ng</td>\n      <td>2020-08-12 15:14:19</td>\n      <td>An irresistible scent makes locusts swarm, study finds</td>\n      <td>en</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>SCIENCE</td>\n      <td>https://www.express.co.uk/news/science/1322607/artificial-intelligence-warning-machine-learning-algorithm-social-media-data</td>\n      <td>express.co.uk</td>\n      <td>2020-08-13 21:01:00</td>\n      <td>Artificial intelligence warning: AI will know us better than we know ourselves</td>\n      <td>en</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>SCIENCE</td>\n      <td>https://www.ndtv.com/world-news/glaciers-could-have-sculpted-mars-valleys-study-2273648</td>\n      <td>ndtv.com</td>\n      <td>2020-08-03 22:18:26</td>\n      <td>Glaciers Could Have Sculpted Mars Valleys: Study</td>\n      <td>en</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>SCIENCE</td>\n      <td>https://www.thesun.ie/tech/5742187/perseid-meteor-shower-tonight-time-uk-see/</td>\n      <td>thesun.ie</td>\n      <td>2020-08-12 19:54:36</td>\n      <td>Perseid meteor shower 2020: What time and how to see the huge bright FIREBALLS over UK again tonight</td>\n      <td>en</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>SCIENCE</td>\n      <td>https://interestingengineering.com/nasa-releases-in-depth-map-of-beirut-explosion-damage</td>\n      <td>interestingengineering.com</td>\n      <td>2020-08-08 11:05:45</td>\n      <td>NASA Releases In-Depth Map of Beirut Explosion Damage</td>\n      <td>en</td>\n      <td>5</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>SCIENCE</td>\n      <td>https://www.thequint.com/tech-and-auto/spacex-nasa-demo-2-rocket-launch-set-for-saturday-how-to-watch</td>\n      <td>thequint.com</td>\n      <td>2020-05-28 09:09:46</td>\n      <td>SpaceX, NASA Demo-2 Rocket Launch Set for Saturday: How to Watch</td>\n      <td>en</td>\n      <td>6</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>SCIENCE</td>\n      <td>https://www.thespacereview.com/article/4003/1</td>\n      <td>thespacereview.com</td>\n      <td>2020-08-10 22:48:23</td>\n      <td>Orbital space tourism set for rebirth in 2021</td>\n      <td>en</td>\n      <td>7</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>SCIENCE</td>\n      <td>https://www.businessinsider.com/greenland-melting-ice-sheet-past-tipping-point-2020-8</td>\n      <td>businessinsider.com</td>\n      <td>2020-08-16 00:28:54</td>\n      <td>Greenland's melting ice sheet has 'passed the point of no return'</td>\n      <td>en</td>\n      <td>8</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>SCIENCE</td>\n      <td>https://www.thehindubusinessline.com/news/science/nasa-invites-engineering-students-to-help-harvest-water-on-mars-moon/article32352915.ece</td>\n      <td>thehindubusinessline.com</td>\n      <td>2020-08-14 07:43:25</td>\n      <td>NASA invites engineering students to help harvest water on Mars, Moon</td>\n      <td>en</td>\n      <td>9</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":"Each document must have a unique `id` associated with it and it is up to you to check that there are no duplicate ids. \n\nAdding data to collection will take some time to run, especially when there is a lot of data. In the cell below, we intentionally write only a subset of data to the collection to speed things up. \n","metadata":{}},{"cell_type":"code","source":"# Add documents to a collection\ncollection.add(\n    # Extract the first 100 titles and convert to a list\n    documents=pdf_subset[\"title\"][:100].tolist(),\n    \n    # Associate each document with its corresponding topic as metadata\n    metadatas=[{\"topic\": topic} for topic in pdf_subset[\"topic\"][:100].tolist()],\n    \n    # Generate unique IDs for each document in the collection\n    ids=[f\"id{x}\" for x in range(100)],\n)\n","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:57:37.341563Z","iopub.execute_input":"2023-12-28T10:57:37.342179Z","iopub.status.idle":"2023-12-28T10:57:37.502638Z","shell.execute_reply.started":"2023-12-28T10:57:37.342147Z","shell.execute_reply":"2023-12-28T10:57:37.501702Z"},"trusted":true},"execution_count":23,"outputs":[{"output_type":"display_data","data":{"text/plain":"Batches:   0%|          | 0/4 [00:00<?, ?it/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"fa83724159304262a5e8ec94ccefc365"}},"metadata":{}}]},{"cell_type":"markdown","source":"## Step 2: Query for 10 relevant documents on \"space\"\n\nWe will return 10 most relevant documents. You can think of `10` as 10 nearest neighbors. You can also change the number of results returned as well. \n","metadata":{}},{"cell_type":"code","source":"import json\n\nresults = collection.query(query_texts=[\"space\"], n_results=10)\n\nprint(json.dumps(results, indent=4))","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:58:40.883055Z","iopub.execute_input":"2023-12-28T10:58:40.883488Z","iopub.status.idle":"2023-12-28T10:58:40.918310Z","shell.execute_reply.started":"2023-12-28T10:58:40.883462Z","shell.execute_reply":"2023-12-28T10:58:40.917294Z"},"trusted":true},"execution_count":24,"outputs":[{"output_type":"display_data","data":{"text/plain":"Batches:   0%|          | 0/1 [00:00<?, ?it/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"e61baa9332454388a10fe7add15624f9"}},"metadata":{}},{"name":"stdout","text":"{\n    \"ids\": [\n        [\n            \"id72\",\n            \"id7\",\n            \"id30\",\n            \"id26\",\n            \"id23\",\n            \"id76\",\n            \"id69\",\n            \"id40\",\n            \"id47\",\n            \"id75\"\n        ]\n    ],\n    \"embeddings\": null,\n    \"documents\": [\n        [\n            \"Beck teams up with NASA and AI for 'Hyperspace' visual album experience\",\n            \"Orbital space tourism set for rebirth in 2021\",\n            \"NASA drops \\\"insensitive\\\" nicknames for cosmic objects\",\n            \"\\u2018It came alive:\\u2019 NASA astronauts describe experiencing splashdown in SpaceX Dragon\",\n            \"Hubble Uses Moon As \\u201cMirror\\u201d to Study Earth\\u2019s Atmosphere \\u2013 Proxy in Search of Potentially Habitable Planets Around Other Stars\",\n            \"Australia's small yet crucial part in the mission to find life on Mars\",\n            \"NASA Astronauts in SpaceX Capsule Splashdown in Gulf Of Mexico\",\n            \"SpaceX's Starship spacecraft saw 150 meters high\",\n            \"NASA\\u2019s InSight lander shows what\\u2019s beneath Mars\\u2019 surface\",\n            \"Alien base on Mercury: ET hunters claim to find huge UFO\"\n        ]\n    ],\n    \"metadatas\": [\n        [\n            {\n                \"topic\": \"TECHNOLOGY\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            }\n        ]\n    ],\n    \"distances\": [\n        [\n            1.225035309791565,\n            1.3089772462844849,\n            1.391038179397583,\n            1.4064617156982422,\n            1.4391298294067383,\n            1.4898790121078491,\n            1.5728241205215454,\n            1.5738130807876587,\n            1.583530068397522,\n            1.5864627361297607\n        ]\n    ]\n}\n","output_type":"stream"}]},{"cell_type":"markdown","source":"## Bonus: Add filter statement\n\nIn addition to conducting relevancy search, we can also add filter statements. Refer to the [documentation](https://docs.trychroma.com/usage-guide#using-where-filters) for more information.\n","metadata":{}},{"cell_type":"code","source":"results = collection.query(query_texts=[\"space\"], where={\"topic\": \"SCIENCE\"}, n_results=10)\n\nprint(json.dumps(results, indent=4))","metadata":{"execution":{"iopub.status.busy":"2023-12-28T10:59:27.890512Z","iopub.execute_input":"2023-12-28T10:59:27.890861Z","iopub.status.idle":"2023-12-28T10:59:27.931676Z","shell.execute_reply.started":"2023-12-28T10:59:27.890836Z","shell.execute_reply":"2023-12-28T10:59:27.930826Z"},"trusted":true},"execution_count":26,"outputs":[{"output_type":"display_data","data":{"text/plain":"Batches:   0%|          | 0/1 [00:00<?, ?it/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"f693e6f3c74649aeacf95e9bbaac9048"}},"metadata":{}},{"name":"stdout","text":"{\n    \"ids\": [\n        [\n            \"id7\",\n            \"id30\",\n            \"id26\",\n            \"id23\",\n            \"id76\",\n            \"id69\",\n            \"id40\",\n            \"id47\",\n            \"id75\",\n            \"id52\"\n        ]\n    ],\n    \"embeddings\": null,\n    \"documents\": [\n        [\n            \"Orbital space tourism set for rebirth in 2021\",\n            \"NASA drops \\\"insensitive\\\" nicknames for cosmic objects\",\n            \"\\u2018It came alive:\\u2019 NASA astronauts describe experiencing splashdown in SpaceX Dragon\",\n            \"Hubble Uses Moon As \\u201cMirror\\u201d to Study Earth\\u2019s Atmosphere \\u2013 Proxy in Search of Potentially Habitable Planets Around Other Stars\",\n            \"Australia's small yet crucial part in the mission to find life on Mars\",\n            \"NASA Astronauts in SpaceX Capsule Splashdown in Gulf Of Mexico\",\n            \"SpaceX's Starship spacecraft saw 150 meters high\",\n            \"NASA\\u2019s InSight lander shows what\\u2019s beneath Mars\\u2019 surface\",\n            \"Alien base on Mercury: ET hunters claim to find huge UFO\",\n            \"SpaceX Crew-1 mission with NASA, first fully operational crewed mission to space to launch in October\"\n        ]\n    ],\n    \"metadatas\": [\n        [\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            },\n            {\n                \"topic\": \"SCIENCE\"\n            }\n        ]\n    ],\n    \"distances\": [\n        [\n            1.3089772462844849,\n            1.391038179397583,\n            1.4064617156982422,\n            1.4391298294067383,\n            1.4898790121078491,\n            1.5728241205215454,\n            1.5738130807876587,\n            1.583530068397522,\n            1.5864627361297607,\n            1.598428726196289\n        ]\n    ]\n}\n","output_type":"stream"}]},{"cell_type":"markdown","source":"## Bonus: Update data in a collection\n\nUnlike a vector library, vector databases support changes to the data so we can update or delete data. \n\nIndeed, we can update or delete data in a Chroma collection. \n","metadata":{}},{"cell_type":"code","source":"collection.delete(ids=[\"id0\"])","metadata":{"execution":{"iopub.status.busy":"2023-12-28T11:00:02.129560Z","iopub.execute_input":"2023-12-28T11:00:02.130339Z","iopub.status.idle":"2023-12-28T11:00:02.139919Z","shell.execute_reply.started":"2023-12-28T11:00:02.130305Z","shell.execute_reply":"2023-12-28T11:00:02.139045Z"},"trusted":true},"execution_count":27,"outputs":[{"execution_count":27,"output_type":"execute_result","data":{"text/plain":"[UUID('7f32105a-861b-4006-83c8-7acbee29303b')]"},"metadata":{}}]},{"cell_type":"markdown","source":"The record with `ids=0` is no longer present.","metadata":{}},{"cell_type":"code","source":"collection.get(\n    ids=[\"id0\"],\n)","metadata":{"execution":{"iopub.status.busy":"2023-12-28T11:00:18.303202Z","iopub.execute_input":"2023-12-28T11:00:18.304152Z","iopub.status.idle":"2023-12-28T11:00:18.311959Z","shell.execute_reply.started":"2023-12-28T11:00:18.304113Z","shell.execute_reply":"2023-12-28T11:00:18.310990Z"},"trusted":true},"execution_count":28,"outputs":[{"execution_count":28,"output_type":"execute_result","data":{"text/plain":"{'ids': [], 'embeddings': None, 'documents': [], 'metadatas': []}"},"metadata":{}}]},{"cell_type":"markdown","source":"We can also update a specific data point.","metadata":{}},{"cell_type":"code","source":"collection.get(\n    ids=[\"id2\"],\n)","metadata":{"execution":{"iopub.status.busy":"2023-12-28T11:00:30.790362Z","iopub.execute_input":"2023-12-28T11:00:30.791066Z","iopub.status.idle":"2023-12-28T11:00:30.799057Z","shell.execute_reply.started":"2023-12-28T11:00:30.791026Z","shell.execute_reply":"2023-12-28T11:00:30.798125Z"},"trusted":true},"execution_count":29,"outputs":[{"execution_count":29,"output_type":"execute_result","data":{"text/plain":"{'ids': ['id2'],\n 'embeddings': None,\n 'documents': ['Artificial intelligence warning: AI will know us better than we know ourselves'],\n 'metadatas': [{'topic': 'SCIENCE'}]}"},"metadata":{}}]},{"cell_type":"code","source":"collection.update(\n    ids=[\"id2\"],\n    metadatas=[{\"topic\": \"TECHNOLOGY\"}],\n)","metadata":{"execution":{"iopub.status.busy":"2023-12-28T11:00:37.841783Z","iopub.execute_input":"2023-12-28T11:00:37.842198Z","iopub.status.idle":"2023-12-28T11:00:37.850778Z","shell.execute_reply.started":"2023-12-28T11:00:37.842165Z","shell.execute_reply":"2023-12-28T11:00:37.849737Z"},"trusted":true},"execution_count":30,"outputs":[]},{"cell_type":"markdown","source":"Updated specific data point.","metadata":{}},{"cell_type":"code","source":"collection.get(\n    ids=[\"id2\"],\n)","metadata":{"execution":{"iopub.status.busy":"2023-12-28T11:00:47.013545Z","iopub.execute_input":"2023-12-28T11:00:47.014176Z","iopub.status.idle":"2023-12-28T11:00:47.021928Z","shell.execute_reply.started":"2023-12-28T11:00:47.014143Z","shell.execute_reply":"2023-12-28T11:00:47.021126Z"},"trusted":true},"execution_count":31,"outputs":[{"execution_count":31,"output_type":"execute_result","data":{"text/plain":"{'ids': ['id2'],\n 'embeddings': None,\n 'documents': ['Artificial intelligence warning: AI will know us better than we know ourselves'],\n 'metadatas': [{'topic': 'TECHNOLOGY'}]}"},"metadata":{}}]},{"cell_type":"markdown","source":"# <b>5 <span style='color:#78D118'>|</span> Prompt engineering for question answering</b>\n\nNow that we have identified documents about space from the news dataset, we can pass these documents as additional context for a language model to generate a response based on them.\n\nWe first need to pick a `text-generation` model. Below, we use a Hugging Face model. You can also use OpenAI as well, but you will need to get an Open AI token and [pay based on the number of tokens](https://openai.com/pricing). \n","metadata":{}},{"cell_type":"code","source":"from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline\n\nmodel_id = \"gpt2\"\ntokenizer = AutoTokenizer.from_pretrained(model_id, cache_dir=cache_dir)\nlm_model = AutoModelForCausalLM.from_pretrained(model_id, cache_dir=cache_dir)\n\npipe = pipeline(\n    \"text-generation\",\n    model=lm_model,\n    tokenizer=tokenizer,\n    max_new_tokens=512,\n    device_map=\"auto\",\n)","metadata":{"execution":{"iopub.status.busy":"2023-12-28T11:02:38.810820Z","iopub.execute_input":"2023-12-28T11:02:38.811903Z","iopub.status.idle":"2023-12-28T11:02:47.529570Z","shell.execute_reply.started":"2023-12-28T11:02:38.811863Z","shell.execute_reply":"2023-12-28T11:02:47.528582Z"},"trusted":true},"execution_count":33,"outputs":[{"output_type":"display_data","data":{"text/plain":"config.json:   0%|          | 0.00/665 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"306e0974377b4934abdad155ce45dd6e"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"vocab.json:   0%|          | 0.00/1.04M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"94ea049da4d4432d85ac6b7eaef6e3e4"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"6d40d32a3b8e40d98d1661954acbc90d"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"tokenizer.json:   0%|          | 0.00/1.36M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"469c6dec848c47baa5ca0f829c82bf64"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"model.safetensors:   0%|          | 0.00/548M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"e08798d78d2043c29a98901c7e701bc5"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"generation_config.json:   0%|          | 0.00/124 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"238b7d862e8145498a8bae0c4769df15"}},"metadata":{}}]},{"cell_type":"markdown","source":"Here's where prompt engineering, which is developing prompts, comes in. We pass in the context in our `prompt_template` but there are numerous ways to write a prompt. Some prompts may generate better results than the others and it requires some experimentation to figure out how best to talk to the model. Each language model behaves differently to prompts. \n\nOur prompt template below is inspired from a [2023 paper on program-aided language model](https://arxiv.org/pdf/2211.10435.pdf). The authors have provided their sample prompt template [here](https://github.com/reasoning-machines/pal/blob/main/pal/prompt/date_understanding_prompt.py).\n\nThe following links also provide some helpful guidance on prompt engineering: \n- [Prompt engineering with OpenAI](https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api)\n- [GitHub repo that compiles best practices to interact with ChatGPT](https://github.com/f/awesome-chatgpt-prompts)\n","metadata":{}},{"cell_type":"code","source":"question = \"What's the latest news on space development?\"\ncontext = \" \".join([f\"#{str(i)}\" for i in results[\"documents\"][0]])\n\nprompt_template = f\"Relevant context: {context}\\n\\n The user's question: {question}\"\nprint(\"Results:\")\nprint(\"\\n\")\ndisplay(results)\n\nprint(\"\\n\")\nprint(\"Results of documents 0:\")\nprint(\"\\n\")\ndisplay(results[\"documents\"][0])\n\nprint(\"\\n\")\nprint(\"Context:\")\nprint(\"\\n\")\n\ndisplay(context)\n\nprint(\"\\n\")\nprint(\"Question:\")\nprint(\"\\n\")\ndisplay(question)\n\nprint(\"\\n\")\nprint(\"Prompt Template:\")\nprint(\"\\n\")\ndisplay(prompt_template)","metadata":{"execution":{"iopub.status.busy":"2023-12-28T11:10:19.178625Z","iopub.execute_input":"2023-12-28T11:10:19.179281Z","iopub.status.idle":"2023-12-28T11:10:19.198583Z","shell.execute_reply.started":"2023-12-28T11:10:19.179249Z","shell.execute_reply":"2023-12-28T11:10:19.197766Z"},"trusted":true},"execution_count":51,"outputs":[{"name":"stdout","text":"Results:\n\n\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"{'ids': [['id7',\n   'id30',\n   'id26',\n   'id23',\n   'id76',\n   'id69',\n   'id40',\n   'id47',\n   'id75',\n   'id52']],\n 'embeddings': None,\n 'documents': [['Orbital space tourism set for rebirth in 2021',\n   'NASA drops \"insensitive\" nicknames for cosmic objects',\n   '‘It came alive:’ NASA astronauts describe experiencing splashdown in SpaceX Dragon',\n   'Hubble Uses Moon As “Mirror” to Study Earth’s Atmosphere – Proxy in Search of Potentially Habitable Planets Around Other Stars',\n   \"Australia's small yet crucial part in the mission to find life on Mars\",\n   'NASA Astronauts in SpaceX Capsule Splashdown in Gulf Of Mexico',\n   \"SpaceX's Starship spacecraft saw 150 meters high\",\n   'NASA’s InSight lander shows what’s beneath Mars’ surface',\n   'Alien base on Mercury: ET hunters claim to find huge UFO',\n   'SpaceX Crew-1 mission with NASA, first fully operational crewed mission to space to launch in October']],\n 'metadatas': [[{'topic': 'SCIENCE'},\n   {'topic': 'SCIENCE'},\n   {'topic': 'SCIENCE'},\n   {'topic': 'SCIENCE'},\n   {'topic': 'SCIENCE'},\n   {'topic': 'SCIENCE'},\n   {'topic': 'SCIENCE'},\n   {'topic': 'SCIENCE'},\n   {'topic': 'SCIENCE'},\n   {'topic': 'SCIENCE'}]],\n 'distances': [[1.3089772462844849,\n   1.391038179397583,\n   1.4064617156982422,\n   1.4391298294067383,\n   1.4898790121078491,\n   1.5728241205215454,\n   1.5738130807876587,\n   1.583530068397522,\n   1.5864627361297607,\n   1.598428726196289]]}"},"metadata":{}},{"name":"stdout","text":"\n\nResults of documents 0:\n\n\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"['Orbital space tourism set for rebirth in 2021',\n 'NASA drops \"insensitive\" nicknames for cosmic objects',\n '‘It came alive:’ NASA astronauts describe experiencing splashdown in SpaceX Dragon',\n 'Hubble Uses Moon As “Mirror” to Study Earth’s Atmosphere – Proxy in Search of Potentially Habitable Planets Around Other Stars',\n \"Australia's small yet crucial part in the mission to find life on Mars\",\n 'NASA Astronauts in SpaceX Capsule Splashdown in Gulf Of Mexico',\n \"SpaceX's Starship spacecraft saw 150 meters high\",\n 'NASA’s InSight lander shows what’s beneath Mars’ surface',\n 'Alien base on Mercury: ET hunters claim to find huge UFO',\n 'SpaceX Crew-1 mission with NASA, first fully operational crewed mission to space to launch in October']"},"metadata":{}},{"name":"stdout","text":"\n\nContext:\n\n\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"'#Orbital space tourism set for rebirth in 2021 #NASA drops \"insensitive\" nicknames for cosmic objects #‘It came alive:’ NASA astronauts describe experiencing splashdown in SpaceX Dragon #Hubble Uses Moon As “Mirror” to Study Earth’s Atmosphere – Proxy in Search of Potentially Habitable Planets Around Other Stars #Australia\\'s small yet crucial part in the mission to find life on Mars #NASA Astronauts in SpaceX Capsule Splashdown in Gulf Of Mexico #SpaceX\\'s Starship spacecraft saw 150 meters high #NASA’s InSight lander shows what’s beneath Mars’ surface #Alien base on Mercury: ET hunters claim to find huge UFO #SpaceX Crew-1 mission with NASA, first fully operational crewed mission to space to launch in October'"},"metadata":{}},{"name":"stdout","text":"\n\nQuestion:\n\n\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"\"What's the latest news on space development?\""},"metadata":{}},{"name":"stdout","text":"\n\nPrompt Template:\n\n\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"'Relevant context: #Orbital space tourism set for rebirth in 2021 #NASA drops \"insensitive\" nicknames for cosmic objects #‘It came alive:’ NASA astronauts describe experiencing splashdown in SpaceX Dragon #Hubble Uses Moon As “Mirror” to Study Earth’s Atmosphere – Proxy in Search of Potentially Habitable Planets Around Other Stars #Australia\\'s small yet crucial part in the mission to find life on Mars #NASA Astronauts in SpaceX Capsule Splashdown in Gulf Of Mexico #SpaceX\\'s Starship spacecraft saw 150 meters high #NASA’s InSight lander shows what’s beneath Mars’ surface #Alien base on Mercury: ET hunters claim to find huge UFO #SpaceX Crew-1 mission with NASA, first fully operational crewed mission to space to launch in October\\n\\n The user\\'s question: What\\'s the latest news on space development?'"},"metadata":{}}]},{"cell_type":"code","source":"lm_response = pipe(prompt_template)\nprint(lm_response[0][\"generated_text\"])","metadata":{"execution":{"iopub.status.busy":"2023-12-28T11:11:32.873364Z","iopub.execute_input":"2023-12-28T11:11:32.873714Z","iopub.status.idle":"2023-12-28T11:11:53.133651Z","shell.execute_reply.started":"2023-12-28T11:11:32.873685Z","shell.execute_reply":"2023-12-28T11:11:53.132728Z"},"trusted":true},"execution_count":53,"outputs":[{"name":"stderr","text":"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n","output_type":"stream"},{"name":"stdout","text":"Relevant context: #Orbital space tourism set for rebirth in 2021 #NASA drops \"insensitive\" nicknames for cosmic objects #‘It came alive:’ NASA astronauts describe experiencing splashdown in SpaceX Dragon #Hubble Uses Moon As “Mirror” to Study Earth’s Atmosphere – Proxy in Search of Potentially Habitable Planets Around Other Stars #Australia's small yet crucial part in the mission to find life on Mars #NASA Astronauts in SpaceX Capsule Splashdown in Gulf Of Mexico #SpaceX's Starship spacecraft saw 150 meters high #NASA’s InSight lander shows what’s beneath Mars’ surface #Alien base on Mercury: ET hunters claim to find huge UFO #SpaceX Crew-1 mission with NASA, first fully operational crewed mission to space to launch in October\n\n The user's question: What's the latest news on space development? As such, we've listed a timeline so you may make it part of your experience with the space shuttle mission. We take your question seriously. If you don't consider it a threat to the future of the entire U.S. space program, let's have some fun:1.The first manned space exploration by an American citizen has taken place in 1965. This is no ordinary manned mission. It has taken the lives of tens of thousands of people, including hundreds of astronauts and cosmonauts. This has created the unique opportunity to explore Mars 2.The American space capsule is based on the S-class, high-speed rocket motor that is the most advanced of the three. Since we are now only a year into a manned mission, it's not possible to start on schedule. NASA and the American Space Agency (ACSA) have managed to establish a lunar module using lunar lander technology. This is the first American lunar lander. The first American space shuttle ever to reach Mars, the Challenger carried over 3,000 scientists to Mars. NASA spent over two yearspeeging for the lander with the goal of increasing its capability and funding. NASA has even started developing the first commercial craft that can reach Mars; the Orion rocket which would give it new powers of flight. With this capability, American entrepreneurs have been testing their landing craft on Earth for an extended period of time. We're thrilled to announce the inaugural Orion rocket – the culmination of a decade of research. Orion is designed to be launched in 2015 with a successful launch and landing with the Mars rover on the way.2.The first manned spaceflight has been launched in 1965 by James Webb Space Telescope. This was the first space flight to be successful by any American citizen. In 1965, the Apollo 11 spacecraft successfully flew for ten days from Cape Canaveral Air Force Station, Florida. This was the first manned spaceflight to ever land on an asteroid or comet. This mission was a major milestone in the development of NASA's mission to Venus.3.In 1961 we launched the first U.S. space shuttle. U.S. Ambassador to the United Nations Martin Luther King Jr. (agonscar driver) was assassinated. This is the last public space shuttle ever to launch. This is the last U.S. space launch mission to have a crew of five and be completed on time. It was a milestone that had already been recognized by the Kennedy Space Center of the Kennedy Space Center. The shuttle program ended just two years before spacecraft made their first flight.4\n","output_type":"stream"}]},{"cell_type":"markdown","source":"Just completed the implementation of your first text vectorization, search, and question answering workflow (that requires prompt engineering)\n","metadata":{}}]}