Move document_index to document_index dir

.pre-commit-config.yaml

@@ -31,4 +31,4 @@ repos:
     hooks:
       - id: codespell
         args: ["-L newyorker"]
-        exclude: '^(poetry\.lock|log-viewer/.*|tests/retrievers/test_document_index\.py)$'
+        exclude: '^(poetry\.lock|log-viewer/.*|tests/connectors/retrievers/test_document_index_retriever\.py)$'

README.md

@@ -21,7 +21,8 @@ The key features of the Intelligence Layer are:
 3. [How to use this in your project](#how-to-make-your-own-use-case)
 4. [Use-case index](#use-case-index)
 5. [How to make your own use-case](#how-to-make-your-own-use-case)
-6. [License](#license)
+6. [References](#references)
+7. [License](#license)
 
 ### How to make your own
 

src/examples/document_index.ipynb

@@ -73,9 +73,17 @@
    "outputs": [],
    "source": [
     "# change this value if you want to use a collection of a different name\n",
+    "from intelligence_layer.connectors.document_index.document_index import CollectionPath\n",
+    "\n",
+    "\n",
     "COLLECTION = \"demo\"\n",
     "\n",
-    "document_index.create_collection(namespace=NAMESPACE, collection=COLLECTION)"
+    "collection_path = CollectionPath(\n",
+    "    namespace=NAMESPACE,\n",
+    "    collection=COLLECTION\n",
+    ")\n",
+    "\n",
+    "document_index.create_collection(collection_path)"
    ]
   },
   {
@@ -170,8 +178,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from intelligence_layer.connectors.document_index.document_index import DocumentContents, DocumentPath\n",
+    "\n",
+    "\n",
     "for doc in documents:\n",
-    "    document_index.add_document(namespace=NAMESPACE, collection=COLLECTION, name=doc[\"name\"], content=doc[\"content\"])"
+    "    document_path = DocumentPath(collection_path=collection_path, document_name=doc[\"name\"])\n",
+    "    document_index.add_document(document_path, contents=DocumentContents.from_text(doc[\"content\"]))"
    ]
   },
   {
@@ -187,28 +199,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "document_index.list_documents(namespace=NAMESPACE, collection=COLLECTION)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Once the documents are uploaded, they are split into chunks.\n",
-    "Chunks are the subparts of the original texts, usually a few hundred tokens long.\n",
-    "You can think of a chunk as a paragraph of text.\n",
-    "While uploading, the DI splits each document into chunks and generates one embedding each.\n",
-    "\n",
-    "Let's see what a chunked document looks like:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "document_index.get_document(namespace=NAMESPACE, collection=COLLECTION, name=\"robert_moses\", get_chunks=True)"
+    "document_index.list_documents(collection_path)"
    ]
   },
   {

src/intelligence_layer/connectors/__init__.py

@@ -1,4 +1,4 @@
-from .document_index import DocumentIndex
+from .document_index.document_index import DocumentIndex
 from .retrievers.base_retriever import BaseRetriever, Document, SearchResult
 from .retrievers.document_index_retriever import DocumentIndexRetriever
 from .retrievers.qdrant_in_memory_retriever import (