Drop/create Mongo index before/after transformation, and fix import…

… bug
microbiomedata · Nov 13, 2024 · 4ca46b1 · 4ca46b1
1 parent cab6cb9
commit 4ca46b1
Showing 1 changed file with 53 additions and 1 deletion.
diff --git a/demo/metadata_migration/notebooks/migrate_11_0_3_to_11_1_0.ipynb b/demo/metadata_migration/notebooks/migrate_11_0_3_to_11_1_0.ipynb
@@ -175,7 +175,7 @@
     "from bookkeeper import Bookkeeper, MigrationEvent\n",
     "\n",
     "# Dynamic imports:\n",
-    "migrator_module = importlib.import_module(MIGRATOR_MODULE_NAME, package=\"nmdc_schema.migrators\")\n",
+    "migrator_module = importlib.import_module(f\".{MIGRATOR_MODULE_NAME}\", package=\"nmdc_schema.migrators\")\n",
     "Migrator = getattr(migrator_module, \"Migrator\")  # gets the class\n",
     "assert isclass(Migrator), \"Failed to import Migrator class.\""
    ]
@@ -505,6 +505,32 @@
     "print(f\"\\nReturn code: {completed_process.returncode}\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "84a5d514",
+   "metadata": {},
+   "source": [
+    "### Do pre-transformation steps\n",
+    "\n",
+    "In this section, we'll delete an index that is not consistent with the new schema—it is a compound index that includes a field that the migrators will be removing from all documents in the collection.\n",
+    "\n",
+    "#### References\n",
+    "\n",
+    "| Description         | Link                                                                                                          |\n",
+    "|---------------------|---------------------------------------------------------------------------------------------------------------|\n",
+    "| `drop_index` method | https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.drop_index |"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "21bb1ba2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "transformer_mongo_client[cfg.transformer_mongo_database_name].get_collection(\"functional_annotation_agg\").drop_index(\"gene_function_id_1_metagenome_annotation_id_1\")"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "c3e3c9c4",
@@ -544,6 +570,32 @@
     "migrator.upgrade()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "d568ed23",
+   "metadata": {},
+   "source": [
+    "### Do post-transformation steps\n",
+    "\n",
+    "In this section, we'll create an index that is just like the one we deleted earlier, but uses the new field name instead of the old one.\n",
+    "\n",
+    "#### References\n",
+    "\n",
+    "| Description           | Link                                                                                                            |\n",
+    "|-----------------------|-----------------------------------------------------------------------------------------------------------------|\n",
+    "| `create_index` method | https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.create_index |"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ce460536",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "transformer_mongo_client[cfg.transformer_mongo_database_name].get_collection(\"functional_annotation_agg\").create_index([\"gene_function_id\", \"was_generated_by\"], unique=True)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "4c090068",