MontyStore cannot be used with a pre-existing local DB

[Andrew-S-Rosen]

MontyStore relies on a DB written to the local disk. However, it'd be nice to be able to use MontyStore to read a pre-existing SQLite database (for instance) in a more Mongo-friendly way. I tried that below without luck.

from maggma.stores.mongolike import MontyStore
ms = MontyStore("cow", "/home/rosen/.local/share/covalent/dispatcher_db.sqlite")
ms.connect()

That returns

---------------------------------------------------------------------------
FileExistsError                           Traceback (most recent call last)
Cell In[5], line 1
----> 1 ms.connect()

File ~/software/miniconda/envs/quacc/lib/python3.9/site-packages/maggma/stores/mongolike.py:948, in MontyStore.connect(self, force_reset)
    940 """
    941 Connect to the database store.
    942 
    943 Args:
    944     force_reset: Force connection reset.
    945 """
    946 from montydb import set_storage, MontyClient  # type: ignore
--> 948 set_storage(self.database_path, storage=self.storage, **self.storage_kwargs)
    949 client = MontyClient(self.database_path, **self.client_kwargs)
    950 if not self._coll or force_reset:

File ~/software/miniconda/envs/quacc/lib/python3.9/site-packages/montydb/configure.py:232, in set_storage(repository, storage, mongo_version, use_bson, **kwargs)
    229 setup = os.path.join(repository, _storage_ident_fname)
    231 if not os.path.isdir(repository):
--> 232     os.makedirs(repository)
    234 with open(setup, "w") as fp:
    235     fp.write(storage)

File ~/software/miniconda/envs/quacc/lib/python3.9/os.py:225, in makedirs(name, mode, exist_ok)
...
    227     # Cannot rely on checking for EEXIST, since the operating system
    228     # could give priority to other errors like EACCES or EROFS
    229     if not exist_ok or not path.isdir(name):

FileExistsError: [Errno 17] File exists: '/home/rosen/.local/share/covalent/dispatcher_db.sqlite'

I'm happy to share a sample SQLite database if required for reproducing the error.

[rkingsbury]

Thanks for reporting @arosen93 ; I agree this should work. Looking at the montdby docs, it's not totally clear to me what the right way to connect to a pre-existing database is within montydb itself. Can you see if the following code works, when "/db/repo" contains a pre-existing file?

from montydb import set_storage, MontyClient

set_storage("/db/repo", storage="sqlite")  # required, to set sqlite as engine
client = MontyClient("/db/repo")

Once we know how this needs to look within montydb it should be easy enough to implement a fix in maggma

[Andrew-S-Rosen]

This is the traceback:

In [1]: from montydb import set_storage, MontyClient
   ...:
   ...: set_storage("dispatcher_db.sqlite", storage="sqlite")  # required, to set sqlit
   ...: e as engine
   ...: client = MontyClient("dispatcher_db.sqlite")
---------------------------------------------------------------------------
FileExistsError                           Traceback (most recent call last)
Cell In[1], line 3
      1 from montydb import set_storage, MontyClient
----> 3 set_storage("dispatcher_db.sqlite", storage="sqlite")  # required, to set sqlite as engine
      4 client = MontyClient("dispatcher_db.sqlite")

File ~/software/miniconda/envs/quacc/lib/python3.9/site-packages/montydb/configure.py:232, in set_storage(repository, storage, mongo_version, use_bson, **kwargs)
    229 setup = os.path.join(repository, _storage_ident_fname)
    231 if not os.path.isdir(repository):
--> 232     os.makedirs(repository)
    234 with open(setup, "w") as fp:
    235     fp.write(storage)

File ~/software/miniconda/envs/quacc/lib/python3.9/os.py:225, in makedirs(name, mode, exist_ok)
    223         return
    224 try:
--> 225     mkdir(name, mode)
    226 except OSError:
    227     # Cannot rely on checking for EEXIST, since the operating system
    228     # could give priority to other errors like EACCES or EROFS
    229     if not exist_ok or not path.isdir(name):

FileExistsError: [Errno 17] File exists: 'dispatcher_db.sqlite'

I'm not sure how it works in montydb either! The joys of only ever thinking about maggma. I'll take a look at the upstream montydb code and see if I can figure that out.

Linking: davidlatwe/montydb#78

[Andrew-S-Rosen]

@rkingsbury: Oops, that's because I can't follow instructions. Your suggestion does work! But note it's the directory to the SQLite database and not the SQLite file. Your code snippet from their README does indeed create a valid MontyClient.

Edit: Thought using the directory fixed it but it returns a store with 0 documents. Will report back once I have more.

[Andrew-S-Rosen]

@munrojm, @rkingsbury: Okay, we got the answer from the source themself.

As described in davidlatwe/montydb#78, montydb cannot be used to read an existing SQLite database and make it mongo-like (unless it was made with montydb, presumably). It can only be used to write out a (new) local SQLite database to the disk and let the user interact with it mongo-style. So, this just isn't a feature that's available in the MontyStore at the moment, although it'd be cool!

In the linked PR, I clarified the docstring which will close this issue.