mongoengine.connection.ConnectionFailure: You have not defined a default connection

[KaerbEmEvig]

Trying to upgrade from mongoengine==0.10.5 to mongoengine==0.24.2. I get the following issue in one of the tests.

Traceback:

File "/usr/local/lib/python3.6/dist-packages/mongoengine/fields.py", line 958, in __get__
return super().__get__(instance, owner)
File "/usr/local/lib/python3.6/dist-packages/mongoengine/base/fields.py", line 310, in __get__
ref_values=ref_values, instance=instance, name=self.name, max_depth=1
File "/usr/local/lib/python3.6/dist-packages/mongoengine/base/fields.py", line 281, in _lazy_load_refs
name=name,
File "/usr/local/lib/python3.6/dist-packages/mongoengine/dereference.py", line 102, in __call__
self.object_map = self._fetch_objects(doc_type=doc_type)
File "/usr/local/lib/python3.6/dist-packages/mongoengine/dereference.py", line 197, in _fetch_objects
references = get_db()[collection].find({"_id": {"$in": refs}})
File "/usr/local/lib/python3.6/dist-packages/mongoengine/connection.py", line 380, in get_db
conn = get_connection(alias)
File "/usr/local/lib/python3.6/dist-packages/mongoengine/connection.py", line 282, in get_connection
raise ConnectionFailure(msg)
mongoengine.connection.ConnectionFailure: You have not defined a default connection

MRE:

class FirstDoc(mongoengine.DynamicDocument):
    field = mongoengine.StringField(required=True)
    meta = {"db_alias": "my_db"}

class SecondDoc(mongoengine.DynamicDocument):
    field = mongoengine.ListField(required=True)
    meta = {"db_alias": "my_db"}

fd = FirstDoc(field="abc")
fd.save()
sd = SecondDoc(field=[fd])
sd.save()
print(sd.field)

In mongoengine==0.24.2:

In [20]: sd.to_mongo()
Out[20]: SON([('_id', ObjectId('670e37b84bd9cb6fcd25ded8')), ('field', [DBRef('first_doc', ObjectId('670e37ae4bd9cb6fcd25ded7'))])])

In mongoengine==0.10.5:

In [9]: sd.to_mongo()
Out[9]: SON([('_id', ObjectId('670e3b86f45b3d002a4850b9')), ('field', [SON([('_cls', 'FirstDoc'), ('_ref', DBRef('first_doc', ObjectId('670e3b86f45b3d00
2a4850b8')))])])])

As visible above, 0.10.5 embeds the DBRef inside a SON object, while 0.24.2 stores the DBRef bare.

RCA:

https://github.com/MongoEngine/mongoengine/blob/v0.24.2/mongoengine/dereference.py
In 0.24.2 DeReference._find_references(), we enter and add items at line 151 instead of 153, using the collection name instead of the document class as the key in our reference map.

This means that ref_document_cls_exists is False in _fetch_objects(), hence we try to get the references with references = get_db()[collection].find({"_id": {"$in": refs}}), which results in the aforementioned error.

https://github.com/MongoEngine/mongoengine/blob/v0.10.5/mongoengine/dereference.py
In 0.10.5 DeReference._find_references(), we enter at line 116 instead of 114, using the document class as the key.

This means that hasattr(collection, 'objects') is True in _fetch_objects(), hence we get the references with references = collection.objects.in_bulk(refs), which has access to the "db_alias" inside the meta attribute of the document class.

Question

Is this an issue with my document classes or an actual bug in mongoengine?

[KaerbEmEvig]

RCA cont.

Upon further investigation, the discrepancy in behaviour stems from the following change in implementation:

https://github.com/MongoEngine/mongoengine/blob/v0.10.5/mongoengine/base/document.py#L40

        if self._dynamic:
            dynamic_data = {}
            for key, value in values.iteritems():
                if key in self._fields or key == '_id':
                    setattr(self, key, value)  # The value assignment takes place here.
                elif self._dynamic:
                    dynamic_data[key] = value

https://github.com/MongoEngine/mongoengine/blob/v0.24.2/mongoengine/base/document.py#L65

        # Set actual values
        dynamic_data = {}
        FileField = _import_class("FileField")
        for key, value in values.items():
            field = self._fields.get(key)
            if field or key in ("id", "pk", "_cls"):
                if __auto_convert and value is not None:
                    if field and not isinstance(field, FileField):
                        value = field.to_python(value)  # The value assignment takes place here.
                setattr(self, key, value)
            else:
                if self._dynamic:
                    dynamic_data[key] = value
                else:
                    # For strict Document
                    self._data[key] = value

Is the conversion using to_python() intended? Setting the __auto_convert to False alleviates the issue but does not look to be a feasible solution.

[KaerbEmEvig]

Downgrading (still upgrading but not as high) to mongoengine==0.20.0 makes the issue disappear.

[KaerbEmEvig]

Is this an issue with the fact that FirstDoc is a DynamicDocument instead of being a DynamicEmbeddedDocument?