Promote and demote inherited classes.

[amcgregor]

Copied the year-old ticket from hmarr/mongoengine.

A feature I could really, really use is class promotion and demotion when using inheritance. As an example in BDD style:

• Given three classes: Asset(Document), Folder(Asset), Gallery(Folder).
• And given an instance of Folder.
  • When the system promotes the Folder instance to Gallery.
    • Then a new instance of type Gallery should be returned.
    • And then old instances (from caches, say) should be invalidated. [Attempts to use these should result in exceptions.]
    • And then data values common to both classes should be transferred. [E.g. name is common to both.]
  • When the system demotes the Folder instance to Asset. [This may not be specific; if a class isn't specified it defaults to the immediate parent class.]
    • Then a new instance of type Asset should be returned.
    • And then old instances (from caches, say) should be invalidated.
    • And then data values common to both classes should be transferred.
    • And then data values not common should optionally be removed from the underlying record. [This is not possible when promoting as you can't "delete" existing definitions, only override them. E.g. the view (list, icon, etc.) would be deleted.]

The first is a real use-case. ;) The second is needed for the use case of demoting a User subclass instance back to User, then promoting it back up to a different subclass in order to change the capabilities and data-based "role" of the given user.

There is also the possibility of automatically working out (via type(Foo).mro() inspection) the nearest common root between two arbitrary classes, then working down and back up to mutate more widely. If the common root is Document, this is an error (unless very explicitly requested) as the classes are not related at all and thus no data would be preserved during migration.

Redefinition of a field with differing types between a parent and child class would be a bad thing.

Currently I have to run some raw MongoDB queries to update the _cls and _types references. While this works, it's butt-ugly and prone to user error.

Here are some code snippets:

// Demote
db.foo.update({_id: ObjectId('…')}, {$set: {_cls: 'Foo'}, $pop: {_types: 1}}, false, false);

// Promote
db.foo.update({_id: ObjectId('…')}, {$set: {_cls: 'Foo.Bar'}, $push: {_types: 'Bar'}}, false, false);

[amcgregor]

This issue is now nearly a year and a half old (combined across both old and new GitHub repositories), and would still be a really awesome feature.

[rozza]

This might be easier to accomplish in 0.8 as _types is being removed, so all that is needed is updating _cls

I'm maxed so pull requests really help and if your interested in helping drive mongoengine forward then that would be great also.

[Hugh-wong]

vote up.

[amcgregor]

Any hope for a 2015 implementation?

[mmellison]

I very much like this suggestion. I have several use-cases for this in some current projects which could easily benefit from this feature.

I recently requested to become a full maintainer of the project, and once I can get some direction from the other maintainers as to the plans for the 0.9, I would love to add this to my feature list to work on.

[wojcikstefan]

I'd love to hear more about real-life scenarios in which this would be used. @seglberg @amcgregor @Hugh-wong could you elaborate on the specific scenarios you'd use this in?

Technically, nothing stops you from updating the _cls of a given object. It might leave some redundant (demotion) or empty (promotion) fields in MongoDB, which would need to be handled manually, but other than that it should work just fine. Are you simply looking for a cleaner abstraction of this process?

[amcgregor]

@wojcikstefan Beyond the fact that the initial description four years ago was excessively detailed and covered two distinct use cases, no. I've accepted the fact that MongoEngine is stagnant and broken, and have moved on; ref. that Contentment ticket and the fact that I now maintain a competing ODM. (This particular issue is to be addressed in 1.2.0 ~January, ref the placeholder PR: marrow/mongo#55)

[wojcikstefan]

Thanks for getting back to me @amcgregor , marrow.mongo looks cool! It's unfortunate this ticket hasn't been addressed properly in this package. I myself just started contributing to MongoEngine and I'm going through the old tickets trying to see what's still relevant.

For everybody else interested in this issue, please provide more use cases - the more explicit the better. Personally I still don't see the benefits outweighing the cost of added complexity (especially if cache/object invalidation would indeed be necessary), and my gut feeling is that the need for promoting/demoting classes might mainly spur from poorly modeled data. I might be oblivious to some valid scenarios though, so please keep the requests coming :)

The second is needed for the use case of demoting a User subclass instance back to User, then promoting it back up to a different subclass in order to change the capabilities and data-based "role" of the given user.

This is a good example where IMO having different user roles as different subclasses might be an abuse of the inheritance system, especially if the roles can be upgraded/downgraded. There should be a better/simpler/more normalized way to model this.

[Kobnar]

Not to necro-bump, but I have a use case:

I am writing a personal citation management tool that keeps track of metadata for various sources (books, articles, podcasts) of various types. MongoDB/MongoEngine is a good fit for this project because I can have all of these sources in a single "Sources" collection.

My inheritance model looks something like this:

sources
 +-- text
 |   +-- books
 |   +-- journals
 |   +-- articles
 |   +-- (...)
 +-- media
 |    +-- audio
 |    |  +-- podcasts
 |    |  +-- (...)
 |    +-- video
 |    +-- (...)
 + (...)

Obviously, different sources can have arbitrarily and sometimes radically different fields. All "media" has some kind of run time length, for example, but no page numbers. Books have authors, but so do introductions. Articles might have one or more authors and reference back to a journal.

Having all of these in a single collection allows me to do complex query operations on the entire collection, but in order to invoke specific methods for each type I have to cast between them first. As one example, I have a standard ISBN-13 field for books with a @property that converts between that and an ISBN-10 format so I don't have to effectively duplicate that specific field.

In my specific case, I am serving this through an API using Pyramid's traversal system, which is otherwise an absolute dream. The only hiccup is that my traversal tree is tightly associated with the base collection type. I have a single SourceResource that performs CRUD ops on the entire source collection, but if a user wants to create, retrieve, or update a sub-class this approach gets annoyingly verbose.

Since this is obviously an edge case, I'll take a crack at it. Good to know there's a specific (and open) ticket though :)

[wojcikstefan]

Hi @Kobnar, thanks you very much for a detailed description (and please necrobump by all means!) :)

Do I understand correctly that your primary issue is that you can't expose API endpoints that allow you to traverse and perform CRUD operations on a specific subtype of a source? If so, I bet there's a way to do it in Pyramid - we do something similar at my company where we have one big activity collection with a bunch of different subclasses (call, email, sms, note, etc.) and we expose particular subtypes as /api/v1/activity/call/, /api/v1/activity/email/, etc. Under the hood that simply means that a queryset with a different filter is used for each resource (e.g. Activity.objects.filter(_cls='Activity.Call')).

Where does the promotion/demotion come in?

[Kobnar]

@wojcikstefan I think you pretty much nailed it on all counts.

TBH I already forgot the specific thing I needed to do with this kind of promotion/demotion. It seems easier to shift around my own application to manage this kind of thing, rather than dig through mongoengine to figure out a highly generalization solution. Besides, sub-types are never changing in my application (especially not between branches).

Thanks for the tip about querying the _cls field, also. That never really occurred to me 👍

If I run across a better, more specific example of whatever blocker I thought I had I'll post it. Otherwise I understand this is a pretty low-priority feature.

[bagerard]

Issue is quite old and didn't receive much attention in the last years, let's close this.

[amcgregor]

After giving up on MongoEngine a while ago—this issue is from 2011 (a year before the switch to GitHub issues!)—I implemented this functionality myself. Not entirely complete (e.g. doesn't "clean up" the data in any way), but sufficient to get started.

[bagerard]

k, thanks @amcgregor If we ever consider adding this feature to MongoEngine, we'll make sure to look into it.