How mature is this project?

[rittersport3]

Hey, hi!
I really liked the Marrow Mongo approach.

We have been bitten several times by mongoengine and were considering migrating to something else...

• Is Marrow Mongo used on production? By many people/companies? In which use cases?
• How confident are you of it correctness, currently?
• How was your experience migrating from Mongoengine?

I just want to learn more about the project and see if is a fit for us. I am sorry for the trouble!

Thanks and congratulations!

[amcgregor]

Howdy, and thank you for your interest in our project! Having used MongoEngine ourselves for a number of years, we were quite torn when it began to deteriorate after 0.9, failing to keep up with MongoDB feature support and introducing all sorts of crazy edge cases. (The losing the limit after you get the .count() of a cursor was a doozie!) We only wrote Marrow Mongo after much careful consideration on approaches and desired features—too-tight coupling, turns out, is a bad idea.

1. It is certainly used in production, on about half a dozen applications at Illico Hodes including CMS and a vertical market B2B job offer distribution platform. Due to my employment at Illico Hodes, and the fact that much of Marrow Mongo's development is directly driven by business needs, the project (and collective a as a whole) is effectively commercially sponsored.

2. Exceptionally confident. Please refer to the manual section on Code Quality for a breakdown of the reasons why, which include exceptional test coverage (100% prior to any release), careful planning, continuous integration, and code quality tracking. Experimental features are clearly marked as such and isolated from stable components (such as the Queryable trait… that one we need to be careful with to make sure we don't end up another MongoEngine ourselves…) and everything is modular, with a "take as little as you need" approach instead of a "kitchen sink" one. (Document itself has no idea what an _id field or collection is, and contains no non-DAO methods at all, and itself does not perform any form of implicit subclass tracking or registration, that's what explicit entry_points are for.) This will only improve as we continue to add more tests which interact with Pymongo itself instead of just unit testing components in isolation. Some of these tests (such as for capped collection tailing, and use of the Collection and Queryable traits) are already present.

3. Amazingly trivial for the most part. When we started using MongoEngine at work was prior to compression being added, so we remapped all long attribute names in Python to single-character MongoDB field names. In the quick migration we literally just dropped Field from the ends of all fields class names, changed a few imports, and tweaked some of the metadata… and that's about it. Most field semantics (defaults, required, choices, etc.) are the same or require only minimal modification. Even the parametric helpers (U, F, etc.) follow the same MongoEngine query/update syntax, primarily provided as a migration mechanism—query fragment generation is about 100x faster. A properly defined model utilizes methods in a defined internal API to contain the business logic and manipulate those records, this kept the "brain damage" as isolated as we could make it, making it much easier to migrate. Some of the things we had crazy custom QuerySet classes to do before are simple plain object subclasses now just to hold the search methods. I added the Period field to Marrow Mongo in response to our at-work pre-aggregate statistical calculation needs, tracking clicks and other events in one hour periods. Also easy. ;)

4. Adding a fourth point, here, in that switching to Marrow Mongo has also given our model way, way more freedom, flexibility, and agility. For example, e-mail address and URL validation in MongoEngine is somewhat misplaced. It took an afternoon to construct a Link field type in Marrow Mongo that can contain any URI, URL, or URN, of any protocol (optionally restricted, e.g. to only mailto), with a very rich object representation of of those links. (E.g. manipulate query string arguments prior to rendering.) Constructing new derivative types is nearly trivial. The encouragement of use of Pymongo standard methods (e.g. collection methods), the syntax of short query fragment generation using comparison operators, and ability to directly use Document instances with Pymongo also makes our standard Pymongo code just way prettier. So many fewer braces and quotes everywhere. And, of course, it's just dictionaries all the way down. MyDoc(name="Bob Dole") == dict(name="Bob Dole") to Pymongo. And remember, if you don't need typecasting on access, you can always access the field as a dictionary element: MyDoc[~MyDoc.name] (the ~ thing looks up the field name to be portable with renamed fields ;)

I hope this helps to clarify things a bit, and I look forward to any feedback you might have. We're always looking to improve things. (Documentation included!)

[amcgregor]

As a quick follow-up, Marrow Schema, which Marrow Mongo is built upon as the basis of the declarative syntax, has encouraged us at work to make everything declarative that we can, including things like controllers/endpoints and data processing pipeline descriptions. This ends up with us having database objects that can do things like self-render. Without MongoEngine's separation between Document and EmbeddedDocument (never understood that) our user session model can just embed the current in-progress invoice—people don't use unstructured storage for sessions any more, right? ;) This approach has been in production for four years at Illico. We eat a lot of our own dog food, to use an unfortunate but apt phrase!

You can see the public side of the work-in-progress migration of Contentment. (See also Contentment#12.)

[rittersport3]

A very encouraging and thorough answer, that was very generous of you!

We will definitely try porting our application to mongo marrow. We'll post our progress and experience, for sure.

[amcgregor]

Feel free to resurrect this thread at any time if you have general updates and re-open for questions. If you identify any issues or specific requests do not hesitate to create a separate issue.

Give the Queryable trait a try, it's missing a lot for which you can fall back on Pymongo, but can ease porting a bit. If there's anything you need added there, see #36. Trying really hard to not start wrapping literally everything the way prior systems did… we need more generators. 💃

If your team create any new Field derivatives or traits you'd like to share, fix any issues you encounter, or add features, I encourage you to fork and contribute changes back via pull request.