MongoDB persistence backend

[debrando]

Hi All! We're planning to adopt Dagster in our data modelling platform, which currently works with a MongoDB database - and we love it!.

Adding another database just for Dagster persistence would anyway raise objections by ITs, so I'm wondering if a MongoDB persistence backend was somewhat evaluated and if the community could be interested in it. As far as I see it's doesn't seem difficult, but we would be glad to contribute back and also make it a standard persistence option to avoid shipping a fork, so a general consensus and maybe some help would be great.

[sryza]

Hi @debrando - persistence can mean a couple different things in Dagster:

• The database where we store the list of runs and events that have been emitted during runs (right now usually Postgres).
• The place where we store values that solids output (right now often a cloud filesystem like S3 etc.).

Which one of those (or both) are you interested in using MongoDB for?

[debrando]

Ho, I was referring to the first, the job persistence. The platform stores "actual" data on FS or optionally RDBMS, and we already have an abstraction layer to take care of them.

[sryza]

That makes sense - thanks for the clarification

[Kilo59]

I would also be happy to help with this for exactly the same reason.
Also, like the original poster, we don't need it for storing the solid outputs, just runs & events and other metadata needed by dagster.

Can you point us in the right direction for getting starting with creating this.
This section references creating our own MongoRunStorage but doesn't link to any interface, ABC or datamodel to go about doing that.
https://docs.dagster.io/deployment/overview#configuring-dagster-deployments

For example, Dagster uses the SqliteRunStorage by default to store information about pipeline runs. You can swap it out with the Dagster-provided PostgresRunStorage instead or even write a MongoRunStorage.

[helloworld]

If you end up going down this path I'd be happy to provide more guidance as you implement this! Please feel free to reach out with any follow up questions.

[Kilo59]

Thanks for the links.
I couldn't find where the Postgres or MySQL implementations were defined.

I'm not sure exactly where to start.

My two initial ideas were ...

1. See if the Dagster team had done any thinking about a generalized NoSQL Runstorage interface that's hopefully simpler to implement.

2. Work on denormalizing these Tables/Schemas. Although first I probably need to understand them 😄 .
   https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/storage/runs/schema.py

https://www.mongodb.com/blog/post/6-rules-of-thumb-for-mongodb-schema-design-part-1
https://www.mongodb.com/blog/post/6-rules-of-thumb-for-mongodb-schema-design-part-2
https://www.mongodb.com/blog/post/6-rules-of-thumb-for-mongodb-schema-design-part-3

[sidkmenon-zz]

Hello! I wrote the MySQL support additions so I figure those commits might come in handy - here is a range of commits that also include a SQL-based polling event watcher. This could be useful as a reference if you'd like to implement event log storage for MongoDB as well.

As far as where the Postgres & MySQL implementations are defined, they're located under python_modules/libraries/dagster-postgres and python_modules/libraries/dagster-mysql, respectively.

As far as your other two questions, I'm not quite sure - I'll circle back with the team & update this issue tomorrow! Also echoing @helloworld, happy to help with any and all questions you might have!

[bernardocaldas]

Anyone started this? @debrando @Kilo59

[Kilo59]

@bernardocaldas
No.
And after looking at the RunStorage interface I'm not going to attempt it.

We will probably just end up using the Postgres DB, (if we end up using Dagster).