Version controlled seeder

[wzrdtales]

Implementation of a version controlled seeder, plus seeder in general.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[wzrdtales]

@BorntraegerMarc To answer on the state asked over in #292

No short answer here currently about the state.

There are some things I can do quickly and some that need a bit of love. Lets start with the things I can do quite quickly:

• Push out seeders as is, the interface separatation is there basically just hooks would need to be added
• Add some of the basic operations: insert, delete, update, search

What needs a bit of more love:

• Make seeders actually useful and more than a separation of two completely different things (DML vs DDL)*
• Rewrite the loader
• Rewrite the seed and migrator logic
• Separate file from seed and migrator logic
• Separate walker from seed and migrator logic
• Add version selector to choose correct seed and migrator logic
• Add a mode switcher, to switch between seeds and migrations
• Add a fail safe processor for large manipulations

• To make them useful I would want to add some helpers that really help the user to pursue what he wants to do.

Thoughts? Would it be from benefit to get a seeder version quickly with just the very basic actions? Already thought about doing exactly this though, currently I do work to rewrite the whole logic that loads and processes the migrations and seeds as I look forward to have the new migration schema in as quickly as possible. Especially for seeds those will include important benefits, like atomic operations and some other goodies. If it would really benefit the community to have the rough implementation I might go ahead to push that out first, but that would mean that you could expect quite soon a change of how they work again. Which would not break in any as we already have support for multiple different versions of seed versions. A last option would also be do the rough version and release the seeders v1 version already with the new schema, but with hard limitations of features for them.

To add the fail safe processor was targeted anyway to be added later, but just listed it here for completeness.

[wzrdtales]

To go a bit more into detail, mentioned goodies include:

A Concurrency manager - This will finally allow to safely integrate db-migrate into your application and just let it run when your application starts. Just one instance will operate at the same time. Basically I plan to insert a json metadata row as row 0 into the migrations and seed table (or maybe introducing a new table not sure yet, but that would break with the drivers, so I would like to avoid that)

an event emitter that emits events to let your application know when changes successfully processed which would allow to roll out changes very dynamically. I.e. you've got a new feature that depends on the existence of that one particular column, but does not care about the rest of the migrations, your application could then use feature toggles and toggle those with the help of those events.

Real atomic operations that can be rolled back in any scenario if wanted. Even without the help of transactions. Basically db-migrate would by default rely on transactions, as those are way faster than safe guards we can implement. But if the driver tells db-migrate there are no transactions for this db, than db-migrate could fall back to those routines. Or for example in the case of MySQL transactions can't be used to rollback DDL, in MariaDB there is an open task for this to enable this, but that is open for years already. Also here we can easily rollback without the help of transactions. And this also is possible in the other direction, i.e. for cockroachdb, where DDL within a transaction can return as completed instantly, while cockroach might delay the real execution of those as cockroachdb does by default online DDL.

[wzrdtales]

With this changes I just pushed to this branch 6f40d6b, seeders basically could be used already as far as the core is concerned though. Naturally the drivers would need to get the functionality added, that is not yet there.

[BorntraegerMarc]

Nice summary, thanks! My point of view: I'm eager to test the seeders but not necessarily depending on it. So I would be totally fine to test out seeders in the separate branch and wait to use it in my application once all features have been implemented. I think the community thinks the same.

But something that would make my life easier: can we remove those long warnings in the console insert is obsolete use version controlled seeders instead? If we decide that we're gonna wait to release seeders until they are fully ready :)

[BorntraegerMarc]

Hi @wzrdtales just wanted to ask if you've considered my idea to disable the console warning 🙂

[wzrdtales]

Sorry! Yes I do consider it, it is a fair deal!

[wzrdtales]

I do appreciate your engagement @BorntraegerMarc, it is just always an up and down for me when it comes to time that I've got left :)

[BorntraegerMarc]

@wzrdtales no problemo. I understand the feeling only to well :)

Could you hit us with a message in case you decide to release a new version with the logs disabled? (Would really appreciate it)

[wzrdtales]

Will do @BorntraegerMarc , currently am writing down some of the new logic and also refactoring some of the files to use const and let instead of var now that support prior 4.0.0 is dropped :)

[wzrdtales]

Good news are, I had some time today and I mostly finished the rough stuff for the loader. it at least works already.

[wzrdtales]

@BorntraegerMarc
Released

• [email protected]

This includes the refactoring to remove the deprecation notice for now.

Also the following fixes flown in

https://github.com/db-migrate/node-db-migrate/blob/467bf2a7ae72a05805015f7595335d5efbacd3ce/CHANGELOG.md#0103-2018-02-03

[stanislavt]

@wzrdtales Any updates?

[wzrdtales]

I wont have any time before before june, concerning the seeders they are still on hold if you want to help working on them you're welcome though :)

[Ash-oi]

hey @wzrdtales love your work, just wondering how this feature is going?

[wzrdtales]

closed in favour of redeclaration