feat: migration to single table SQL schema #707
Conversation
… into feat/persistence-migration-path
| @@ -0,0 +1 @@ | |||
| DROP TABLE "keto_namespace"; | |||
There was a problem hiding this comment.
This table was actually never used, not even in the old versions of Keto. Therefore just dropping it.
cmd/namespace/migrate_legacy.go
Outdated
| if err := migrator.MigrateDown(cmd.Context(), n); err != nil { | ||
| _, _ = fmt.Fprintf(cmd.ErrOrStderr(), "Could not migrate down: %s\n", err.Error()) | ||
| return cmdx.FailSilently(cmd) | ||
| } |
There was a problem hiding this comment.
I was first unsure whether we should migrate down directly without user consent, but I think it is fine because we do it after we have successfully transferred the data. Otherwise we would need a separate command just to migrate the namespace down, which is a bit of an overkill IMO.
There was a problem hiding this comment.
Hm, the problem here would be that if something goes wrong, the old data is gone. We should definitely respect this when executing this on our DB!
There was a problem hiding this comment.
But if something goes wrong before, we would have already returned.
There was a problem hiding this comment.
I assume the comment is no longer relevant as per https://github.com/ory/keto/pull/707/files#r712027278
… into feat/persistence-migration-path
… into feat/persistence-migration-path
cmd/namespace/migrate_legacy.go
Outdated
| if err := migrator.MigrateDown(cmd.Context(), n); err != nil { | ||
| _, _ = fmt.Fprintf(cmd.ErrOrStderr(), "Could not migrate down: %s\n", err.Error()) | ||
| return cmdx.FailSilently(cmd) | ||
| } |
There was a problem hiding this comment.
Hm, the problem here would be that if something goes wrong, the old data is gone. We should definitely respect this when executing this on our DB!
| for _, r := range rs { | ||
| ri, err := r.toInternal() | ||
| if err != nil { | ||
| m.d.Logger().WithField("relation_tuple", r).WithField("hint", "https://github.com/ory/keto/issues/661").WithError(err).Warn("Skipping relation tuple, it seems to be in a broken state. Please recreate it manually.") |
There was a problem hiding this comment.
Can this actually happen? How would one recover from this, given that we delete the data after the migration is done?
There was a problem hiding this comment.
As described in the issue, you can theoretically create a tuple that is serialized for the legacy table into a format that is not de-serializable anymore. The only way to recover is to recreate the tuple once the rest is migrated. But you are right, we should maybe not migrate down in this case.
Co-authored-by: hackerman <[email protected]>
… into feat/persistence-migration-path
Related issue(s)
closes #628
Further Comments
There is a new GH action now that runs an "e2e" kind of script. It first downloads Keto v0.6.0-alpha.3, applies it's migrations, and then adds some data. After that we migrate the data over with the current checked out branch and then use the check-API to see if the data are available and work properly. The edge cases are covered in the unit tests.
Commit Message (WIP)
This change adds a migration path from Keto version v0.6.x to the new persistence structure introduced by #638. Every namespace has to be migrated separately, or you can use the CLI to detect and migrate all namespaces at once. Have a look at
keto help namespace migrate legacyfor all details.Please make sure that you backup the database before running the migration command. For live-migrations, it is possible to run Keto v0.6 along v0.7 while migrating. Please note that this migration might be a bit slower than usual, as we have to pull the data from the database, transcode it in Keto, and then write it to the new table structure.
Future versions of Keto will not include this migration script, so you will have to migrate to v0.7 to move on from there.