New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v2 migration algorithm docs for rewriting saved object id's #93002

Merged

mshustov merged 8 commits into elastic:master from rudolf:v2-migrations-docs

Apr 29, 2021

Contributor

rudolf commented Mar 1, 2021 •

edited

Loading

Summary

The saved object migrations RFC had documentation for the v2 migrations algorithm, but when implementing it some steps were divided into two so it's no longer straightforward to compare the documentation with the implementation. This PR adds documentation that uses the same control state names as the source code to make it easier to compare.

beb6b57 changes the current algorithm to support rewriting saved object id's to address #86247

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

rudolf added Feature:Saved Objects project:ResilientSavedObjectMigrations labels

rudolf added the release_note:skip label

rudolf added 2 commits

March 1, 2021 10:52


          Document v2 migration algorithm using control state names from source…

10ec053

… code


          v2 migrations support for rewriting document _id's

beb6b57

rudolf force-pushed the v2-migrations-docs branch from 0c3c8c6 to beb6b57 Compare

March 1, 2021 10:57

rudolf changed the title ~~Document v2 migration algorithm using control state names~~ v2 migration algorithm docs for rewriting saved object id's

rudolf marked this pull request as ready for review

March 1, 2021 11:09

rudolf requested a review from a team as a code owner

March 1, 2021 11:09

rudolf force-pushed the v2-migrations-docs branch from 4895b3d to 9f5a4a8 Compare

March 1, 2021 11:37


          Minor edits and clarifications

551d57c

rudolf force-pushed the v2-migrations-docs branch from 9f5a4a8 to 551d57c Compare

March 1, 2021 11:37

rudolf added v7.13.0 v8.0.0 labels

rudolf mentioned this pull request

[Discuss] Too many saved-objects + migrations == 💥 #92933

Closed

mshustov reviewed

View reviewed changes

rfcs/text/0013_saved_object_migrations.md Outdated Show resolved Hide resolved

src/core/server/saved_objects/migrationsv2/README.md Outdated Show resolved Hide resolved

src/core/server/saved_objects/migrationsv2/README.md Outdated Show resolved Hide resolved

src/core/server/saved_objects/migrationsv2/README.md

+              to the same index. This version's migration has already been completed. Since
+              the same version could have plugins enabled at any time that would introduce
+              new transforms or mappings.
+                →  `OUTDATED_DOCUMENTS_SEARCH`

Contributor

mshustov Mar 3, 2021

Maybe we should add a diagram of transitions between states?

src/core/server/saved_objects/migrationsv2/README.md


		Atomically apply the `versionIndexReadyActions` using the _alias actions API. By performing the following actions we guarantee that if multiple versions of Kibana started the upgrade in parallel, only one version will succeed.

		1. verify that the current alias is still pointing to the source index

Contributor

mshustov Mar 3, 2021

Mind adding examples for this section?

Contributor Author

rudolf Mar 15, 2021

noticed I got UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK mixed up with MARK_VERSION_INDEX_READY will revisit this section

src/core/server/saved_objects/migrationsv2/README.md Show resolved Hide resolved

mshustov approved these changes

View reviewed changes

mshustov reviewed

View reviewed changes

src/core/server/saved_objects/migrationsv2/README.md Show resolved Hide resolved

src/core/server/saved_objects/migrationsv2/README.md

+              ## REINDEX_SOURCE_TO_TEMP_INDEX
+              ### Next action
+              `transformRawDocs` + `bulkIndexTransformedDocuments`

Contributor

mshustov Mar 3, 2021

Not a directly related question: Is space a part of SO id still?

Contributor Author

rudolf Mar 3, 2021

yes it is, but in 8.x all saved objects will become "share-aware" (I think that's the terminology we're using) which means we will use this algorithm plus the work in #80945 to rewrite their id's and remove the space out of the id.

mshustov reviewed

View reviewed changes

rfcs/text/0013_saved_object_migrations.md Outdated Show resolved Hide resolved

mshustov reviewed

View reviewed changes

src/core/server/saved_objects/migrationsv2/README.md

+              ## OUTDATED_DOCUMENTS_TRANSFORM
+              ### Next action
+              `transformRawDocs` + `bulkOverwriteTransformedDocuments`

Contributor

mshustov Mar 5, 2021

On the REINDEX_SOURCE_TO_TEMP_INDEX step, we transform all the known SO types. On this step, we transform outdated docs that haven't been processed on REINDEX_SOURCE_TO_TEMP_INDEX. Is that correct?

Contributor Author

rudolf Mar 15, 2021

yes 👍

Contributor Author

rudolf Mar 15, 2021

I had a brief description under CLONE_TEMP_TO_TARGET but I expanded that a little bit and moved it under OUTDATED_DOCUMENTS_SEARCH

rudolf added 3 commits

March 15, 2021 23:41


          Merge branch 'master' into v2-migrations-docs

f9104f8


          Fix markdown formatting

3bfc0e5


          Review comments: improve algorithm description

b7ae3d4

mshustov mentioned this pull request

Migration v2 waits for yellow cluster #96788

Merged

2 tasks

mshustov mentioned this pull request

Rewriting SO id during migration #97222

Merged

2 tasks

TinaHeiligers reviewed

View reviewed changes

src/core/server/saved_objects/migrationsv2/README.md

+              ## OUTDATED_DOCUMENTS_TRANSFORM
+              ### Next action
+              `transformRawDocs` + `bulkOverwriteTransformedDocuments`

Contributor

TinaHeiligers Apr 20, 2021

These are broken down into two separate steps to handle #90279

Contributor

mshustov Apr 29, 2021

we will need to update the docs either in #90279 or right after

spalger added v7.14.0 and removed v7.13.0 labels


          Merge branch 'master' into v2-migrations-docs


          add WAIT_FOR_YELLOW_SOURCE step

af75919

mshustov enabled auto-merge (squash)

April 29, 2021 12:13

mshustov added auto-backport backport:skip and removed auto-backport labels

mshustov merged commit c03a830 into elastic:master

madirey pushed a commit to madirey/kibana that referenced this pull request


          v2 migration algorithm docs for rewriting saved object id's (elastic#…

9cf5705

…93002)

* Document v2 migration algorithm using control state names from source code

* v2 migrations support for rewriting document _id's

* Minor edits and clarifications

* Fix markdown formatting

* Review comments: improve algorithm description

* add WAIT_FOR_YELLOW_SOURCE step

Co-authored-by: restrry <[email protected]>

rudolf mentioned this pull request

[7.x] v2 migration algorithm docs for rewriting saved object id's (#93002) #109472

Merged

rudolf added a commit to rudolf/kibana that referenced this pull request


          v2 migration algorithm docs for rewriting saved object id's (elastic#…

a15cfa6

…93002)

* Document v2 migration algorithm using control state names from source code

* v2 migrations support for rewriting document _id's

* Minor edits and clarifications

* Fix markdown formatting

* Review comments: improve algorithm description

* add WAIT_FOR_YELLOW_SOURCE step

Co-authored-by: restrry <[email protected]>
# Conflicts:
#	rfcs/text/0013_saved_object_migrations.md

rudolf added a commit that referenced this pull request


          v2 migration algorithm docs for rewriting saved object id's (#93002) (#…

80127a7

…109472)

* Document v2 migration algorithm using control state names from source code

* v2 migrations support for rewriting document _id's

* Minor edits and clarifications

* Fix markdown formatting

* Review comments: improve algorithm description

* add WAIT_FOR_YELLOW_SOURCE step

Co-authored-by: restrry <[email protected]>
# Conflicts:
#	rfcs/text/0013_saved_object_migrations.md

rudolf mentioned this pull request

[7.15] v2 migration algorithm docs for rewriting saved object id's (#93002) #109475

Merged

rudolf added a commit to rudolf/kibana that referenced this pull request


          v2 migration algorithm docs for rewriting saved object id's (elastic#…

…93002)

* Document v2 migration algorithm using control state names from source code

* v2 migrations support for rewriting document _id's

* Minor edits and clarifications

* Fix markdown formatting

* Review comments: improve algorithm description

* add WAIT_FOR_YELLOW_SOURCE step

Co-authored-by: restrry <[email protected]>
# Conflicts:
#	rfcs/text/0013_saved_object_migrations.md

rudolf added a commit that referenced this pull request


          v2 migration algorithm docs for rewriting saved object id's (#93002) (#…

53afc45

…109475)

* Document v2 migration algorithm using control state names from source code

* v2 migrations support for rewriting document _id's

* Minor edits and clarifications

* Fix markdown formatting

* Review comments: improve algorithm description

* add WAIT_FOR_YELLOW_SOURCE step

Co-authored-by: restrry <[email protected]>
# Conflicts:
#	rfcs/text/0013_saved_object_migrations.md

rudolf deleted the v2-migrations-docs branch

August 20, 2021 19:06

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip Feature:Saved Objects project:ResilientSavedObjectMigrations release_note:skip v7.14.0 v8.0.0