Migrations v2 docs #88820

rudolf · 2021-01-20T11:28:00Z

Summary

Related: #75780
Closes: #86315

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be whitelisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2021-01-20T11:28:03Z

Pinging @elastic/kibana-core (Team:Core)

rudolf · 2021-01-21T09:33:31Z

@elasticmachine merge upstream

…points

joshdover

From our testing, do we have any indication that there may be high load on ES from the parallel indexing from all the kibana nodes? If so, we may want to add a section about this, targeted at users that have high numbers of Kibana nodes.

It may even worth having a dedicated section about "Migrations with multiple Kibana nodes" that also mentions that all old nodes should be shutdown before starting the new nodes. Moving that detail out of the other sections into a section about that may help those users find that detail more effectively.

joshdover · 2021-01-21T13:25:21Z

docs/setup/upgrade/upgrade-migrations.asciidoc

+ - Prior to 6.5.0, saved objects were stored directly in an index named `.kibana`. After upgrading to version 7.12+, {kib} will migrate this index into `.kibana_{kibana_version}_001` and set `.kibana` up as alias to this index. +
+ - Prior to 7.4.0, task manager tasks were stored directly in an index name `.kibana_task_manager`. After upgrading to version 7.12+, {kib} will migrate this index into `.kibana_task_manager_{kibana_version}_001` and set `.kibana_task_manager` up as an index alias. +
+ - Prior to 7.12.0, saved objects and task manager tasks were stored in `.kibana_N` and `.kibana_task_manager_N` where N is a number that's increased every time an upgrade migration was performed. After upgrading to version 7.12+, {kib} will migrate these indices into `.kibana_{kibana_version}_001` and `.kibana_task_manager_{kibana_version}_001` respectively.


Instead of the "Prior to X" language, I think it would be more clear if we said which versions each statement applied to, example:

In Kibana versions 4.0.0 through 6.4.2, ...

In Kibana versions 6.5.0 through 7.3.2, ...

In Kibana versions 7.4.0 through 7.11.2, ...

I tried to condense this into a table

joshdover · 2021-01-21T13:26:54Z

docs/setup/upgrade/upgrade-migrations.asciidoc

-2. Identify any migration locks by comparing the output of `GET /_cat/aliases` and `GET /_cat/indices`. If e.g. `.kibana` is pointing to `.kibana_4` and there is a `.kibana_5` index, the `.kibana_5` index will act like a migration lock blocking further attempts. Be sure to check both the `.kibana` and `.kibana_task_manager` aliases and their indices.
-3. Remove any migration locks e.g. `DELETE /.kibana_5`. 
-4. Start up all {kib} instances.
+If {kib} terminates unexpectedly while migrating a saved object index it will automatically attempt to perform the migration again once the process has restarted. If upgrade migrations fail repeatedly, follow the advice in (preventing migration failures)[preventing-migration-failures]. Unlike previous versions of {kib}, v7.12+ does not require deleting any indices to release a failed migration lock. Once the root cause for the migration failure has been addressed, {kib} will automatically retry the migration without any further intervention.


Maybe we should add a very clearly worded warning that in 7.12+ it is not recommended to delete any indices or aliases without first contacting Elastic Support.

We should also add a note that if you're dealing with a version prior to 7.12 to refer to the older docs. Not sure if our doc system supports linking to previous versions, but if so that'd be nice.

I used stronger wording in 927b60a

docs/setup/upgrade/upgrade-standard.asciidoc

Co-authored-by: Josh Dover <[email protected]>

docs/setup/upgrade/upgrade-migrations.asciidoc

…into migrations-v2-docs

rudolf · 2021-01-22T15:17:02Z

From our testing, do we have any indication that there may be high load on ES from the parallel indexing from all the kibana nodes? If so, we may want to add a section about this, targeted at users that have high numbers of Kibana nodes.

I did some rudimentary testing by running migrations on 100k saved objects (much larger than "normal") and looking at the output of watch curl -s -X GET "elastic:changeme@localhost:9200/_cat/nodes?h=heap*\&v" -H 'Content-Type: application/json'. There isn't a notable impact on the current heap between 1 kibana node or 10 (the largest kibana deployment we know of). But the runtime of the migration doubles for 10 nodes, so there is a performance impact.

Migrations runtime with 100k objects:

Nodes	Runtime
1	2.27 minutes
2	2.27 minutes
3	3.49 minutes
10	4.21 minutes

Although at the end of the day, the impact of Kibana running migrations is most likely much less than the impact of users using Kibana and searching over much larger data sets than the saved objects index. If someone deploys 10 Kibana nodes it would be to support a large amount of concurrent users (or concurrent background work).

It may even worth having a dedicated section about "Migrations with multiple Kibana nodes" that also mentions that all old nodes should be shutdown before starting the new nodes. Moving that detail out of the other sections into a section about that may help those users find that detail more effectively.

We do also mention it in the actual step-by-step upgrade instructions https://www.elastic.co/guide/en/kibana/current/upgrade-standard.html

I feel like all the upgrade pages might benefit from an overhaul, there's a lot going on in them (with lots of warnings and callouts) which doesn't help users feel confident that they're doing things right. But I will add a dedicated multi-instance section and maybe just mention that for very large kibana deployments upgrade performance (and associated downtime) will be better if a single instance finished the upgrade before other instances were brought online.

* Migrations v2 docs * Not all kibana distributions automatically restarted a killed process * Mention that we add a write block to the outdated index * Formating: collapse three notes into a single note with three bullet points * Update docs/setup/upgrade/upgrade-standard.asciidoc Co-authored-by: Josh Dover <[email protected]> * Add table of outdated / upgraded indices per version of Kibana * Review feedback: separate section for multi-instance upgrade migrations * Review feedback: link to saved objects management * Review feedback: stronger wording for not deleting any indices Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Josh Dover <[email protected]>

* Migrations v2 docs * Not all kibana distributions automatically restarted a killed process * Mention that we add a write block to the outdated index * Formating: collapse three notes into a single note with three bullet points * Update docs/setup/upgrade/upgrade-standard.asciidoc Co-authored-by: Josh Dover <[email protected]> * Add table of outdated / upgraded indices per version of Kibana * Review feedback: separate section for multi-instance upgrade migrations * Review feedback: link to saved objects management * Review feedback: stronger wording for not deleting any indices Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Josh Dover <[email protected]> Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Josh Dover <[email protected]>

rudolf added 2 commits January 20, 2021 10:35

Migrations v2 docs

3e4f639

Not all kibana distributions automatically restarted a killed process

42e1bdd

Mention that we add a write block to the outdated index

fae6569

kibanamachine and others added 2 commits January 21, 2021 04:33

Merge branch 'master' into migrations-v2-docs

7b0e427

Formating: collapse three notes into a single note with three bullet …

cc31d7a

…points

joshdover reviewed Jan 21, 2021

View reviewed changes

Update docs/setup/upgrade/upgrade-standard.asciidoc

b791bde

Co-authored-by: Josh Dover <[email protected]>

mshustov reviewed Jan 22, 2021

View reviewed changes

docs/setup/upgrade/upgrade-migrations.asciidoc Outdated Show resolved Hide resolved

docs/setup/upgrade/upgrade-migrations.asciidoc Show resolved Hide resolved

rudolf added 3 commits January 22, 2021 14:22

Merge branch 'master' into migrations-v2-docs

35f4e3e

Add table of outdated / upgraded indices per version of Kibana

deaf6fc

Merge branch 'migrations-v2-docs' of https://github.com/rudolf/kibana …

4630bcf

…into migrations-v2-docs

rudolf added 4 commits January 22, 2021 17:15

Review feedback: separate section for multi-instance upgrade migrations

3ce8a18

Review feedback: link to saved objects management

8cca471

Merge branch 'master' into migrations-v2-docs

3c89168

Review feedback: stronger wording for not deleting any indices

927b60a

joshdover approved these changes Jan 25, 2021

View reviewed changes

rudolf merged commit eed9383 into elastic:master Jan 25, 2021

rudolf deleted the migrations-v2-docs branch January 25, 2021 14:37

rudolf mentioned this pull request Jan 26, 2021

[7.x] Migrations v2 docs (#88820) #89298

Merged

rudolf added the docs label Feb 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrations v2 docs #88820

Migrations v2 docs #88820

rudolf commented Jan 20, 2021 •

edited

Loading

elasticmachine commented Jan 20, 2021

rudolf commented Jan 21, 2021

joshdover left a comment

joshdover Jan 21, 2021

rudolf Jan 22, 2021

rudolf Jan 22, 2021

joshdover Jan 21, 2021

rudolf Jan 23, 2021

rudolf commented Jan 22, 2021 •

edited

Loading

Migrations v2 docs #88820

Migrations v2 docs #88820

Conversation

rudolf commented Jan 20, 2021 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Jan 20, 2021

rudolf commented Jan 21, 2021

joshdover left a comment

Choose a reason for hiding this comment

joshdover Jan 21, 2021

Choose a reason for hiding this comment

rudolf Jan 22, 2021

Choose a reason for hiding this comment

rudolf Jan 22, 2021

Choose a reason for hiding this comment

joshdover Jan 21, 2021

Choose a reason for hiding this comment

rudolf Jan 23, 2021

Choose a reason for hiding this comment

rudolf commented Jan 22, 2021 • edited Loading

rudolf commented Jan 20, 2021 •

edited

Loading

rudolf commented Jan 22, 2021 •

edited

Loading