Add reindex feature to Upgrade Assistant #27457

joshdover · 2018-12-18T23:30:08Z

Summary

Related to #26368
Fixes #18469

This PR introduces the MVP reindexing feature to the 7.0 Upgrade Assistant.

Scope

This first iteration includes the following features:

Reindex indices created in 5.x
Persistence to persist reindex operation independent of the user's browser
Automatic handling of a few required index changes at reindex time (more on this below)

In a second PR to be merged in for 6.7 I will add:

Ability to cancel a running reindex operation
Cleanup of reindex saved objects once after completion.
Localized UI
Upgrading of internal indices (eg. Security)

Automatic index changes

There are two key situations that we are going to handle for the cluster admin automatically:

Removal of _all meta field from mappings
Conversion of deprecated boolean fields ("yes", "off", etc.). This is accomplished with a painless script added to the reindex command if any boolean fields are detected in the mapping.

Why are we doing this?

In 5.x, any indices that were created by indexing data without defining mappings defaulted to using the _all meta field. Because of this, we expect this to be a very common issue when reindexing 5.x indices.
If we chose not to handle converting boolean fields, cluster admins are going to handle this situation regardless. If we don't automatically handle this for them, we're not really saving them from any headaches.

Both of these changes are potentially breaking changes depending on the index is being used. Before starting a reindex on an index that has either of these problems, the UI alerts the user of the issue, how we handle it, and how it may affect any applications or scripts using this index (see screenshot below). We also require that the user go through a second confirmation before starting the reindex.

Screenshots

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

This was checked for cross-browser compatibility, including a check against IE11
Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios
This was checked for keyboard-only and screenreader accessibility

For maintainers

This was checked for breaking API changes and was labeled appropriately
This includes a feature addition or change that requires a release note and was labeled appropriately

elasticmachine · 2018-12-18T23:45:11Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-12-21T17:35:27Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-02T23:34:59Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-03T21:35:07Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-03T23:21:40Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

joshdover · 2019-01-04T00:03:19Z

retest

elasticmachine · 2019-01-04T00:47:40Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-04T20:15:38Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-04T22:54:12Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-07T23:17:50Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-08T16:41:02Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-08T17:15:43Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

joshdover · 2019-01-08T17:31:32Z

retest

elasticmachine · 2019-01-08T18:06:09Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-08T23:46:42Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-09T00:26:07Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-09T18:56:23Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-09T23:34:57Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-10T19:04:34Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

joshdover · 2019-01-25T20:39:31Z

@jbudz i18n is going to be in a follow-up PR. Want to get this in and forward-ported to master so that we can complete work for APM in 7.0

jbudz · 2019-01-25T21:11:02Z

Is it expected for a new template put every second? The polling is updating .kibana?

[2019-01-25T15:09:45,021][INFO ][o.e.c.m.MetaDataIndexTemplateService] [XoHVpDr] adding template [kibana_index_template:.kibana] for index patterns [.kibana]

joshdover · 2019-01-25T21:38:03Z

Is it expected for a new template put every second? The polling is updating .kibana?
[2019-01-25T15:09:45,021][INFO ][o.e.c.m.MetaDataIndexTemplateService] [XoHVpDr] adding template [kibana_index_template:.kibana] for index patterns [.kibana]

When I looked into this before, I found that this is from the current behavior of the Saved Object Service that @chrisdavies wants to remove in 7.0.

jbudz · 2019-01-25T21:41:58Z

Might be useful to select multiple indices at once, e.g. I have 30 filebeat-* indices with the same format and want to queue them all (flyout clicks can make it sorta slow). Not worried about this just noting at I go through things

joelgriffith · 2019-01-25T21:53:57Z

Pardon my noobishness: if I wanted to test this out (locally, for reporting purposes), would the required steps be:

Startup ES/Kibana @ 5.x.
Throw some data in it.
Tear down 5.x, spin up 7.x with this branch.
Initiate re-index (I'm assuming it's in management).
???
Profit

joshdover · 2019-01-25T21:57:45Z

@joelgriffith Mostly right. You'll want to spin up 6.7 (this is on 6.x branch right now) and go to Management > 7.0 Upgrade Assistant > Indices to find the UI.

elasticmachine · 2019-01-25T22:36:47Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-25T23:33:04Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

tylersmalley · 2019-01-28T07:13:44Z

x-pack/plugins/upgrade_assistant/server/lib/reindexing/index_settings.ts

+    throw new Error(`Indices with more than one mapping type are not supported in 7.0.`);
+  }
+
+  return mappings[mappingTypes[0]];


In 7.0 we will want to add a warning for types which are not _doc and enforce on re-indexing.

Should be easy enough, I can create a follow up PR for 7.0 after this gets merged.

elasticmachine · 2019-01-28T16:04:36Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-28T16:49:27Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

tylersmalley · 2019-01-28T18:18:28Z

I have gone through the code and done some pretty extensive functional testing on this PR, including adding a feature to tag and re-index APM specific indices. There are a few things left outstanding which will be handled in follow-up PR's.

Handle .tasks index Upgrade Assistant cannot reindex Elasticsearch's .tasks index #29454
Ability to cancel a reindex operation
Add a 'reset' button for reindexes that fail to delete the newly created index
Consolidate reindex warnings and progress UIs into single flyout UI
Cleanup reindex operation saved objects after completion

joelgriffith · 2019-01-28T23:24:08Z

Sorry for the massive delay @joshdover, but I was able to test this with some confidence locally! I think you're good to proceed

elasticmachine · 2019-01-29T00:01:07Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-01-29T16:14:22Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* Fixup saved objects * WIP reindex state machine * WIP UI * Separate status from current step * Add error messages * Copy types to CallWithInternalUser * Use backend worker * Add progress bar for large indices * Cleanup worker implementation * Add tests for ReindexService * Fix types * Fix CI * Add support for mapping coerced boolean fields * Add basic functional integration test * Cleanup reindexing backend, add more checks * Cleanup frontend code and add tests * Support removal of _default_ mapping + add tests * Add reindex warnings to reindex service * Add confirmation modal for reindex warnings * Cleanup flyout appearance * Generate new index name intelligently * Design tweaks * Show reindex button in both grouping modes * Add data archive integration tests * Change flyout design to two step process * Reorganize flyout files * Use custom steps component * Allow writes to indices if anything fails * Misc cleanup * Design edits * Only show warnings panel when reindex has not begun * Fix types * Handle moving existing aliases * Move integration tests to separate config * Fix ReindexButton bugs when changing pages * Fix polling service tests * Refactor ReindexService and ML handling (backend only) * Add credential caching and paused state to backend * Add paused state to frontend * Fix ts errors * Update copy * Add check for node version before ML step * Update data archive format * Update snapshots for React upgrade * Handle _default_ mappings correctly in warning checks * Use hashed state of reindex operation for credential storage * Address tyler's comments * Add watcher stop/starting * Use json-stable-stringify and sha256 for CredentialStore * Add types for json-stable-stringify

joshdover force-pushed the upgrade-checkup-reindex branch from 53cf03c to 262adc2 Compare December 21, 2018 17:24

joshdover force-pushed the upgrade-checkup-reindex branch from 262adc2 to 639529c Compare January 2, 2019 23:22

joshdover requested a review from a team as a code owner January 3, 2019 20:51

joshdover force-pushed the upgrade-checkup-reindex branch from a55c278 to 3b5fa5b Compare January 4, 2019 20:03

joshdover force-pushed the upgrade-checkup-reindex branch from 3b5fa5b to 39cb4bb Compare January 4, 2019 22:17

joshdover force-pushed the upgrade-checkup-reindex branch from 1ecdcb5 to d838ff5 Compare January 8, 2019 16:29

joshdover force-pushed the upgrade-checkup-reindex branch from d838ff5 to 85ee5dc Compare January 8, 2019 16:42

joshdover force-pushed the upgrade-checkup-reindex branch from df028d3 to 34cc97d Compare January 9, 2019 18:21

joshdover force-pushed the upgrade-checkup-reindex branch from 7487b0e to 4ca7c52 Compare January 10, 2019 18:28

joshdover closed this Jan 10, 2019

joshdover reopened this Jan 10, 2019

Add watcher stop/starting

adc7188

Use json-stable-stringify and sha256 for CredentialStore

609f412

tylersmalley reviewed Jan 28, 2019

View reviewed changes

Merge remote-tracking branch 'upstream/6.x' into upgrade-checkup-reindex

80e31f3

Add types for json-stable-stringify

574ed90

tylersmalley approved these changes Jan 28, 2019

View reviewed changes

jbudz approved these changes Jan 28, 2019

View reviewed changes

joshdover added release_note:enhancement and removed review labels Jan 28, 2019

Merge remote-tracking branch 'upstream/6.x' into upgrade-checkup-reindex

f4342df

joshdover force-pushed the upgrade-checkup-reindex branch from 95c76a3 to f4342df Compare January 29, 2019 15:30

joshdover merged commit 468eabb into elastic:6.x Jan 29, 2019

joshdover mentioned this pull request Jan 29, 2019

[7.0] Add reindex feature to Upgrade Assistant (#27457) #29516

Merged

3 tasks

joshdover added a commit that referenced this pull request Jan 29, 2019

[7.0] Add reindex feature to Upgrade Assistant (#27457) (#29516)

908db0d

ppf2 mentioned this pull request Jul 16, 2019

Upgrade assistant|reindex: Prompt to remove include_in_all in addition to _all #41213

Closed

Add reindex feature to Upgrade Assistant #27457

Add reindex feature to Upgrade Assistant #27457

Conversation

joshdover commented Dec 18, 2018 • edited Loading

Summary

Scope

Automatic index changes

Screenshots

Checklist

For maintainers

elasticmachine commented Dec 18, 2018

💔 Build Failed

elasticmachine commented Dec 21, 2018

💔 Build Failed

elasticmachine commented Jan 2, 2019

💔 Build Failed

elasticmachine commented Jan 3, 2019

💔 Build Failed

elasticmachine commented Jan 3, 2019

💔 Build Failed

joshdover commented Jan 4, 2019

elasticmachine commented Jan 4, 2019

💔 Build Failed

elasticmachine commented Jan 4, 2019

💔 Build Failed

elasticmachine commented Jan 4, 2019

💚 Build Succeeded

elasticmachine commented Jan 7, 2019

💔 Build Failed

elasticmachine commented Jan 8, 2019

💔 Build Failed

elasticmachine commented Jan 8, 2019

💔 Build Failed

joshdover commented Jan 8, 2019

elasticmachine commented Jan 8, 2019

💔 Build Failed

elasticmachine commented Jan 8, 2019

💔 Build Failed

elasticmachine commented Jan 9, 2019

💔 Build Failed

elasticmachine commented Jan 9, 2019

💚 Build Succeeded

elasticmachine commented Jan 9, 2019

💔 Build Failed

elasticmachine commented Jan 10, 2019

💔 Build Failed

joshdover commented Jan 25, 2019

jbudz commented Jan 25, 2019

joshdover commented Jan 25, 2019

jbudz commented Jan 25, 2019

joelgriffith commented Jan 25, 2019

joshdover commented Jan 25, 2019

elasticmachine commented Jan 25, 2019

💔 Build Failed

elasticmachine commented Jan 25, 2019

💔 Build Failed

tylersmalley Jan 28, 2019 • edited Loading

Choose a reason for hiding this comment

joshdover Jan 28, 2019

Choose a reason for hiding this comment

elasticmachine commented Jan 28, 2019

💔 Build Failed

elasticmachine commented Jan 28, 2019

💚 Build Succeeded

tylersmalley commented Jan 28, 2019

joelgriffith commented Jan 28, 2019

elasticmachine commented Jan 29, 2019

💔 Build Failed

elasticmachine commented Jan 29, 2019

💚 Build Succeeded

joshdover commented Dec 18, 2018 •

edited

Loading

tylersmalley Jan 28, 2019 •

edited

Loading